EtcdNoLeader

EtcdNoLeader

Description

This alert fires when an etcd member reports that it has no cluster leader. Without a leader, etcd cannot process any write requests, which means the Kubernetes API server is unable to persist state changes — effectively making the cluster read-only or unresponsive.

Possible Causes:

• Loss of quorum: too many etcd members are down or unreachable (e.g. 2 of 3 members unavailable)
• Network partition isolating the etcd member(s) from the rest of the cluster
• All etcd members are running but unable to communicate with each other
• etcd process crashed or restarted on the leader node during an election
• Clock skew between etcd nodes causing election timeouts

Severity estimation

Critical — this alert requires immediate attention.

Without a leader, etcd rejects all write requests. The Kubernetes API server will fail to create, update, or delete any resources. Running workloads continue operating but no new scheduling, scaling, or configuration changes are possible. If left unresolved, this will cascade into broader cluster failure.

Troubleshooting steps

1. Identify which etcd members are affected

   • Command / Action:
     • Check the status of all etcd pods and identify which are running

     • kubectl get pods -n kube-system -l component=etcd -o wide

   • Expected result:
     • All etcd pods should be in Running state; any in CrashLoopBackOff, Pending, or Error are the likely cause
   • additional info:
     • In RKE2/K3s clusters etcd runs as a static pod; check the node directly if the pod is not listed

2. Check etcd cluster member health

   • Command / Action:
     • Exec into a running etcd pod and query endpoint health

     • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint health –cluster –write-out=table

   • Expected result:
     • All endpoints report healthy: true; at least a quorum of members must be healthy for a leader to be elected
   • additional info:
     • Pass --cacert, --cert, --key if TLS is required. For RKE2: certs are at /var/lib/rancher/rke2/server/tls/etcd/

3. Check etcd member list and leader status

   • Command / Action:
     • Retrieve the current member list and see which node held leadership

     • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –cluster –write-out=table

   • Expected result:
     • Exactly one member shows IS LEADER: true; all others show false
   • additional info:
     • If no member shows IS LEADER: true, quorum has been lost — proceed to step 5

4. Check etcd pod logs for election or network errors

   • Command / Action:
     • Look for leader election failures, timeouts, or connectivity errors

     • kubectl logs -n kube-system <etcd-pod> –tail=100

   • Expected result:
     • Logs show a successful leader election: became leader at term X
   • additional info:
     • Look for lines containing lost leader, failed to send out heartbeat, elected leader, or raft: proposal dropped — these indicate the election timeline and failure mode

5. Check node connectivity between etcd members

   • Command / Action:
     • Verify the etcd nodes can reach each other on port 2380 (peer communication)

     • kubectl get nodes -o wide

     • Check network policies or firewall rules blocking inter-node traffic on port 2379/2380
   • Expected result:
     • All etcd nodes are reachable from each other; no firewall rules blocking etcd peer ports
   • additional info:
     • etcd uses port 2379 for client requests and 2380 for peer communication; both must be open between all etcd nodes

6. Restart unresponsive etcd members

   • Command / Action:
     • If a member is in CrashLoopBackOff or not responding, delete the pod to trigger a restart (static pods restart automatically)

     • kubectl delete pod -n kube-system <etcd-pod>

   • Expected result:
     • The pod restarts and rejoins the cluster; a new leader election completes within seconds
   • additional info:
     • For static pods, you can also move the manifest temporarily out of /etc/kubernetes/manifests/ and back to force a restart
     • Do not restart all etcd members simultaneously — this will permanently lose quorum

7. Verify the alert resolves after leader election

   • Command / Action:
     • Confirm a leader is now elected and the API server is responsive

     • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –cluster –write-out=table

     • kubectl get nodes

   • Expected result:
     • One member shows IS LEADER: true; kubectl get nodes returns results without timeout
   • additional info:
     • If quorum cannot be restored automatically, etcd cluster recovery from backup may be required — refer to the etcd disaster recovery documentation

Additional resources

• etcd disaster recovery
• etcd runtime reconfiguration
• Kubernetes etcd cluster administration
• Related alert: EtcdMembersDown
• Related alert: EtcdInsufficientMembers
• Related alert: EtcdHighNumberOfLeaderChanges