EtcdMembersDown

EtcdMembersDown

Description

This alert fires when one or more etcd cluster members are unreachable or not responding. While the cluster may still have quorum and continue to function, a reduced member count leaves no tolerance for further failures and puts the cluster at risk of losing quorum entirely.

Possible Causes:

• etcd process crashed or is in CrashLoopBackOff on one or more nodes
• Node running etcd is down, rebooting, or under heavy resource pressure
• Network partition preventing a member from communicating with the rest of the cluster
• Disk I/O saturation causing etcd to miss heartbeats and appear unresponsive
• Certificate expiry preventing a member from re-joining the cluster

Severity estimation

Medium to High — depends on the cluster size and how many members are down.

• Medium if one member is down in a 3-node cluster (quorum still held at 2/3) — cluster continues to operate but has zero fault tolerance
• High if two members are down in a 3-node cluster — quorum is lost and EtcdNoLeader or EtcdInsufficientMembers will likely fire as well
• A single member down should be resolved promptly: any further failure will halt the entire cluster

Troubleshooting steps

1. Identify which etcd member(s) are down

   • Command / Action:
     • Check the status of all etcd pods

     • kubectl get pods -n kube-system -l component=etcd -o wide

   • Expected result:
     • All pods in Running state; any pod in CrashLoopBackOff, Error, or Pending is the affected member
   • additional info:
     • Note which node the affected pod runs on — node-level issues (disk, memory, network) are a common root cause

2. Check the node running the affected etcd member

   • Command / Action:
     • Verify the node itself is healthy and not under pressure

     • kubectl describe node <node-name>

     • kubectl get node <node-name>

   • Expected result:
     • Node is in Ready state with no MemoryPressure, DiskPressure, or PIDPressure conditions
   • additional info:
     • If the node is NotReady, address the node issue first — the etcd member will recover once the node is healthy

3. Check logs of the affected etcd pod

   • Command / Action:
     • Look for the root cause of the crash or connectivity failure

     • kubectl logs -n kube-system <etcd-pod> –tail=100

     • kubectl describe pod -n kube-system <etcd-pod>

   • Expected result:
     • Logs indicate why the member is failing (e.g. disk full, certificate error, network timeout)
   • additional info:
     • Common log patterns to look for: no space left on device, certificate has expired, failed to send out heartbeat on time, took too long

4. Verify cluster health and quorum from a healthy member

   • Command / Action:
     • Exec into a healthy etcd pod and check overall cluster status

     • kubectl exec -n kube-system <healthy-etcd-pod> – etcdctl endpoint health –cluster –write-out=table

     • kubectl exec -n kube-system <healthy-etcd-pod> – etcdctl endpoint status –cluster –write-out=table

   • Expected result:
     • Healthy members report health: true; the down member reports a connection error
   • additional info:
     • Confirm a leader is still elected (IS LEADER: true on one member) — if no leader exists, refer to EtcdNoLeader

5. Restart the affected etcd member

   • Command / Action:
     • Delete the pod to trigger an automatic restart (static pods restart automatically)

     • kubectl delete pod -n kube-system <etcd-pod>

   • Expected result:
     • Pod restarts and rejoins the cluster; etcdctl endpoint health shows the member as healthy again
   • additional info:
     • If the pod keeps crash-looping, check disk space on the node: etcd is sensitive to disk pressure and will fail to start if the data directory is on a full volume

     • df -h /var/lib/etcd

6. Check etcd data directory disk usage

   • Command / Action:
     • SSH into the affected node and check disk space

     • df -h /var/lib/etcd

     • du -sh /var/lib/etcd

   • Expected result:
     • Disk usage is well below capacity; no no space left on device errors
   • additional info:
     • If the volume is full, check for oversized etcd snapshots or a missing compaction/defragmentation routine; refer to EtcdDatabaseHighFragmentationRatio and EtcdExcessiveDatabaseGrowth

7. Confirm the member has rejoined and the alert resolves

   • Command / Action:
     • Verify all members are healthy and reporting metrics

     • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –cluster –write-out=table

   • Expected result:
     • All members show health: true; the previously failing member is back with an up-to-date RAFT INDEX
   • additional info:
     • A rejoining member may lag briefly behind the leader’s RAFT index while catching up — this is normal and resolves within seconds

Additional resources

• etcd operation guide — failure modes
• etcd disaster recovery
• Kubernetes etcd cluster administration
• Related alert: EtcdNoLeader
• Related alert: EtcdInsufficientMembers
• Related alert: EtcdDatabaseHighFragmentationRatio
• Related alert: EtcdExcessiveDatabaseGrowth