Alert Runbooks
Alert RunbooksAlert runbooksalerts-k8s-exporterEtcdInsufficientMembers

EtcdInsufficientMembers

EtcdInsufficientMembers

Description

This alert fires when the number of healthy etcd cluster members falls below the quorum threshold. etcd requires a strict majority of members to be available in order to accept write requests. When quorum is lost, etcd halts all writes and the Kubernetes API server is no longer able to persist state changes, effectively freezing cluster control-plane operations.

Possible Causes:

Severity estimation

Critical — quorum loss is a cluster-halting event.

Troubleshooting steps

  1. Determine how many members are down and which ones

    • Command / Action:
      • List all etcd pods and check their status

      • kubectl get pods -n kube-system -l component=etcd -o wide

    • Expected result:
      • All pods should be in Running state; pods in CrashLoopBackOff, Error, Pending, or Terminating are the unhealthy members
    • additional info:
      • Note the node name for each unhealthy pod — multiple pods on different nodes failing simultaneously suggests a broader infrastructure issue (network segment, storage backend, cloud AZ)

  1. Check the status of the nodes running the failed etcd members

    • Command / Action:
      • Verify whether the nodes themselves are alive and healthy

      • kubectl get nodes -o wide

      • kubectl describe node <node-name>

    • Expected result:
      • Nodes are in Ready state with no MemoryPressure, DiskPressure, or PIDPressure conditions
    • additional info:
      • If nodes are NotReady or unreachable, the root cause is infrastructure-level (network partition, hardware failure, cloud outage) — address the node issue first
      • If nodes are Ready but etcd pods are failing, the problem is within etcd itself

  1. Check logs of the failed etcd pods

    • Command / Action:
      • Identify why the members are failing to start or stay running

      • kubectl logs -n kube-system <failed-etcd-pod> –tail=100

      • kubectl logs -n kube-system <failed-etcd-pod> –previous –tail=100

      • kubectl describe pod -n kube-system <failed-etcd-pod>

    • Expected result:
      • Logs reveal a specific failure reason (disk full, certificate expired, data corruption, network error)
    • additional info:
      • Common log patterns that indicate the failure mode:
        • no space left on device — disk full, check /var/lib/etcd volume usage
        • certificate has expired — TLS cert renewal required before member can rejoin
        • failed to find database snapshot — possible data corruption, recovery may be needed
        • rafthttp: failed to dial — network issue preventing peer communication

  1. Attempt to restart the failed etcd members

    • Command / Action:
      • Delete the failing pod(s) to trigger an automatic restart; static pods are managed by kubelet and restart automatically

      • kubectl delete pod -n kube-system <failed-etcd-pod>

    • Expected result:
      • The pod restarts, rejoins the cluster, and quorum is restored; EtcdInsufficientMembers and EtcdNoLeader should resolve
    • additional info:
      • Do not restart all failed members simultaneously if any healthy member remains — restart them one at a time to avoid permanently losing quorum
      • A restarting member may lag behind the leader’s RAFT index briefly; this is normal and self-heals within seconds

  1. Verify quorum has been restored from a healthy member

    • Command / Action:
      • Exec into a running etcd pod and check the cluster health and member list

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint health –cluster –write-out=table

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –cluster –write-out=table

    • Expected result:
      • All members report healthy: true; exactly one member shows IS LEADER: true; the previously failed member is back with a synced RAFT INDEX
    • additional info:
      • Pass --cacert, --cert, --key if TLS is required. For RKE2: certs are at /var/lib/rancher/rke2/server/tls/etcd/
      • If a member cannot rejoin after restart (e.g. due to data corruption), it may need to be removed from the cluster and re-added as a fresh learner member

  1. Check disk space on affected etcd nodes

    • Command / Action:
      • SSH into each affected node and verify the etcd data volume is not full

      • df -h /var/lib/etcd

      • du -sh /var/lib/etcd

    • Expected result:
      • Disk usage is well below capacity on all etcd nodes
    • additional info:
      • A full disk is a common silent killer — etcd will fail to write WAL entries and crash without obvious networking symptoms
      • If disk is full, free space by triggering compaction and defragmentation; refer to EtcdDatabaseHighFragmentationRatio and EtcdExcessiveDatabaseGrowth

  1. Escalate to etcd disaster recovery if quorum cannot be restored

    • Command / Action:
      • If too many members have lost data or cannot rejoin, restore the cluster from the most recent etcd snapshot
      • Locate the latest snapshot:

      • ls -lht /var/lib/etcd/member/snap/

      • Follow the etcd disaster recovery procedure to restore from snapshot
    • Expected result:
      • Cluster is restored to the state at snapshot time; control-plane operations resume
    • additional info:
      • Restoring from snapshot loses any state written after the snapshot was taken — this is a last resort
      • In managed Kubernetes environments (EKS, GKE, AKS), contact the cloud provider; etcd management may be handled externally
      • Ensure all etcd nodes are stopped before starting a snapshot restore to avoid split-brain

Additional resources