Alert Runbooks
Alert RunbooksAlert runbooksalerts-k8s-exporterEtcdHighNumberOfLeaderChanges

EtcdHighNumberOfLeaderChanges

EtcdHighNumberOfLeaderChanges

Description

This alert fires when the etcd cluster is experiencing an abnormally high rate of leader elections.
In a healthy cluster, leader changes are rare events — they occur only during planned maintenance, member restarts, or genuine failures. A high rate of leader changes means members are repeatedly losing connectivity or failing to send heartbeats in time, forcing the cluster to re-elect a leader. Frequent elections degrade write performance, increase latency for Kubernetes API requests, and indicate an underlying instability that can escalate to EtcdNoLeader or quorum loss.

Possible Causes:

Severity estimation

Medium to High severity — the cluster is functional but unstable.

Frequent leader elections are a warning sign; unresolved, they tend to escalate into harder failures.

Troubleshooting steps

  1. Check the current leader election rate in Prometheus

    • Command / Action:
      • Query the metric that counts leader changes to quantify the rate

      • increase(etcd_server_leader_changes_seen_total[1h])

    • Expected result:
      • Value close to 0 over the last hour; alert fires when changes are significantly elevated
    • additional info:
      • Run by (instance) to see which member(s) are involved in the elections; a single member repeatedly triggering elections points to a node-level problem

  1. Check etcd pod status and restart counts

    • Command / Action:
      • Look for CrashLoopBackOff, frequent restarts, or OOMKill events

      • kubectl get pods -n kube-system -l component=etcd -o wide

      • kubectl describe pod -n kube-system <etcd-pod>

    • Expected result:
      • All pods in Running state with low restart counts and no OOMKilled events in the event log
    • additional info:
      • A pod with a high RESTARTS count that is currently Running may have been crashing and recovering frequently — check the previous container logs with --previous

  1. Check etcd logs for heartbeat or election messages

    • Command / Action:
      • Look for slow heartbeats, election events, and disk latency warnings

      • kubectl logs -n kube-system <etcd-pod> –tail=200 | grep -E ’leader|heartbeat|election|slow|took too long'

    • Expected result:
      • Logs show stable leadership with no repeated election messages
    • additional info:
      • Key patterns to look for:
        • failed to send out heartbeat on time — disk or CPU is too slow
        • elected leader repeated many times — elections are happening
        • took too long — disk write latency is above etcd’s heartbeat interval
        • wal: sync duration — WAL fsync is slow, often the root cause

  1. Check disk I/O latency on the etcd node

    • Command / Action:
      • etcd is highly sensitive to disk latency; check the node’s I/O wait and disk throughput

      • kubectl exec -n kube-system <etcd-pod> – sh -c “cat /proc/$(pgrep etcd)/io”

      • Or SSH to the node and run: iostat -x 2 5
    • Expected result:
      • await (average I/O wait) under ~1 ms for the etcd disk; higher values indicate disk pressure
    • additional info:
      • etcd requires single-digit millisecond disk latency for its WAL writes; HDDs or overloaded SSDs are a common root cause
      • Check whether noisy-neighbor workloads share the same disk as etcd’s data directory (/var/lib/etcd by default)

  1. Check node CPU and memory pressure

    • Command / Action:
      • Verify the node is not under CPU starvation or memory pressure

      • kubectl describe node <node-name>

      • kubectl top node <node-name>

    • Expected result:
      • Node conditions show no MemoryPressure, DiskPressure, or PIDPressure; CPU usage is not consistently at 100%
    • additional info:
      • If the etcd node is shared with other workloads, consider adding a taint to prevent non-etcd pods from scheduling on it: node-role.kubernetes.io/etcd=:NoSchedule

  1. Check network latency between etcd members

    • Command / Action:
      • Verify round-trip latency between etcd peer nodes is within acceptable bounds
      • From one etcd node, ping the other etcd node IPs:

      • ping <etcd-peer-ip>

      • Or check the Prometheus metric: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[5m]))
    • Expected result:
      • Peer round-trip latency under ~5 ms; heartbeat timeout (default 1s) must exceed network latency significantly
    • additional info:
      • If nodes are across availability zones with higher latency, the etcd --heartbeat-interval and --election-timeout may need to be increased accordingly
      • Check for network policies or security groups that could be throttling or dropping etcd peer traffic on port 2380

  1. Verify etcd cluster stability after investigation

    • Command / Action:
      • Confirm the leader is stable and the election rate has dropped

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –cluster –write-out=table

    • Expected result:
      • One consistent leader; RAFT TERM is not incrementing rapidly when checked multiple times
    • additional info:
      • If the RAFT TERM counter is incrementing rapidly (visible by running the command twice with a few seconds gap), elections are still occurring and the root cause has not been resolved

Additional resources