Alert Runbooks
Alert RunbooksAlert runbooksalerts-k8s-exporterEtcdExcessiveDatabaseGrowth

EtcdExcessiveDatabaseGrowth

EtcdExcessiveDatabaseGrowth

Description

This alert fires when the etcd database is growing at an abnormally fast rate, predicting that the quota will be exhausted sooner than expected.
Unlike EtcdDatabaseQuotaLowSpace which reacts to current size, this alert is predictive — it detects a growth trend and warns before the quota is hit. Rapid growth typically signals an underlying problem with object churn, missing compaction, or an abnormal workload event rather than normal incremental growth.

Possible Causes:

Severity estimation

Medium to High severity — the cluster is still functional but is on a trajectory toward quota exhaustion.

The main risk is that rapid growth can outpace manual intervention; act early.

Troubleshooting steps

  1. Measure current DB size and growth rate

    • Command / Action:
      • Check current database size across all members

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=table

    • Expected result:
      • DB SIZE is stable or growing slowly; note the current value and recheck after a few minutes
    • additional info:
      • Use the Prometheus query rate(etcd_mvcc_db_total_size_in_bytes[5m]) to visualize growth rate over time

  1. Verify compaction is running

    • Command / Action:
      • Check etcd pod logs for compaction activity

      • kubectl logs -n kube-system <etcd-pod> | grep -i compact

    • Expected result:
      • Regular compaction entries appear in the logs
    • additional info:
      • If compaction is absent, check the kube-apiserver flag --etcd-compaction-interval (default 5m) and --auto-compaction-retention

  1. Identify what is growing — check key counts by prefix

    • Command / Action:
      • List etcd key counts by prefix to find the noisiest resource type

      • kubectl exec -n kube-system <etcd-pod> – etcdctl get / –prefix –keys-only | sed ’s|/[^/]*$||’ | sort | uniq -c | sort -rn | head -20

    • Expected result:
      • Key distribution matches expected cluster state; no single prefix dominates unexpectedly
    • additional info:
      • A prefix like /registry/events or /registry/pods with unexpectedly high counts points to the responsible workload

  1. Find high-churn namespaces or workloads

    • Command / Action:
      • Look for namespaces with abnormally high object counts or event rates

      • kubectl get events –all-namespaces –sort-by=’.lastTimestamp’ | tail -30

      • kubectl get pods –all-namespaces | wc -l

    • Expected result:
      • No single namespace or workload is generating an unusual volume of events or objects
    • additional info:
      • A runaway Job, CronJob, or operator reconciliation loop will appear as a flood of events in a specific namespace

  1. Run compaction manually if needed

    • Command / Action:
      • Force a compaction to trim accumulated revisions immediately

      • REVISION=$(kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=json | jq ‘.[0].Status.header.revision’)

      • kubectl exec -n kube-system <etcd-pod> – etcdctl compact $REVISION

    • Expected result:
      • compacted revision <N> — old revisions are removed
    • additional info:
      • Follow up with defragmentation if DB SIZE remains high after compaction

  1. Defragment etcd members to reclaim space

    • Command / Action:
      • Defragment one member at a time

      • kubectl exec -n kube-system <etcd-pod> – etcdctl defrag –endpoints=https://127.0.0.1:2379

    • Expected result:
      • DB SIZE decreases after defragmentation; growth rate slows or stabilizes
    • additional info:
      • Defragment sequentially — one member at a time — to avoid quorum loss

  1. Address the root cause of excessive growth

    • Command / Action:
      • Stop or fix the workload responsible for the growth
      • For runaway controllers: scale down, patch, or delete the offending deployment
      • For bulk operations: throttle or reschedule them to off-peak times
      • For excessive Events: consider reducing --event-ttl on the API server (default 1h)
    • Expected result:
      • etcd growth rate returns to baseline
    • additional info:
      • If the growth cannot be stopped immediately, consider temporarily increasing --quota-backend-bytes (max 8 GiB) to buy time

Additional resources