Alert Runbooks
Alert RunbooksAlert runbooksalerts-k8s-exporterEtcdDatabaseHighFragmentationRatio

EtcdDatabaseHighFragmentationRatio

EtcdDatabaseHighFragmentationRatio

Description

This alert fires when the etcd database has a high ratio of fragmented space relative to its total allocated size.
Fragmentation builds up over time as keys are deleted or overwritten — etcd retains the allocated disk space but leaves gaps that are no longer in use. A high fragmentation ratio means etcd is holding significantly more disk space than the actual data requires, which wastes storage and can push the database toward its quota limit unnecessarily.

Possible Causes:

Severity estimation

Low to Medium severity — etcd continues to function normally, but left unattended this leads to quota exhaustion.

Defragmentation is a routine maintenance operation and should be performed without urgency unless quota pressure is also present.

Troubleshooting steps

  1. Check current etcd database size and fragmentation

    • Command / Action:
      • Query etcd metrics to see in-use vs allocated size

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=table

    • Expected result:
      • Output shows DB SIZE (allocated) and DB SIZE IN USE columns; the gap between them is fragmented space
    • additional info:
      • Set ETCDCTL_API=3 and pass --cacert, --cert, --key flags if required by your cluster

  1. Verify compaction is running

    • Command / Action:
      • Check the etcd pod logs for compaction activity

      • kubectl logs -n kube-system <etcd-pod> | grep -i compact

    • Expected result:
      • Regular compaction log entries are visible
    • additional info:
      • Kubernetes runs auto-compaction via kube-apiserver --etcd-compaction-interval (default 5 min); if missing, compaction may be disabled

  1. Check the Prometheus metric directly

    • Command / Action:
      • Evaluate the fragmentation ratio in Prometheus

      • (1 - etcd_mvcc_db_total_size_in_use_in_bytes / etcd_mvcc_db_total_size_in_bytes) * 100

    • Expected result:
      • Value below ~50%; alert typically fires above that threshold
    • additional info:
      • Run per member (by (endpoint)) to identify if fragmentation is isolated to one etcd member

  1. Run defragmentation on each etcd member

    • Command / Action:
      • Defragment one member at a time to avoid quorum loss

      • kubectl exec -n kube-system <etcd-pod> – etcdctl defrag –endpoints=https://127.0.0.1:2379

      • kubectl exec -n kube-system <etcd-pod> – etcdctl defrag –endpoints=<member-2-url>,<member-3-url>

    • Expected result:
      • Each member reports Finished defragmenting etcd member[...]; DB SIZE drops after each run
    • additional info:
      • Defragmentation briefly locks the member — always defragment one member at a time and confirm it returns healthy before proceeding to the next

  1. Verify etcd cluster health after defragmentation

    • Command / Action:
      • Confirm all members are healthy and the leader is stable

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint health –write-out=table

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=table

    • Expected result:
      • All endpoints report healthy: true; DB SIZE IN USE is now close to DB SIZE
    • additional info:
      • If a member remains unhealthy after defrag, check its logs and cross-reference with EtcdMembersDown or EtcdNoLeader

  1. Confirm the alert resolves

    • Command / Action:
      • Re-check the fragmentation ratio metric after defragmentation completes

      • (1 - etcd_mvcc_db_total_size_in_use_in_bytes / etcd_mvcc_db_total_size_in_bytes) * 100

    • Expected result:
      • Ratio drops below the alert threshold
    • additional info:
      • If fragmentation returns quickly, investigate object churn — high Pod/Event/Job turnover may require tuning --etcd-compaction-interval or --auto-compaction-retention

Additional resources