EtcdDatabaseQuotaLowSpace

EtcdDatabaseQuotaLowSpace

Description

This alert fires when the etcd database is approaching its storage quota limit.
etcd enforces a hard quota on its database size (default 2 GiB, configurable up to 8 GiB). Once the quota is exceeded, etcd switches to a read-only alarm state and rejects all write requests, which causes the Kubernetes API server to stop accepting any changes to cluster state — no new Pods, ConfigMaps, Secrets, or any other resource can be created or updated.

Possible Causes:

• Excessive accumulation of etcd revisions due to infrequent or disabled compaction
• High object churn (frequent creates/deletes of Pods, Jobs, Events) inflating history
• Fragmented space not reclaimed — compaction ran but defragmentation was never performed
• Unusually large objects stored in etcd (e.g. large ConfigMaps or Secrets)
• A sudden bulk creation of Kubernetes resources (namespace migrations, large deployments)
• etcd quota set too low for the cluster’s workload

Severity estimation

High severity — this is a leading indicator of a full etcd outage.

• High if the database is approaching quota but writes still succeed
• Critical if quota is exceeded — etcd enters alarm state and the Kubernetes control plane stops accepting writes

Act promptly: once etcd hits quota, recovery requires defragmentation AND manually disarming the quota alarm before writes resume.

Troubleshooting steps

1. Check current database size vs quota

   • Command / Action:
     • Query endpoint status to see how close the DB is to the quota

     • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=table

   • Expected result:
     • DB SIZE is well below the configured quota (default 2 GiB)
   • additional info:
     • Check DB SIZE IN USE vs DB SIZE — a large gap means fragmentation is contributing; address that with defragmentation first

2. Check if etcd has already entered alarm state

   • Command / Action:
     • List active etcd alarms

     • kubectl exec -n kube-system <etcd-pod> – etcdctl alarm list

   • Expected result:
     • No alarms listed; if NOSPACE appears, etcd is already read-only
   • additional info:
     • If NOSPACE alarm is active, writes to the Kubernetes API are already failing — treat this as a critical incident

3. Run compaction to remove old revisions

   • Command / Action:
     • Get the current revision and compact up to it

     • REVISION=$(kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=json | jq ‘.[0].Status.header.revision’)

     • kubectl exec -n kube-system <etcd-pod> – etcdctl compact $REVISION

   • Expected result:
     • Compaction completes without error: compacted revision <N>
   • additional info:
     • Compaction removes old revision history but does not reclaim disk space yet — defragmentation must follow

4. Defragment each etcd member to reclaim space

   • Command / Action:
     • Defragment one member at a time to avoid quorum loss

     • kubectl exec -n kube-system <etcd-pod> – etcdctl defrag –endpoints=https://127.0.0.1:2379

     • Repeat for each member endpoint sequentially
   • Expected result:
     • Finished defragmenting etcd member[...]; DB SIZE drops significantly after each run
   • additional info:
     • Always confirm the member is healthy before defragmenting the next one

5. Disarm the NOSPACE alarm if it was triggered

   • Command / Action:
     • After defragmentation has freed enough space, clear the alarm

     • kubectl exec -n kube-system <etcd-pod> – etcdctl alarm disarm

   • Expected result:
     • No alarms remain; Kubernetes API server resumes accepting writes
   • additional info:
     • Do not disarm the alarm before defragmentation — etcd will immediately re-trigger it if the DB is still over quota

6. Verify etcd cluster health

   • Command / Action:
     • Confirm all members are healthy and writes are accepted

     • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint health –write-out=table

   • Expected result:
     • All endpoints report healthy: true and DB SIZE is well within quota
   • additional info:
     • Test a write via kubectl (e.g. kubectl create configmap test-cm --from-literal=key=val -n default) to confirm the API server is functional

7. Address root cause to prevent recurrence

   • Command / Action:
     • Review compaction and quota settings

     • kubectl get pod -n kube-system <apiserver-pod> -o yaml | grep -E ‘compaction|quota’

   • Expected result:
     • --etcd-compaction-interval is set (default 5m) and --quota-backend-bytes is appropriate for the cluster size
   • additional info:
     • If high object churn is the root cause, consider increasing the quota (--quota-backend-bytes, max 8589934592 = 8 GiB) or reducing event/job retention

Additional resources

• etcd maintenance — compaction and defragmentation
• etcd storage quota documentation
• Kubernetes etcd maintenance guide
• Related alert: EtcdDatabaseHighFragmentationRatio
• Related alert: EtcdExcessiveDatabaseGrowth
• Related alert: EtcdNoLeader