KubeQuotaAlmostFull

KubeQuotaAlmostFull

Description

This alert fires when a Kubernetes ResourceQuota in a namespace is approaching its configured limit, typically at or above 90% utilization for one or more tracked resources (CPU, memory, pods, services, PVCs, etc.).

A ResourceQuota approaching its limit means new workloads may soon be rejected, causing deployment failures or pod scheduling issues even before the quota is fully exhausted.

Possible Causes:

• Rapid growth in the number of pods, services, or other objects within the namespace
• Resource requests or limits set too high relative to the quota
• Accumulated unused or stale objects (completed jobs, unused ConfigMaps/Secrets, old PVCs)
• Misconfigured quota thresholds that are too low for the workload
• Namespace quota not updated to reflect increased workload requirements
• Burst in workload replicas due to autoscaling or manual scaling events

Severity estimation

Medium severity — the quota is not yet exceeded, but action is required to prevent imminent failures:

• Low: Single resource type approaching the limit with other resources well within bounds
• Medium: Multiple resource types near the limit, deployments or scaling may fail soon
• High: Quota almost full across critical resources (CPU, memory, pods); new workloads actively failing to schedule
• Critical: Combined with active deployment failures or pod evictions

Severity increases with:

• Proximity to the quota limit (95%+ is more urgent than 90%)
• Number of resource types affected
• Rate at which consumption is growing
• Criticality of the namespace and its workloads

Troubleshooting steps

1. Identify which namespace and resource type is affected

   • Command / Action:
     • Check the alert labels for the namespace and resource, then inspect quota usage

     • kubectl get resourcequota -n <namespace>

   • Expected result:
     • The specific quota object and the resource type approaching its limit are identified
   • additional info:
     • Multiple ResourceQuotas can exist in a namespace; check all of them

2. Describe the quota for detailed usage breakdown

   • Command / Action:
     • Get a full view of used vs. hard limits for each tracked resource

     • kubectl describe resourcequota <quota-name> -n <namespace>

   • Expected result:
     • A table showing used and hard values per resource type, identifying which is near the limit
   • additional info:
     • Pay attention to requests.cpu, requests.memory, limits.cpu, limits.memory, pods, services, persistentvolumeclaims

3. Identify the largest resource consumers in the namespace

   • Command / Action:
     • List pods and their resource requests/limits to find top consumers

     • kubectl top pod -n <namespace> –sort-by=cpu

     • kubectl top pod -n <namespace> –sort-by=memory

   • Expected result:
     • A ranked list of pods by resource usage, identifying potential candidates for optimization
   • additional info:
     • Cross-reference with kubectl get pods -n <namespace> to identify stale or completed pods still consuming quota

4. Check for and clean up stale or completed objects

   • Command / Action:
     • Remove completed jobs, succeeded/failed pods, and unused resources to free up quota

     • kubectl get pods -n <namespace> –field-selector=status.phase=Succeeded

     • kubectl delete pod –field-selector=status.phase=Succeeded -n <namespace>

     • kubectl get jobs -n <namespace>

   • Expected result:
     • Quota usage drops after removing objects that no longer serve a purpose
   • additional info:
     • Also check for orphaned PVCs, unused ConfigMaps, and old Secrets

5. Review and adjust resource requests/limits on workloads

   • Command / Action:
     • Identify workloads with oversized requests or limits relative to actual usage

     • kubectl describe pod <pod-name> -n <namespace>

     • kubectl set resources deployment <deployment-name> –requests=cpu=<value>,memory=<value> -n <namespace>

   • Expected result:
     • Reduced total requested resources, freeing quota for other workloads
   • additional info:
     • Use kubectl top pod to compare actual usage against configured requests/limits before reducing them

6. Increase the ResourceQuota limit if capacity is available

   • Command / Action:
     • Edit the ResourceQuota if the namespace legitimately requires more resources and cluster capacity allows

     • kubectl edit resourcequota <quota-name> -n <namespace>

   • Expected result:
     • The hard limit for the affected resource type is increased; the alert clears
   • additional info:
     • Confirm available cluster capacity before raising the quota: kubectl describe nodes | grep -A5 "Allocated resources"
     • Coordinate with the team responsible for the namespace before modifying quotas

7. Check if autoscaling is driving consumption growth

   • Command / Action:
     • Inspect HPA or KEDA configurations that may be scaling workloads aggressively

     • kubectl get hpa -n <namespace>

   • Expected result:
     • Autoscaler is configured with max replicas appropriate for the namespace quota
   • additional info:
     • If HPA max replicas would exceed quota when fully scaled, either increase the quota or lower the max replicas

8. Monitor quota consumption trend

   • Command / Action:
     • Query Prometheus to understand how quickly quota consumption is growing

     • kube_resourcequota{namespace="<namespace>", type=“used”}

   • Expected result:
     • A stable or slowly growing usage trend, not a rapid spike
   • additional info:
     • A rapid increase in consumption may indicate a misconfigured autoscaler, a runaway process, or a deployment error

Additional resources

• Kubernetes Resource Quotas
• Managing Resources for Containers
• Kubernetes LimitRange
• Related alert: KubeQuotaExceeded
• Related alert: KubeCPUQuotaOvercommit
• Related alert: KubeMemoryQuotaOvercommit