KubeDeploymentGenerationMismatch

KubeDeploymentGenerationMismatch

Description

This alert fires when a Kubernetes Deployment’s desired generation does not match the observed generation.
It means a change was applied to the Deployment spec, but the Deployment controller has not yet reconciled it, usually indicating a stalled or failing rollout.

Possible Causes:

• Ongoing rollout that has not completed
• Pods failing to start or become Ready
• Image pull failures (ImagePullBackOff, ErrImagePull)
• Insufficient cluster resources (CPU, memory, quotas)
• Failing readiness or liveness probes
• Misconfigured Deployment spec (labels, selectors, volumes)
• Deployment is paused
• API server or controller delays

Severity estimation

Medium severity by default, increasing with impact and duration.

• Low if the mismatch is brief during a normal rollout
• Medium if it persists longer than expected
• High if the Deployment is critical or user-facing
• Critical if available replicas drop to zero

Severity increases if:

• The alert persists for several minutes
• Multiple Deployments are affected
• The Deployment serves production traffic

Troubleshooting steps

1. Check Deployment generation status

   • Command / Action:
     • Inspect desired vs observed generation

     • kubectl get deployment <deployment-name> -n <namespace>

   • Expected result:
     • GENERATION and OBSERVED GENERATION match
   • additional info:
     • A mismatch means the controller has not reconciled the latest spec

2. Describe the Deployment

   • Command / Action:
     • Review events and rollout status

     • kubectl describe deployment <deployment-name> -n <namespace>

   • Expected result:
     • Events show normal ReplicaSet creation and scaling
   • additional info:
     • Errors here often explain why reconciliation is blocked

3. Check rollout progress

   • Command / Action:
     • Inspect rollout state

     • kubectl rollout status deployment <deployment-name> -n <namespace>

   • Expected result:
     • Rollout completes successfully
   • additional info:
     • A stuck rollout usually correlates with this alert

4. Inspect ReplicaSets

   • Command / Action:
     • List ReplicaSets created by the Deployment

     • kubectl get rs -n <namespace>

   • Expected result:
     • New ReplicaSet scales up while old ones scale down
   • additional info:
     • A new ReplicaSet with 0 ready replicas indicates a problem

5. Inspect Pods

   • Command / Action:
     • List pods and check their status

     • kubectl get pods -n <namespace>

   • Expected result:
     • Pods are Running and Ready
   • additional info:
     • Investigate Pending, CrashLoopBackOff, or ImagePullBackOff

6. Check pod logs

   • Command / Action:
     • Review logs for failing pods

     • kubectl logs <pod-name> -n <namespace>

   • Expected result:
     • Application starts without repeated errors
   • additional info:
     • Use --previous for restarted containers

7. Verify Deployment is not paused

   • Command / Action:
     • Check paused status

     • kubectl get deployment <deployment-name> -n <namespace> -o jsonpath=’{.spec.paused}'

   • Expected result:
     • Output is false or empty
   • additional info:
     • Resume if needed:

       kubectl rollout resume deployment <deployment-name> -n <namespace>

8. Roll back if necessary

   • Command / Action:
     • Roll back to the previous stable revision

     • kubectl rollout undo deployment <deployment-name> -n <namespace>

   • Expected result:
     • Deployment stabilizes and generations match
   • additional info:
     • Use rollback if the new generation is broken

Additional resources

• Kubernetes Deployment documentation
• Kubernetes rollout status
• Related alert: KubeDeploymentReplicasMismatch
• Related alert: KubeDeploymentRolloutStuck