KubeStatefulSetUpdateNotRolledOut

KubeStatefulSetUpdateNotRolledOut

Description

This alert fires when a Kubernetes StatefulSet update has not been fully rolled out and the observed revision does not match the desired update revision for longer than expected.
It indicates that one or more StatefulSet pods are blocking the update process, leaving the workload in a partially updated state.

StatefulSet updates are performed sequentially, so a single failing pod can prevent the entire rollout from completing.

Possible Causes:

• Pods failing to start or repeatedly crashing (CrashLoopBackOff)
• Failing readiness or liveness probes
• PersistentVolumeClaim (PVC) mount or permission issues
• Insufficient node resources (CPU, memory, disk)
• Pod stuck in Pending state
• Node failures or nodes in NotReady state
• Strict updateStrategy (e.g. OnDelete)
• PodDisruptionBudget blocking pod replacement
• Application-level startup or migration failures

Severity estimation

Medium to High severity, depending on workload criticality.

• Low if update delay is acceptable and workload is stable
• Medium if rollout delay blocks feature delivery or bug fixes
• High if update is required for stability or security
• Critical if the workload is degraded or partially unavailable

Severity increases with:

• Duration of the incomplete rollout
• Importance of the update (security, schema changes)
• Role of the blocking pod (leader, primary)

Troubleshooting steps

1. Check StatefulSet update status

   • Command / Action:
     • Inspect current and update revisions

     • kubectl get statefulset <statefulset-name> -n <namespace>

   • Expected result:
     • CURRENT and UPDATE revisions match
   • additional info:
     • A mismatch confirms rollout is incomplete

2. Describe the StatefulSet

   • Command / Action:
     • Review update strategy and events

     • kubectl describe statefulset <statefulset-name> -n <namespace>

   • Expected result:
     • Events show sequential pod updates
   • additional info:
     • Look for pods blocking the update

3. Inspect pods and ordinals

   • Command / Action:
     • List StatefulSet pods with their revisions

     • kubectl get pods -n <namespace> -l <statefulset-label> -o wide

   • Expected result:
     • Pods are updated in reverse ordinal order
   • additional info:
     • The first failing pod blocks the rest

4. Describe the blocking pod

   • Command / Action:
     • Inspect pod status, events, and conditions

     • kubectl describe pod <pod-name> -n <namespace>

   • Expected result:
     • Pod becomes Running and Ready
   • additional info:
     • PVC or probe failures are common blockers

5. Check pod logs

   • Command / Action:
     • Review application logs

     • kubectl logs <pod-name> -n <namespace>

   • Expected result:
     • Application starts without fatal errors
   • additional info:
     • Use --previous if the pod restarted during update

6. Verify PVCs

   • Command / Action:
     • Ensure PVCs are bound and accessible

     • kubectl get pvc -n <namespace>

   • Expected result:
     • All PVCs are Bound
   • additional info:
     • Storage issues will block StatefulSet updates

7. Check update strategy

   • Command / Action:
     • Inspect update strategy configuration

     • kubectl get statefulset <statefulset-name> -n <namespace> -o yaml

   • Expected result:
     • RollingUpdate strategy allows progress
   • additional info:
     • OnDelete requires manual pod deletion

8. Force update after fixing root cause

   • Command / Action:
     • Delete the blocking pod to retry update

     • kubectl delete pod <pod-name> -n <namespace>

   • Expected result:
     • Pod is recreated with the new revision
   • additional info:
     • Always resolve the underlying issue first

Additional resources

• Kubernetes StatefulSet documentation
• Kubernetes rolling updates for StatefulSets
• Kubernetes Pod lifecycle and troubleshooting
• Related alert: KubeStatefulSetReplicaMismatch
• Related alert: KubePodNotReady