Alert Runbooks

KubeVersionMismatch

KubeVersionMismatch

Description

Different semantic versions of Kubernetes components running. Usually happens during kubernetes cluster upgrade process. Kubernetes control plane nodes or worker nodes use different versions.


Possible Causes:


Severity estimation

Incompatible API versions between kubernetes components may have very broad range of issues, influencing single containers, through app stability, ending at whole cluster stability.


Troubleshooting steps

  1. Check version

    • Command / Action:
      • in rancher, after selecting the cluster, open kubectl shell with the button in the top right corner
      • kubectl get nodes
        
    • Expected result:
      • a list of nodes with the same version
      • kubectl get nodes
        NAME                         STATUS   ROLES                       AGE    VERSION
        elk-alloy-eu-hc-fsn1-01      Ready    worker                      476d   v1.33.1+rke2r1
        elk-alloy-eu-hc-hel1-01      Ready    worker                      476d   v1.33.1+rke2r1
        elk-cp-eu-hc-fsn1-01         Ready    control-plane,etcd,master   476d   v1.33.1+rke2r1
        ...
        elk-warm-eu-hz-hel1-07       Ready    worker                      376d   v1.33.1+rke2r1
        elk-warm-eu-hz-hel1-08       Ready    worker                      376d   v1.33.1+rke2r1
        elkdev-hot-eu-hz-hel1-01     Ready    worker                      390d   v1.33.1+rke2r1
        elkdev-hot-eu-hz-hel1-02     Ready    worker                      390d   v1.33.1+rke2r1
        
  2. deploy same version

    • Command / Action:
      • tbd
      • tbd
        
    • Expected result:
      • nodes all the same version
      • kubectl get nodes
        NAME                         STATUS   ROLES                       AGE    VERSION
        elk-alloy-eu-hc-fsn1-01      Ready    worker                      476d   v1.33.1+rke2r1
        elk-alloy-eu-hc-hel1-01      Ready    worker                      476d   v1.33.1+rke2r1
        elk-cp-eu-hc-fsn1-01         Ready    control-plane,etcd,master   476d   v1.33.1+rke2r1
        ...
        elk-warm-eu-hz-hel1-07       Ready    worker                      376d   v1.33.1+rke2r1
        elk-warm-eu-hz-hel1-08       Ready    worker                      376d   v1.33.1+rke2r1
        elkdev-hot-eu-hz-hel1-01     Ready    worker                      390d   v1.33.1+rke2r1
        elkdev-hot-eu-hz-hel1-02     Ready    worker                      390d   v1.33.1+rke2r1
        

Additional resources