KubeClientCertificateExpiration

Description

A client certificate used to authenticate requests to the Kubernetes API server is expiring soon or has expired.

Client certificates are used by various Kubernetes components (kubelet, kube-proxy, controllers, operators) and external clients to authenticate to the API server. When these certificates expire, the affected components will lose the ability to communicate with the API server, potentially causing service disruptions.

Possible Causes:

Certificate renewal process not configured or not working properly
Expired or near-expiry client certificates used by cluster components
Certificate rotation failed or was not completed
Manual certificates that were not renewed in time
Certificate auto-renewal disabled or misconfigured
Clock skew issues causing premature expiration detection

Severity estimation

High to Critical severity, depending on which client certificate is expiring.

High if the certificate is expiring within 30 days and affects non-critical components
Critical if the certificate is expiring within 7 days or has already expired
Critical if the certificate affects core components (kubelet, controller-manager, scheduler)

Impact assessment:

Expired certificates will cause immediate authentication failures
Components using expired certificates cannot communicate with the API server
Can lead to pod scheduling failures, network policy issues, or complete cluster dysfunction
May affect monitoring, logging, and other cluster operators

Troubleshooting steps

Identify which client certificates are expiring
- Command / Action:
  - Check alert labels to identify the affected certificate
  - Review certificate expiration dates across the cluster
  - kubectl get csr
  - kubeadm certs check-expiration
- Expected result:
  - List of certificates with expiration dates
  - Identification of certificates expiring soon or expired
- additional info:
  - The alert should provide details about which certificate is affected
  - Focus on certificates with less than 30 days remaining

Check certificate details
- Command / Action:
  - Examine specific certificate files to verify expiration
  - openssl x509 -in /path/to/certificate.crt -noout -dates
  - openssl x509 -in /path/to/certificate.crt -noout -text | grep -A2 Validity
- Expected result:
  - Certificate expiration date and time displayed
  - Confirmation of which certificates need renewal
- additional info:
  - Common certificate locations: /etc/kubernetes/pki/, /var/lib/kubelet/pki/
  - Check both client certificates and their corresponding CA certificates

Verify certificate auto-renewal status
- Command / Action:
  - Check if automatic certificate rotation is enabled
  - For kubelet certificates, check kubelet configuration
  - kubectl get node -o yaml | grep rotateCertificates
  - systemctl status kubelet
- Expected result:
  - Certificate rotation should be enabled
  - No errors in kubelet logs related to certificate renewal
- additional info:
  - Kubelet can automatically rotate its certificates if configured with --rotate-certificates=true

Review pending Certificate Signing Requests (CSRs)
- Command / Action:
  - Check for pending CSRs that may need approval
  - kubectl get csr
  - kubectl get csr -o json | jq ‘.items[] | select(.status.conditions == null) | .metadata.name’
- Expected result:
  - Pending CSRs should be reviewed and approved if legitimate
- additional info:
  - Unapproved CSRs can prevent certificate renewal
  - Approve valid CSRs: kubectl certificate approve <csr-name>

Renew certificates using kubeadm (for kubeadm clusters)
- Command / Action:
  - Renew all certificates managed by kubeadm
  - kubeadm certs renew all
  - Restart affected components after renewal
  - systemctl restart kubelet
- Expected result:
  - Certificates renewed successfully
  - New expiration dates set to 1 year from renewal
- additional info:
  - This applies to clusters bootstrapped with kubeadm
  - Backup existing certificates before renewal
  - Some certificates may require control plane component restarts

Manually renew certificates (for non-kubeadm clusters)
- Command / Action:
  - Generate new certificate signing request
  - Sign with cluster CA
  - Replace old certificate with new one
  - Restart affected components
- Expected result:
  - New certificates installed and validated
  - Components successfully authenticate to API server
- additional info:
  - Follow your cluster’s certificate management procedures
  - Test connectivity after certificate replacement

Verify certificate renewal
- Command / Action:
  - Confirm new certificates are in place and valid
  - Check component connectivity to API server
  - kubeadm certs check-expiration
  - kubectl get nodes (to verify kubelet connectivity)
- Expected result:
  - All certificates have extended expiration dates
  - No authentication errors in component logs
- additional info:
  - Monitor for any authentication failures after renewal
  - Verify all cluster components are functioning normally

Implement certificate monitoring and auto-renewal
- Command / Action:
  - Enable automatic certificate rotation for kubelet
  - Set up automated certificate management tools
  - Configure cert-manager for certificate lifecycle management
- Expected result:
  - Certificates renew automatically before expiration
  - Proactive monitoring alerts for upcoming expirations
- additional info:
  - Consider using cert-manager for automated certificate management
  - Schedule regular certificate expiration audits
  - Document certificate renewal procedures

Additional resources

Kubernetes PKI certificates and requirements
Certificate rotation for the kubelet
Kubeadm certificate management
cert-manager documentation
Related alert: KubeletCertificateExpiration