Alert Runbooks
Alert RunbooksAlert runbooksalerts-k8s-exporterEtcdHighNumberOfFailedGRPCRequests

EtcdHighNumberOfFailedGRPCRequests

EtcdHighNumberOfFailedGRPCRequests

Description

This alert fires when the rate of failed gRPC requests to etcd exceeds the configured threshold. etcd exposes its entire API over gRPC, so a high failure rate means that clients — most critically the Kubernetes API server — are receiving error responses instead of successful reads or writes. Sustained failures degrade cluster state management and can cause cascading control plane issues.

Possible Causes:

Severity estimation

Medium to High — depends on which gRPC methods are failing and the failure rate.

Troubleshooting steps

  1. Identify which gRPC methods are failing and their error codes

    • Command / Action:
      • Query Prometheus to see which methods have the highest failure rate and what error codes are returned

      • sum by (grpc_method, grpc_code) (rate(grpc_server_handled_total{grpc_service=“etcdserverpb.KV”, grpc_code!=“OK”}[5m]))

    • Expected result:
      • No methods should have a significant non-OK rate; a zero or near-zero result is healthy
    • additional info:
      • Common gRPC error codes and their meaning:
        • ResourceExhausted — quota exhausted or server overloaded, refer to EtcdDatabaseQuotaLowSpace
        • Unavailable — leader election in progress or member temporarily unreachable
        • Unauthenticated — TLS certificate or token issue
        • InvalidArgument — malformed request from the client

  1. Check etcd pod logs for error messages

    • Command / Action:
      • Look for recurring errors that correspond to the failing gRPC methods

      • kubectl logs -n kube-system <etcd-pod> –tail=200 | grep -E ’error|failed|quota|space|exceeded|auth|certificate'

    • Expected result:
      • No recurring error patterns; occasional transient errors are acceptable
    • additional info:
      • Key log patterns to look for:
        • mvcc: database space exceeded — storage quota hit, refer to EtcdDatabaseQuotaLowSpace
        • certificate has expired — TLS cert renewal is required
        • request is too large — a client is sending oversized payloads
        • slow fdatasync or took too long — disk latency is causing request timeouts

  1. Check etcd cluster health and leader status

    • Command / Action:
      • Verify the cluster has a healthy leader and all members are reachable

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint health –cluster –write-out=table

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –cluster –write-out=table

    • Expected result:
      • All members report healthy: true; exactly one member shows IS LEADER: true
    • additional info:
      • If no leader is present, gRPC write failures are expected until election completes — refer to EtcdNoLeader
      • If a member is unreachable, clients directed to that member will see elevated failures — refer to EtcdMembersDown

  1. Check whether the etcd storage quota is exhausted

    • Command / Action:
      • Query the database size and the configured quota

      • kubectl exec -n kube-system <etcd-pod> – etcdctl endpoint status –write-out=table

      • Check the DB SIZE column against the configured --quota-backend-bytes (default: 2 GiB)
    • Expected result:
      • DB SIZE is significantly below the quota limit (typically below 80%)
    • additional info:
      • If the quota is exhausted, etcd enters a read-only alarm state and rejects all writes; run etcdctl alarm list to confirm
      • Refer to EtcdDatabaseQuotaLowSpace and EtcdDatabaseHighFragmentationRatio for remediation steps

  1. Check etcd TLS certificates for expiry

    • Command / Action:
      • Inspect the expiry date of etcd’s server and peer certificates

      • kubectl exec -n kube-system <etcd-pod> – sh -c “openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -dates”

      • kubectl exec -n kube-system <etcd-pod> – sh -c “openssl x509 -in /etc/kubernetes/pki/etcd/peer.crt -noout -dates”

    • Expected result:
      • Both certificates have a notAfter date well in the future
    • additional info:
      • Expired certificates cause Unauthenticated gRPC errors on all requests; certificate rotation is required
      • In RKE2/K3s clusters, certs are located at /var/lib/rancher/rke2/server/tls/etcd/

  1. Check etcd server resource usage

    • Command / Action:
      • Verify etcd is not being CPU-throttled or hitting memory limits

      • kubectl top pod -n kube-system -l component=etcd

      • kubectl describe pod -n kube-system <etcd-pod>

    • Expected result:
      • CPU and memory usage are within normal bounds; no OOMKill events in the pod events
    • additional info:
      • etcd is typically CPU-light but can spike during compaction or large watch fan-outs
      • If resource limits are set on the etcd pod and CPU is being throttled, consider relaxing them — etcd is a critical component and should not be CPU-limited in production

  1. Verify the failure rate drops after addressing the root cause

    • Command / Action:
      • Re-run the Prometheus query to confirm the failure rate has returned to baseline

      • sum by (grpc_method, grpc_code) (rate(grpc_server_handled_total{grpc_service=“etcdserverpb.KV”, grpc_code!=“OK”}[5m]))

    • Expected result:
      • All gRPC methods return predominantly OK; non-OK rate is at or near zero
    • additional info:
      • Also check that the Kubernetes API server is responding normally: kubectl get nodes and kubectl get pods -A should return quickly without timeout errors

Additional resources