ConfigReloaderSidecarErrors

ConfigReloaderSidecarErrors

Description

This alert fires when the config-reloader sidecar container is repeatedly failing to reload configuration for a monitored component (e.g. Prometheus, Alertmanager, or other Prometheus Operator-managed resources).
The config-reloader watches ConfigMaps and Secrets for changes and triggers a configuration reload in the main container. Errors here mean that the latest configuration changes are not being applied, which may cause the component to run with stale or incorrect configuration.

Possible Causes:

• Invalid or malformed configuration pushed to the watched ConfigMap or Secret
• The controlled app is rejecting the reload due to configuration syntax or validation errors
• The config-reloader cannot reach the app’s reload endpoint (network issue, wrong port, or app crash)
• Insufficient RBAC permissions for the sidecar to read the ConfigMap or Secret
• The main container is not responding to reload signals (SIGHUP or HTTP /-/reload)
• A recent change introduced a misconfiguration in a PrometheusRule, ServiceMonitor, or Alertmanager config

Severity estimation

Low to Medium severity on its own, but can escalate quickly.

• Low if the current running configuration is still valid and the component is healthy
• Medium if recent critical rule or alerting changes are not taking effect
• High if the main container has crashed or stopped serving traffic as a result

The main risk is a silent drift: the monitored component keeps running with the last valid config while new alert rules, scrape configs, or routing changes go unnoticed as unapplied.

Troubleshooting steps

1. Identify the affected component and namespace

   • Command / Action:
     • Check which pod is triggering the alert by inspecting the alert labels (namespace, pod, container)

     • kubectl get pods -n <namespace> | grep -E ‘prometheus|alertmanager’

   • Expected result:
     • You can identify the pod containing the config-reloader sidecar
   • additional info:
     • The sidecar is typically named config-reloader and runs alongside prometheus or alertmanager containers

2. Check config-reloader sidecar logs

   • Command / Action:
     • Inspect the sidecar’s log output for error messages

     • kubectl logs <pod-name> -n <namespace> -c config-reloader

     • kubectl logs <pod-name> -n <namespace> -c config-reloader –previous

   • Expected result:
     • Logs show successful reloads: "reloaded config file" or similar success messages
   • additional info:
     • Common error patterns: error reloading config, connection refused, permission denied

3. Check the main container logs for reload errors

   • Command / Action:
     • The main app may log the reason it rejected the config reload

     • kubectl logs <pod-name> -n <namespace> -c prometheus

     • kubectl logs <pod-name> -n <namespace> -c alertmanager

   • Expected result:
     • No configuration parse or validation errors in the main container logs
   • additional info:
     • Prometheus typically logs "Loading configuration file" on reload; look for lines like "Error loading config" or "invalid config"

4. Validate the configuration content

   • Command / Action:
     • Identify and inspect the ConfigMap or Secret being reloaded

     • kubectl get configmap -n <namespace> | grep prometheus

     • kubectl describe configmap <configmap-name> -n <namespace>

   • Expected result:
     • Configuration is syntactically valid YAML with no obvious errors
   • additional info:
     • For Prometheus, use promtool check config locally against the config file to validate before applying

5. Check RBAC permissions for the sidecar

   • Command / Action:
     • Verify the ServiceAccount has permission to read the relevant ConfigMap/Secret

     • kubectl auth can-i get configmaps -n <namespace> –as=system:serviceaccount:<namespace>:<serviceaccount-name>

   • Expected result:
     • Returns yes
   • additional info:
     • If no, inspect the ClusterRole/Role bound to the ServiceAccount and add the missing permissions

6. Verify the reload endpoint is reachable

   • Command / Action:
     • Confirm the main container’s reload HTTP endpoint is responding

     • kubectl exec <pod-name> -n <namespace> -c config-reloader – wget -qO- http://localhost:<port>/-/reload

   • Expected result:
     • Returns HTTP 200 or a success message
   • additional info:
     • Default reload ports: Prometheus 9090, Alertmanager 9093. If the main container is down, fix that first

7. Check recent changes to PrometheusRules or Alertmanager configs

   • Command / Action:
     • Look for recently modified PrometheusRule or AlertmanagerConfig custom resources

     • kubectl get prometheusrule -n <namespace>

     • kubectl describe prometheusrule <rule-name> -n <namespace>

   • Expected result:
     • All rules are valid and well-formed
   • additional info:
     • A broken PrometheusRule will cause all pending config reloads to fail until it is fixed or removed

8. Describe the pod for sidecar-level events

   • Command / Action:
     • Check pod events for OOMKill, probe failures, or restart signals on the sidecar

     • kubectl describe pod <pod-name> -n <namespace>

   • Expected result:
     • No abnormal events on the config-reloader container
   • additional info:
     • If the sidecar itself is crash-looping, treat this as a KubePodCrashLooping scenario for that container

Additional resources

• Prometheus Operator config-reloader documentation
• promtool check config reference
• Kubernetes debugging running pods
• Related alert: KubePodCrashLooping
• Related alert: KubePodNotReady