Runbook: AvailablePullWorkerInsufficient

Alert Details

• Alert Name: AvailablePullWorkerInsufficient
• Expression: max (pull_audit_new_jobs_queue) without (cluster, prometheus, prometheusK8sClusterProvider) > 0

Description

This alert triggers when the maximum value of the audit jobs queue exceeds a certain threshold, indicating that there are insufficient pull workers available to process the jobs.

Possible Causes

• Not enough pull workers running to handle the job load.
• High volume of audit jobs causing delays.
• Network issues affecting the pull workers.
• Configuration errors in the pull worker setup.

Troubleshooting Steps

1. Check Pull Worker Instances

   • Command: ps aux | grep pull-worker
   • Expected Output: List of running pull worker instances. Ensure there are enough instances running.
   • Example:

     $ ps aux | grep pull-worker
     root      1234  0.0  0.1  123456  1234 ?        Ssl  14:00   0:00 /usr/bin/pull-worker
     root      5678  0.0  0.1  123456  1234 ?        Ssl  14:00   0:00 /usr/bin/pull-worker
     

2. Restart Pull Worker Service

   • Command: sudo systemctl restart pull-worker
   • Expected Output: The service restarts without errors.
   • Example:

     $ sudo systemctl restart pull-worker
     

3. Check Network Connectivity

   • Command: ping -c 4 pull-worker-hostname
   • Expected Output: Successful ping responses.
   • Example:

     $ ping -c 4 pull-worker-hostname
     PING pull-worker-hostname (192.168.1.4) 56(84) bytes of data.
     64 bytes from pull-worker-hostname: icmp_seq=1 ttl=64 time=0.123 ms
     64 bytes from pull-worker-hostname: icmp_seq=2 ttl=64 time=0.124 ms
     64 bytes from pull-worker-hostname: icmp_seq=3 ttl=64 time=0.125 ms
     64 bytes from pull-worker-hostname: icmp_seq=4 ttl=64 time=0.126 ms
     

4. Verify Pull Worker Configuration

   • Command: cat /etc/pull-worker/config.yml
   • Expected Output: Configuration file contents. Ensure all settings are correct.
   • Example:

     $ cat /etc/pull-worker/config.yml
     job_name: 'pull-jobs'
     max_queue_age: 300
     

Additional Steps

• Check Logs for Errors

  • Command: journalctl -u pull-worker --since "1 hour ago"
  • Expected Output: Recent logs for the pull worker service. Look for any error messages.
  • Example:

    $ journalctl -u pull-worker --since "1 hour ago"
    -- Logs begin at Wed 2024-11-13 13:00:00 UTC, end at Wed 2024-11-13 14:00:00 UTC. --
    Nov 13 13:45:00 hostname pull-worker[1234]: Pull job processed
    Nov 13 13:50:00 hostname pull-worker[1234]: Error: Job processing delayed
    

• Check Underlying Infrastructure

  • Ensure the server hosting the pull worker service is up and running.
  • Verify there are no ongoing maintenance activities or outages.