VtransOverload

Runbook: VtransOverload

Description

This alert triggers when a vtrans server has no more availbale slots for new streams.

Possible Causes:

• High load on the vtrans service because bulk ingest of customer
• Load balancing issue
• Configuration errors in the vtrans setup.

Severity estimation

If there are still servers which are not overloaded, this alert is not critical. Due to this alert the vtrans server will be removed from the blanacing and wont be available for new streams. Under usual circumstances it should not be necessary to do anything if this alert comes up. But there can be exceptions, f.e. if a high priority stream has issues or a certain server has issues which need to be resolved. Check out this Dashboard to get an vtrans load overview for all regions and make sure the dashboard filter are set to the desired values.

Troubleshooting steps

1. Log into server

   • create the fqdn of the server with the following scheme Streamcloud server naming and log in to the server

2. Check server load

   • Grafana dashboard: select the wanted server with the Host filter
   • Command on server:
   • if avg server CPU load is >90% or you can, proceed with troubleshootig steps

   • htop

3. Manage streams via vtrans web console

   • Endpoint: <vtrans-host>/apps/console , credentials are saved in 1password under nano-pull-helper
   • under the process tab streams can be selected and stopped

Additional resources

Vtrans Capacity Dashboard Streamcloud server naming todo: streamcloud balancing runbook todo : streamcloud load estimation dashboarad