Common support issues

Important Note

Before you start, take note that the main purpose of this guide is to help customers get back to production as soon as possible. Investigations into the root cause can take place after.
As usual, we will need the following information beforehand :

• Description of the issue.
• Stream name/s of streams involved.
• Time of issue.

Common issues reported by customers

Stream is buffering, lagging, stuttering, etc.

Asking the customer to republish the stream/s would be a first step.
If customer is using OBS for ingesting, we can advise the customer to do a full shutdown and restart of OBS too.
If republishing does not help, then we need to determine if the issue
is on the ingest side or the playout side.

• Ingest Check

  • Using Grafana
    • Open up grafana rtmpstats : https://grafana.nanocosmos.cloud/goto/X-TXp2lNg?orgId=1.
    • Set the orga, stream and time interval (in UTC time) according to the issue description.
    • Check the panels for indications of ingest related degradation.
      • Minimum Stream Time Ratio
        • Small fluctuations are alright.
        • If it is consistently higher than 1.0 (overflow) or consistenly lower than 1.0 (underflow), then possibly a first mile degradation.
      • Stream Uptime
        • Make sure that the stream is ingesting actively.
      • Max Stream Ingest Delay by DC
        • Fluctuations or high delays can indicate first mile or DC issue.
        • If the reported issue occurs with streams ingesting into the same DC and at the same time showing fluctuations, this may indicate a specific DC issue. This may require providing new e.g. GLB URLs to customers to bypass the DC having issue.
      • Video Frame Length Deviation
        • Video difference of shortest and longest frame lengths/gaps. Can indicate unstable framerate and broadcaster side frame drop.
        • Ideally, it should be 0ms or near to 0ms.
      • Video Max Time Offset
        • Video max absolute balance of summed diff between timestamp and realtime progress. Can detect short term fluctuations as well as lasting underflows.
        • Ideally, it should be 0ms or near to 0ms.
      • Stream Codec
        • List the stream info of passthrough streams.
        • Frame Rate is the metadata set in the stream config while Realtime Frame Rate is the actual frame rate. If there is a big difference between both, it can mean that the ingest datarate is insufficient.
  • Ingest Webhook Kibana check
    • https://kibana.elasticsearch.fsn.hz.k8s.nanostream.cloud/goto/b7ef27a0-3688-11ee-82d6-f7772e3ae83b
    • Set the bintu.orgahash.keyword filter and time interval (in UTC time) according to the issue description. Optionally, edit the 2xx, 3xx, 4xx, 5xx filters to see the results list.
    • The column webhook.stat.response_code shows the return code and if it is not 200, then bintu will reject the ingest.
  • Encoder/Streamer setup
    • Request customer to share their encoder/streamer setup and double-check the parameters, e.g H.264 encoding and not H.265, AAC audio format, etc.
  • Try a back up ISP, if available or prepare for 1 if not.
  • Get MTRs
    • mtr -s 1000 -r -c 200 bintu-vtrans.nanocosmos.de or mtr -s 1000 -r -c 200 bintu.nanocosmos.de (if non-vtrans).
    • Check for significant packet losses > 5%.
  • Confirm that stream groups/daisy chain streams ingest into vtrans infrastructure, else ABR will not work.
  • Duplicate ingests.
    • Open up grafana rtmpstats and set up as described earlier.
    • Check Active Ingest by Host and make sure no 2 servers are active at any time for a stream.
  • Try ingest using SRT, if it is available.
    • See this guide : https://docs.nanocosmos.de/docs/cloud/srt_ingest

• Playout Check

  • Last mile bandwidth check

    • If the ingest is to non-vtrans infrastructure, then the passthrough bitrate will be the client playout bitrate. A speed test can be a gauge of the sustainable bandwidth for the client connection.
    • If the ingest is a stream group or daisy-chained vtrans stream, then the customer can test with the lowest transcode profile playback to see if it is a bandwidth issue.

  • Encoder specific issues

    • Roland
      • Muxer AVC1 packetization anomaly.
    • Teradek
      • Set encoder H.264 Number of slices to 1.

  • Secure playout check.

Token with referrers are not supported and rejected for secured RTMP playback

Setup as:
- Domain: bintu-splay.nanocosmos.de
- RTMP application name: splayrtmp

The secure parameters have to be appended to the RTMP stream name.

RTMP URL:
rtmp://bintu-splay.nanocosmos.de/splayrtmp/'Stream name'

Stream name:
XXXXX-YYYYY?expires=[expires]&tag=[tag]&token=[token]&options=[options]

Secure stream playback fails.

• Could not generate secure tokens.
  • ( how to check if bintu service is under attack or other issues ? )
• Token expiry check using Kibana.
  • https://kibana.elasticsearch.fsn.hz.k8s.nanostream.cloud/goto/47783de0-d08e-11ee-9fc1-513fec7c4744
  • Set the bintu.orgahash.keyword filter and time interval (in UTC time) according to the issue description.
  • Check listing, if any.
• CTS APM check.
  • https://kibana.elasticsearch.fsn.hz.k8s.nanostream.cloud/app/apm/services/CTS/overview?comparisonEnabled=false&comparisonType=day&environment=production-eu-a&kuery=transaction.duration.us%20%3E%201000000&rangeFrom=now-12h&rangeTo=now&transactionType=request&latencyAggregationType=p99
  • See if 99th Percentile of “latency > 5 seconds” is significant or not during the current time period. This can point to CTS infrastructure degradation.

Live replay is not working, e.g. assets are not generated.

• Did the customer republish the stream, after a live replay feature setting has been applied ?
• ( how to check if linode E3 or aws S3 service is running properly ? )