Stream related common issues

Important Note

Before you start, take a note that the main purpose of this guide is to help the customers to get back to production as soon as possible. Root cause Investigations can be handled later.
As usual, we will need the following info beforehand :

Description of the issue.
Stream name/s of streams involved.
Time of issue.

Common issues reported by customers

Stream is buffering, lagging, stuttering, etc.

Asking the customer to republish the stream/s would be a first step. \
If customer is using OBS for ingesting, we can advise the customer to do a full shutdown and restart of OBS too. \
If republishing does not help, then we need to determine if the issue
is on the ingest side or the playout side.
Ingest Check
- Using Grafana
  - Open up grafana rtmpstats : https://grafana.nanocosmos.cloud/d/TqDDfDE7k/nginx-rtmpstat?orgId=1&from=now-1h&to=now&var-orga=XpjSI&var-stream=all. Here for eg the orga is set as “XpjSI” and stream to “all” which can be changed.
  - Set the orga, stream and time interval (in UTC time) according to the issue description.
  - Check the panels for indications of ingest related degradation.
    - Minimum Stream Time Ratio
      - Small fluctuations are alright.
      - If it is consistently higher than 1.0 (overflow) or consistenly lower than 1.0 (underflow), then possibly a first mile degradation.
    - Stream Uptime
      - Make sure that the stream is ingesting actively.
    - Max Stream Ingest Delay by DC
      - Fluctuations or high delays can indicate first mile or DC issue.
      - If the reported issue occurs with streams ingesting into the same DC and at the same time showing fluctuations, this may indicate a specific DC issue. This may require providing new e.g. GLB URLs to customers to bypass the DC having issue.
    - Video Frame Length Deviation
      - Video difference of shortest and longest frame lengths/gaps. Can indicate unstable framerate and broadcaster side frame drop.
      - Ideally, it should be near to 0ms.
    - Video Max Time Offset
      - Video max absolute balance of summed diff between timestamp and realtime progress. Can detect short term fluctuations as well as lasting underflows.
      - Should be < 500ms. Spikes of more than that is ok but must not be sustained for longer than a few minutes.
    - Stream Codec
      - Video must be H264 format and audio must be AAC format
      - List the stream info of passthrough streams.
      - Frame Rate is the metadata set in the stream config while Realtime Frame Rate is the actual frame rate. If there is a big difference between both, it can mean that the ingest datarate is insufficient.
- Ingest Webhook Kibana check
  - https://kibana.elasticsearch.fsn.hz.k8s.nanostream.cloud/goto/b7ef27a0-3688-11ee-82d6-f7772e3ae83b
  - Set the bintu.orgahash.keyword filter and time interval (in UTC time) according to the issue description. Optionally, edit the 2xx, 3xx, 4xx, 5xx filters to see the results list.
  - The column webhook.stat.response_code shows the return code and if it is not 200, then bintu will reject the ingest.
- Encoder/Streamer setup
  - Request customer to share their encoder/streamer setup and double-check the parameters, e.g H.264 encoding and not H.265, AAC audio format, etc.
- Try a back up ISP, if available or prepare for 1 if not.
- Get MTRs
  - mtr -s 1000 -r -c 200 bintu-vtrans.nanocosmos.de or mtr -s 1000 -r -c 200 bintu.nanocosmos.de (if non-vtrans).
  - Check for significant packet losses > 5%.
- Confirm that stream groups/daisy chain streams ingest into vtrans infrastructure, else ABR will not work.
  - Open up grafana rtmpstats and check the vtrans servers used under Active Ingest by Host
  - For the stream/group id check https://bintu.nanocosmos.de/stream/XpjSI-ZqxVx/group and filter for groupid.
- Duplicate ingests.
  - Open up grafana rtmpstats and set up as described earlier.
  - Check Active Ingest by Host and make sure no 2 servers are active at any time for a stream.
- Try ingest using SRT, if it is available.
  - See this guide : https://docs.nanocosmos.de/docs/cloud/srt_ingest
Playout Check
- Last mile bandwidth check
  - If the ingest is to non-vtrans infrastructure, then the passthrough bitrate will be the client playout bitrate. A speed test can be a gauge of the sustainable bandwidth for the client connection.
  - If the ingest is a stream group or daisy-chained vtrans stream, then the customer can test with the lowest transcode profile playback to see if it is a bandwidth issue.
- Encoder specific issues
  - Roland
    - Muxer AVC1 packetization anomaly.
  - Teradek
    - Set encoder H.264 Number of slices to 1.
- Secure playout check.

Token with referrers are not supported and rejected for secured RTMP playback

Setup as:
- Domain: bintu-splay.nanocosmos.de
- RTMP application name: splayrtmp

The secure parameters have to be appended to the RTMP stream name.

RTMP URL:
rtmp://bintu-splay.nanocosmos.de/splayrtmp/'Stream name'

Stream name:
XXXXX-YYYYY?expires=[expires]&tag=[tag]&token=[token]&options=[options]

Stream is stopped

First check the admin dashboard: https://admin-dashboard.nanostream.cloud/ to verify if the stream is live by filtering with the stream name. Eg:“XpjSI-ZqxVx” under Search Stream field. This will show you the Matching Stream. Click on “See Stream Overview”, it will show as “LIVE” if the stream is live.On the same page click on “Organisation Overview” and see if its a secure Orga.
If its a secure orga,copy the “APIKEY” on the same page which will be required to generate the jwt token for the demo player. NOTE: If orga is not secure, this step is not needed.

Play stream using Demo Player

Link used for demo player: https://demo.nanocosmos.de/nanoplayer/release/nanoplayer.html?group.id=XpjSI-ZqxVx&group.security.jwtoken=$token
To generate the Playback token:
- export BINTU_API_KEY="$APIKEY"
- curl -X POST https://token.nanostream.cloud/api/v1/splay
  -H “Accept: application/json”
  -H “content-type: application/json”
  -H “x-bintu-apikey: ${BINTU_API_KEY}”
  –data “{"exp": $(date -u -v+24H +%s), "groupid": "nNCcf-Kst4Q"}”
- Token is not required if the orga is not secure.In that case, just use: https://demo.nanocosmos.de/nanoplayer/release/nanoplayer.html?group.id=XpjSI-ZqxVx

Secure stream playback fails.

Could not generate secure tokens.
- ( how to check if bintu service is under attack or other issues ? )
Token expiry check using Kibana.
- https://kibana.elasticsearch.fsn.hz.k8s.nanostream.cloud/goto/47783de0-d08e-11ee-9fc1-513fec7c4744
- Set the bintu.orgahash.keyword filter and time interval (in UTC time) according to the issue description.
- Check listing, if any.
CTS APM check.
- https://kibana.elasticsearch.fsn.hz.k8s.nanostream.cloud/app/apm/services/CTS/overview?comparisonEnabled=false&comparisonType=day&environment=production-eu-a&kuery=transaction.duration.us%20%3E%201000000&rangeFrom=now-12h&rangeTo=now&transactionType=request&latencyAggregationType=p99
- See if 99th Percentile of “latency > 5 seconds” is significant or not during the current time period. This can point to CTS infrastructure degradation.