Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

stream not recovering when interrupted (reading from itself) #1611

Open
motiejus opened this issue Feb 19, 2025 · 1 comment
Open

stream not recovering when interrupted (reading from itself) #1611

motiejus opened this issue Feb 19, 2025 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@motiejus
Copy link

motiejus commented Feb 19, 2025

Once a stream disconnects for some reason, it doesn't recover. It can take hours or days until it gets interrupted; but it does not recover itself -- go2rtc needs to be restarted.

I have a specific example of vno4-dome-panorama-low, but sometimes other streams fail (I just captured the full suite of logs with this one).

Here is the moment when it gets disconnected and a bit around (the last recorded frame is 21:31:38):

Feb 19 21:31:24 fwminex go2rtc[847109]: 21:31:24.785 TRC [rtsp] server request:
Feb 19 21:31:24 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 21:31:24 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 21:31:24 fwminex go2rtc[847109]: Session: 32236871
Feb 19 21:31:24 fwminex go2rtc[847109]: Cseq: 6
Feb 19 21:31:24 fwminex go2rtc[847109]:
Feb 19 21:31:24 fwminex go2rtc[847109]: 21:31:24.785 TRC [rtsp] server response:
Feb 19 21:31:24 fwminex go2rtc[847109]: RTSP/1.0 200 OK
Feb 19 21:31:24 fwminex go2rtc[847109]: Cseq: 6
Feb 19 21:31:24 fwminex go2rtc[847109]: Session: 32236871
Feb 19 21:31:24 fwminex go2rtc[847109]:
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 TRC [rtsp] server request:
Feb 19 21:31:38 fwminex go2rtc[847109]: TEARDOWN rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 21:31:38 fwminex go2rtc[847109]: Cseq: 7
Feb 19 21:31:38 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 21:31:38 fwminex go2rtc[847109]: Session: 32236871
Feb 19 21:31:38 fwminex go2rtc[847109]:
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 DBG [rtsp] handle error="read tcp [::1]:8554->[::1]:38742: read: connection reset by peer"
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 DBG [streams] stop producer url=ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264#width=1280
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 TRC [exec] kill with signal=9
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.029 DBG [rtsp] handle error=EOF
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.029 DBG [rtsp] disconnect stream=vno4-dome-panorama-orig
Feb 19 21:31:41 fwminex go2rtc[847109]: 21:31:41.017 TRC [rtsp] server request:
Feb 19 21:31:41 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-orig RTSP/1.0
Feb 19 21:31:41 fwminex go2rtc[847109]: Session: 12820664
Feb 19 21:31:41 fwminex go2rtc[847109]: Cseq: 40
Feb 19 21:31:41 fwminex go2rtc[847109]: User-Agent: go2rtc/ffmpeg
Feb 19 21:31:41 fwminex go2rtc[847109]:
Feb 19 21:31:41 fwminex go2rtc[847109]: 21:31:41.017 TRC [rtsp] server response:
Feb 19 21:31:41 fwminex go2rtc[847109]: RTSP/1.0 200 OK
Feb 19 21:31:41 fwminex go2rtc[847109]: Cseq: 40
Feb 19 21:31:41 fwminex go2rtc[847109]: Session: 12820664
Feb 19 21:31:41 fwminex go2rtc[847109]:
Feb 19 21:31:43 fwminex go2rtc[847109]: 21:31:43.691 TRC [rtsp] server request:
Feb 19 21:31:43 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 21:31:43 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 21:31:43 fwminex go2rtc[847109]: Cseq: 1

When ffmpeg routinely tries to connect to it:

Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.412 TRC [rtsp] server request:
Feb 19 22:46:55 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 22:46:55 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 22:46:55 fwminex go2rtc[847109]: Cseq: 1
Feb 19 22:46:55 fwminex go2rtc[847109]:
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.412 TRC [rtsp] server response:
Feb 19 22:46:55 fwminex go2rtc[847109]: RTSP/1.0 200 OK
Feb 19 22:46:55 fwminex go2rtc[847109]: Cseq: 1
Feb 19 22:46:55 fwminex go2rtc[847109]: Public: OPTIONS, SETUP, TEARDOWN, DESCRIBE, PLAY, PAUSE, ANNOUNCE, RECORD
Feb 19 22:46:55 fwminex go2rtc[847109]:
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.413 TRC [rtsp] server request:
Feb 19 22:46:55 fwminex go2rtc[847109]: DESCRIBE rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 22:46:55 fwminex go2rtc[847109]: Accept: application/sdp
Feb 19 22:46:55 fwminex go2rtc[847109]: Cseq: 2
Feb 19 22:46:55 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 22:46:55 fwminex go2rtc[847109]:
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.413 DBG [rtsp] new consumer stream=vno4-dome-panorama-low
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.413 TRC [streams] check cons=455 media=video, sendonly, ANY

go2rtc config:

api:
  listen: :1984
ffmpeg:
  bin: /nix/store/wcf97bi1lbl299iamk55jqsr5m85sbmq-ffmpeg-headless-7.1-bin/bin/ffmpeg
log:
  format: text
  level: trace
rtsp: {}
streams:
  vno4-dome-panorama-high:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264
  vno4-dome-panorama-low:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264#width=1280
  vno4-dome-panorama-orig:
  - ffmpeg:rtsp://frigate:${FRIGATE_RTSP_PASSWORD}@192.168.188.10/cam/realmonitor?channel=1&subtype=2
  vno4-dome-ptz-high:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-ptz-orig#hardware=vaapi#video=h264
  vno4-dome-ptz-low:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-ptz-orig#hardware=vaapi#video=h264#width=1280
  vno4-dome-ptz-orig:
  - ffmpeg:rtsp://frigate:${FRIGATE_RTSP_PASSWORD}@192.168.188.10/cam/realmonitor?channel=2&subtype=0
webrtc: {}

The -orig streams are h.265 (for bandwidth), which get transcoded to two h.264 streams for local processing.

Camera Dahua DH-SDT3E410-8P-MB-A-PV1

I put whatever I could find here, not sure what's useful.

Image

vno4-dome-panorama-low config:

{
  "producers": [
    {
      "id": 114,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:46042 forwarded localhost:8554",
      "source": "exec:/nix/store/wcf97bi1lbl299iamk55jqsr5m85sbmq-ffmpeg-headless-7.1-bin/bin/ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_flags allow_profile_mismatch -allowed_media_types video -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://localhost:8554/vno4-dome-panorama-orig -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -an -vf \"format=vaapi|nv12,hwupload,scale_vaapi=1280:-1:out_color_matrix=bt709:out_range=tv:format=nv12\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/3932ae285ec55ea2f2cc7c4fb6d1de3c",
      "url": "rtsp://localhost:8554/vno4-dome-panorama-orig",
      "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=go2rtc/1.9.7\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat 61.7.100\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z2QMKawrQCgF/zTUFBgHwAAAAwBAAAAMo8IBCoA=,aO44sA==; profile-level-id=640C29\r\na=control:streamid=0\r\n",
      "user_agent": "ffmpeg/go2rtc",
      "medias": [
        "video, recvonly, H264"
      ],
      "receivers": [
        {
          "id": 115,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 41,
            "profile": "High"
          },
          "bytes": 7302821,
          "packets": 6035
        }
      ],
      "bytes_recv": 7375661
    }
  ],
  "consumers": []
}

vno4-dome-panorama-orig config:

{
  "producers": [
    {
      "id": 67,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:40604 forwarded 192.168.188.10",
      "source": "exec:/nix/store/wcf97bi1lbl299iamk55jqsr5m85sbmq-ffmpeg-headless-7.1-bin/bin/ffmpeg -hide_banner -v error -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://frigate:*@192.168.188.10/cam/realmonitor?channel=1\u0026subtype=2 -c copy -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/abed6807a35cfbfb900ef6ce8bad7790",
      "url": "rtsp://frigate:*@192.168.188.10/cam/realmonitor?channel=1\u0026subtype=2",
      "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=Media Server\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat 61.7.100\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 sprop-vps=QAEMAf//AWAAAAMAoAAAAwAAAwB7rAwAAAMABAAAAwBmqA==; sprop-sps=QgEBAWAAAAMAoAAAAwAAAwB7oAFQIAvFjS7kUvzcFBgUEAAAAwAQAAADAZoQ; sprop-pps=RAHA8o54OzQ=\r\na=control:streamid=0\r\n",
      "user_agent": "ffmpeg/go2rtc",
      "medias": [
        "video, recvonly, H265"
      ],
      "receivers": [
        {
          "id": 68,
          "codec": {
            "codec_name": "hevc",
            "codec_type": "video"
          },
          "childs": [
            69
          ],
          "bytes": 1613523802,
          "packets": 1186657
        }
      ],
      "bytes_recv": 1627798574
    }
  ],
  "consumers": [
    {
      "id": 65,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "[::1]:41450",
      "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.7\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 sprop-vps=QAEMAf//AWAAAAMAoAAAAwAAAwB7rAwAAAMABAAAAwBmqA==; sprop-sps=QgEBAWAAAAMAoAAAAwAAAwB7oAFQIAvFjS7kUvzcFBgUEAAAAwAQAAADAZoQ; sprop-pps=RAHA8o54OzQ=\r\na=control:trackID=0\r\n",
      "user_agent": "go2rtc/ffmpeg",
      "medias": [
        "video, sendonly, ANY",
        "audio, sendonly, ANY"
      ],
      "senders": [
        {
          "id": 69,
          "codec": {
            "codec_name": "hevc",
            "codec_type": "video"
          },
          "parent": 68,
          "bytes": 1613523802,
          "packets": 1186657
        }
      ],
      "bytes_send": 1632510314
    }
  ]
}

I would appreciate some guidance what to do to make it recover, or how to troubleshoot it further.

Thank you!

motiejus added a commit to motiejus/config that referenced this issue Feb 19, 2025
@AlexxIT AlexxIT added the bug Something isn't working label Feb 20, 2025
@AlexxIT AlexxIT self-assigned this Feb 20, 2025
@motiejus
Copy link
Author

Happened again. Here is an example go2rtc log line with non-debug logging:

Feb 21 04:36:22 fwminex go2rtc[296913]: 04:36:22.848 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:170 > error=EOF url=ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264

... which my poor man's watchdog then promptly detected & mitigated.

If there is any more info I can provide, let me know.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants