SRT streaming: A situation where screen flickering always occurs

[zshaobo]

Description'

Please ensure that the markdown structure is maintained.

A situation where screen flickering is always present in SRT streaming.

1. SRS version: 4.0release
2. The log of SRS is as follows:

The log is normal.

1. The configuration of SRS is as follows (Config):

Using the built-in configuration (conf).

Replay

How to replay bug?

Steps to reproduce the bug

Steps to reproduce the bug:

1. obs推srt流，设置输出参数，微调：zerolatency (obs pushes srt stream, sets output parameters, fine-tunes: zerolatency). At this point, the video appears distorted when viewed through rtmp, but it is normal when using srt-live-server.
2. When using ffmpeg to push the stream with fine-tuning zerolatency, everything works fine without this issue.

Expected behavior:

Resolve the issue of distorted video.

TRANS_BY_GPT3

[zhouxiaojun2008]

This design corresponds to a situation where one frame corresponds to multiple slices.

TRANS_BY_GPT3

[xiaozhihong]

I seem to be unable to appear. Can you please take a screenshot of your complete OBS configuration?

TRANS_BY_GPT3

[zhouxiaojun2008]

I seem to be unable to appear, can you please take a complete screenshot of your OBS configuration and show it to me?

'Zerolatency divides frames into multiple slices, usually one frame corresponds to one slice. The reason for the problem is in the on_ts_h264 function inside srt_to_rtmp. When calling write_h264_ipb_frame, originally it was called once per frame. After setting zerolatency, one frame corresponds to n slices, so it should still be called once. However, in reality, it is called n times, which causes the screen to flicker. The overall architecture of the SRT integration module in SRS is generally fine, but there are still some issues in certain details.

TRANS_BY_GPT3

[runner365]

I seem to be unable to appear, can you please take a screenshot of your complete OBS configuration and show it to me?

The zerolatency feature divides frames into multiple slices, usually one frame corresponds to one slice. The problem arises in the "on_ts_h264" function within the "srt_to_rtmp" module. When calling "write_h264_ipb_frame", originally it should be called once per frame. However, when zerolatency is enabled and one frame corresponds to n slices, it should still be called once. But in reality, it is called n times, which causes the screen to flicker. Overall, the architecture of the SRT integration module in SRS is mostly fine, but there are some issues with certain details.

I will go reproduce and solve the issue. Thank you for helping and providing feedback.

TRANS_BY_GPT3

[zshaobo]

Thank you all for your expertise. I reviewed the principles again and found the problem, just as the brother above mentioned.

Each frame of the video stream is composed of multiple slices. The video uses a combination of AVCC and Annex-B format separators, which leads to sending multiple slice units as multiple frames, resulting in a screen glitch.

At the same time, I also learned that...

The start code has two forms: 3-byte 0x000001 and 4-byte 0x00000001.

The 3-byte 0x000001 start code is only used in one situation, which is when a complete frame is divided into multiple slices. The first slice uses 0x00000001, while the other nalus containing these slices use the 3-byte start code. In all other situations, the start code is 4 bytes.

If, when detecting data, the 3-byte 0x000001 data frame is directly considered as one frame without further splitting and sent, this situation will not be sent multiple times, and it can also solve the problem.

srs_error_t SrsRawH264Stream::annexb_demux(SrsBuffer* stream, char** pframe, int* pnb_frame)
{
    srs_error_t err = srs_success;
    
    *pframe = NULL;
    *pnb_frame = 0;
    
    while (!stream->empty()) {
        // each frame must prefixed by annexb format.
        // about annexb, @see ISO_IEC_14496-10-AVC-2003.pdf, page 211.
        int pnb_start_code = 0;
        if (!srs_avc_startswith_annexb(stream, &pnb_start_code)) {
            return srs_error_new(ERROR_H264_API_NO_PREFIXED, "annexb start code");
        }
        int start = stream->pos() + pnb_start_code;
        
        // find the last frame prefixed by annexb format.
        stream->skip(pnb_start_code);
        while (!stream->empty()) {
//        Modification part, after testing, it can also solve the screen flickering problem.
        // if (srs_avc_startswith_annexb(stream, NULL)) {
            if (pnb_start_code==4&&srs_avc_startswith_annexb(stream, NULL)) {
                break;
            }
            stream->skip(1);
        }
        
        // demux the frame.
        *pnb_frame = stream->pos() - start;
        *pframe = stream->data() + start;
        break;
    }
    
    return err;
}

TRANS_BY_GPT3

[zhouxiaojun2008]

Thank you all for your help. I reviewed the principles again and found the problem, just as the brother above mentioned.

Each frame of the video stream is composed of multiple slices. The video uses a combination of AVCC and Annex-B format separators, which leads to sending multiple slice units as multiple frames, resulting in a screen glitch.

At the same time, I also learned that...

startcode has two forms: 3-byte 0x000001 and 4-byte 0x00000001.

The 3-byte 0x000001 is only used in one specific scenario, which is when a complete frame is divided into multiple slices. The first slice uses 0x00000001, while the rest of the nalus containing these slices use the 3-byte start code. In all other scenarios, the start code is 4 bytes.

If, during data detection, the 3-byte 0x000001 data frame is treated as a complete frame without further splitting and sent directly, this situation will not occur multiple times, and it can also solve the problem.

srs_error_t SrsRawH264Stream::annexb_demux(SrsBuffer* stream, char** pframe, int* pnb_frame)
{
    srs_error_t err = srs_success;
    
    *pframe = NULL;
    *pnb_frame = 0;
    
    while (!stream->empty()) {
        // each frame must prefixed by annexb format.
        // about annexb, @see ISO_IEC_14496-10-AVC-2003.pdf, page 211.
        int pnb_start_code = 0;
        if (!srs_avc_startswith_annexb(stream, &pnb_start_code)) {
            return srs_error_new(ERROR_H264_API_NO_PREFIXED, "annexb start code");
        }
        int start = stream->pos() + pnb_start_code;
        
        // find the last frame prefixed by annexb format.
        stream->skip(pnb_start_code);
        while (!stream->empty()) {
//        After modification and testing, it has been found that it can also solve the screen flickering issue.
        // if (srs_avc_startswith_annexb(stream, NULL)) {
            if (pnb_start_code==4&&srs_avc_startswith_annexb(stream, NULL)) {
                break;
            }
            stream->skip(1);
        }
        
        // demux the frame.
        *pnb_frame = stream->pos() - start;
        *pframe = stream->data() + start;
        break;
    }
    
    return err;
}

你这段话' translates to 'Your sentence' in English.

startcode有两种形式：3字节的0x000001和4字节的0x00000001' translates to 'There are two forms of startcode: 3-byte 0x000001 and 4-byte 0x00000001' in English.

The translation of the given text is as follows:

'3-byte 0x000001 is only used in one specific scenario, which is when a complete frame is divided into multiple slices. The first slice uses 0x00000001, while the rest of the nalus containing these slices use the 3-byte start code. In all other scenarios, the start code is 4 bytes.

Understanding the standard is correct, and that's how I interpreted it as well. However, during my actual testing, I found that some situations are not strictly as described.

For example, I noticed that OBS x264 zerolatency produces AUD+SPS+PPS+SEI+IDR, where SEI has two zeros, which can cause the length of PPS to be miscalculated, and subsequently, the following AUD is also affected, leading to parsing errors.

TRANS_BY_GPT3

[ZSC714725]

In a weak network environment, if tlpkdrop is enabled, the SRT underlying layer will drop some data, which can also cause the client to display a frozen screen. It is suggested to add a detection mechanism based on the continuity of the CC field in the TS stream in such cases.

TRANS_BY_GPT3