Frame rate of video streaming is assumed to be 30fps

[shoamano83]

Bug Report

When encoding video frames, timestamps that are fed to the encoder along with the video frames are increased by 1/30 second. This assumes that input video is 30fps. Actual frame rate is specific to app.

This may not be an issue for RAW H.264 streaming since such stream doesn’t contain timestamp information. However, for a RTP stream we should append proper timestamps.

OS & Version Information

• SDL iOS Version: master branch (v4.6.1), develop branch, release/5.0.0 branch

Test Case, Sample Code, and / or Example App

Links to current implementation are provided below for reference:

master branch:

sdl_ios/SmartDeviceLink/SDLStreamingMediaManager.m

Line 218 in 4d0f991

OSStatus status = VTCompressionSessionEncodeFrame(_compressionSession, imageBuffer, CMTimeMake(self.currentFrameNumber++, 30), kCMTimeInvalid, NULL, (__bridge void *)self, NULL);

develop branch:

sdl_ios/SmartDeviceLink/SDLStreamingMediaManager.m

Line 226 in 50c96a7

OSStatus status = VTCompressionSessionEncodeFrame(_compressionSession, imageBuffer, CMTimeMake(self.currentFrameNumber++, 30), kCMTimeInvalid, NULL, (__bridge void *)self, NULL);

release/5.0.0 branch:

sdl_ios/SmartDeviceLink/SDLVideoEncoder.m

Line 122 in e10a5aa

OSStatus status = VTCompressionSessionEncodeFrame(_compressionSession, imageBuffer, CMTimeMake(self.currentFrameNumber++, 30), kCMTimeInvalid, NULL, (__bridge void *)self, NULL);

[shoamano83]

Discussion:

• -sendVideoData: API can be extended to receive timestamp information along with video frames.

• release/5.0.0 adds a new feature to send background frames - see:
  

  sdl_ios/SmartDeviceLink/SDLStreamingMediaLifecycleManager.m

  Line 521 in e10a5aa

  [self.videoEncoder encodeFrame:self.backgroundingPixelBuffer];

  We need to figure out the timestamps for these frames as well.

• videoEncoderSettings can include frame rate information using kVTCompressionPropertyKey_ExpectedFrameRate. However, according to the documentation this is “provided as a hint to the video encoder” and “The actual frame rate depends on frame durations and may vary.”
  https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_expectedframerate
  That is, it is not guaranteed that every video projection app configures this value.