MFormats SDK: frame rate control

In MFormats SDK, the frame rate - also known as frame frequency or frames per second (FPS) - is what defines the time that a frame is displayed for on the screen. For example, 25 FPS means that every video frame is displayed for exactly 40 ms along with its synced chunks of audio.

Frame rate control in MFormats SDK is implemented in sink objects. MFPreview, MFRenderer and MWriter are sink objects because they (in a similar way to the preview and renderer filters in DirectShow) receive samples and render them outside of the application.

These objects are responsible for the display time of a frame (frame rate control). If a frame is sent to the preview, it will be displayed for a pre-defined time. Even if the next frame is received, the ReceiverPutFrame() method will not return until the display of the previous frame is finished. This logic is implemented in MFormats SDK internally.

MFPreview is used to render the video to the preview and to play audio to the sound device. Its behaviour is a little different depending on wether the stream contains audio or not.

  • When the frame contains both video and audio, the clock (timer) of the sound card is used to control the rate (the display of the video frames is adjusted to the audio stream).
  • For video-only frames the system timer is used to control the rate: when the video frame is displayed, the SDK waits for the system timer to trigger the display of the next frame.

MFRenderer is used to send frames to a device (such as Blackmagic or AJA). When rendering to a device, the clock of the device is used to control the frame rate (the device itself controls playback according to the way it had been configured). See also the post about playing out to a device.

Usually devices have a buffer of at least 1 frame and can tell us when the display of the first frame is fihished. MFormats SDK waits for the first frame to complete playback, and then only pushes the buffered frame out for display, at the same time sending the next frame to the buffer. At this moment the ReceiverPutFrame method is returned. Because of this, there usually is a delay of at least one frame between playback and output (this may vary due to the specifics of the output drivers and hardware).

MFWriter encodes (compresses) and writes frames to a file or streams them to the network. The MFWriter object controls the frame rate according to it's internal speed of processing the frames (usually defined by the speed of the encoder) - it returns the ReceiverPutFrame() method once the processing of the previous frame is finished and MFWriter is ready to receive the next one.

How to use rate control

The ReceiverPutFrame method has a _rtMaxWait parameter. It's value is measured in 10 ns as an integer (1 second is "10000000"):

  • If rate control is not required, this parameter should be set to "0". In this case, the ReceiverPutFrame() method will not wait: it will return immediately after the processing of this frame is finished (such as when encoding and writing to a file).
  • If rate control is required, set this parameter to "-1". In this case, ReceiverPutFrame() will wait until the sink object completes the display of the frame. For example, it will wait for 40 ms if the MRenderer is configured to play at 25 FPS via a Blackmagic card. When using serveral sink objects, this value will make this object the master object.
  • In other cases it can be set to a maximum value (such as "400000" for 25 FPS). Note, that waiting time is not yet implemented for some objects - so, its best to use "0" and "-1" values for now.

When there's only one source object and one sink object, the rate is either controlled by the sink object or not controlled at all (the frames are being processed at maximum speed - such as in the fast transcoding use case).

When using several sink objects at the same time (for example, MFRenderer and MFPreview) it is recommended to use just one object for rate control - this object becomes the master object. It is best to use the last object in the cycle to control the rate. If the master sink works faster than the non-master sink, than the non-master object will be dropping frames (for example, MFPreview will be dropping frames if you decide that encoding is more important and define MFWriter as the master object).

This is quite important, so please take it into account when designing your application. See examples below for help.

Examples of using rate control in MFormats SDK

1. Playing out to a device and previewing - when quality playback on the device is the goal. Uses MFRenderer object as a master object. In this case, all the frames will be sent to the output device. On the preview each frame will be displayed for a period of time that is controlled by the output device.

pPreview.ReceiverFramePut( pFrame, 0, "" ) // no rate control
pRenderer.ReceiverFramePut( pFrame, -1, "" ) // master object waits for required time

2. Encoding without frame drops - when good quality is the top priority. In this case it will wait until the frame is recorded, so, all the frames will be captured.

pWriter.ReceiverFramePut( pFrame, -1, "" ) // rate control is enabled

3. Encoding and playing out to a device - when quality of the output is preferable. In this case all the frames will be sent to the output device, but some frames may be dropped while encoding (if there is not enough time).

pWriter.ReceiverFramePut( pFrame, 0, "" ) // don't wait for encoding
pRenderer.ReceiverFramePut( pFrame, -1, "" ) // wait (rate control is enabled) for output

4. Transcoding with maximum speed and previewing - when speed of transcoding is the target goal. Allows frames to be dropped for the preview, but makes sure all frames are encoded by the MFWriter object.

pPreview.ReceiverFramePut( pFrame, 0, "" ) // don't wait
pWriter.ReceiverFramePut( pFrame, -1, "" ) // wait