Raw Video Data

Introduction

During real-time communications, you can pre- and post-process the video data and modify it for desired playback effects.

The Native SDK uses the IVideoFrameObserver class to provide raw video data functions. You can pre-process the data before sending it to the encoder and modify the captured video frames. You can also post-process the data after sending it to the decoder and modify the received video frames.

Sample project

Agora provides the following open-source sample projects on GitHub that implement the raw video data function:

• iOS: RawMediaData
• macOS: RawMediaData

You can download them to try out this function and view the source code.

Implementation

Follow these steps to implement the raw video data function in your project:

1. Before joining the channel, call registerVideoFrameObserver to register the video frame observer, and implement the IVideoFrameObserver class.
2. After registering the observer, the SDK returns the raw video data in the onCaptureVideoFrame, onPreEncodeVideoFrame, or onRenderVideoFrame callbacks for each video frame.
3. Process the raw video data, and then send the processed data back to the SDK through the callbacks mentioned in step 2.

#import <AgoraRtcKit/IAgoraMediaEngine.h>
#import <AgoraRtcKit/IAgoraRtcEngine.h>

API call sequence

The following diagram shows how to implement the raw data function in your project:

Sample code

In addition to the API call sequence diagram, you can refer to the following code samples to implement the raw video data function in your project.

1. Initialize AgoraRtcEngineKit, and enable the video module.

   // Swift
   // Initialize AgoraRtcEngineKit
   let config = AgoraRtcEngineConfig()
   agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)
   
   // Enable the vide module
   agoraKit.enableVideo()

2. Register the video frame observer.

   // Swift
   let videoType:ObserverVideoType = ObserverVideoType(rawValue: ObserverVideoType.captureVideo.rawValue | ObserverVideoType.renderVideo.rawValue | ObserverVideoType.preEncodeVideo.rawValue)
   agoraMediaDataPlugin?.registerVideoRawDataObserver(videoType)
   agoraMediaDataPlugin?.videoDelegate = self;

   In the .mm file, call the C++ APIs to implement registerVideoRawDataObserver.

   - (void)registerVideoRawDataObserver:(ObserverVideoType)observerType {
     // Get the C++ handle of the Native SDK
     agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
     // Create IMediaEngine
     agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
     mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
     // Call registerVideoFrameObserver to register the video frame observer
     mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
     s_videoFrameObserver.mediaDataPlugin = self;
   }

3. Join a channel.

   // Swift
   let result = agoraKit.joinChannel(byToken: KeyCenter.Token, channelId: channelName, info: nil, uid: 0) {[unowned self] (channel, uid, elapsed) -> Void in
     self.isJoined = true
     LogUtils.log(message: "Join \(channel) with uid \(uid) elapsed \(elapsed)ms", level: .info)
   }

4. Get the raw video data, and process the data.

   // Swift
   // Get the video frame captured by the local camera.
   func mediaDataPlugin(_ mediaDataPlugin: AgoraMediaDataPlugin, didCapturedVideoRawData videoRawData: AgoraVideoRawData) -> AgoraVideoRawData {
     return videoRawData
   }
   
   // Get the video frame sent by the remote user.
   func mediaDataPlugin(_ mediaDataPlugin: AgoraMediaDataPlugin, willRenderVideoRawData videoRawData: AgoraVideoRawData, ofUid uid: uint) -> AgoraVideoRawData {
     return videoRawData
   }

   In the .mm file, call the C++ APIs to implement the callbacks that get the raw data.

   // Get the video frame captured by the local camera through the onCaptureVideoFrame callback
   virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override
   {
       if (!mediaDataPlugin && ((mediaDataPlugin.observerVideoType >> 0) == 0)) return true;
       @autoreleasepool {
           AgoraVideoRawData *newData = nil;
           if ([mediaDataPlugin.videoDelegate respondsToSelector:@selector(mediaDataPlugin:didCapturedVideoRawData:)]) {
               AgoraVideoRawData *data = getVideoRawDataWithVideoFrame(videoFrame);
               newData = [mediaDataPlugin.videoDelegate mediaDataPlugin:mediaDataPlugin didCapturedVideoRawData:data];
               modifiedVideoFrameWithNewVideoRawData(videoFrame, newData);
           }
       }
     return true;
   }
   
   // Get the video frame sent by the remote user through the  onRenderVideoFrame callback
   virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) override
   {
       if (!mediaDataPlugin && ((mediaDataPlugin.observerVideoType >> 1) == 0)) return true;
       @autoreleasepool {
           AgoraVideoRawData *newData = nil;
           if ([mediaDataPlugin.videoDelegate respondsToSelector:@selector(mediaDataPlugin:willRenderVideoRawData:ofUid:)]) {
               AgoraVideoRawData *data = getVideoRawDataWithVideoFrame(videoFrame);
               newData = [mediaDataPlugin.videoDelegate mediaDataPlugin:mediaDataPlugin willRenderVideoRawData:data ofUid:uid];
               modifiedVideoFrameWithNewVideoRawData(videoFrame, newData);
           }
       }
     return true;
   }

5. Unregister the video frame observer.

   // Swift
   agoraMediaDataPlugin?.deregisterVideoRawDataObserver(ObserverVideoType(rawValue: 0))

   Call the C++ APIs in the .mm file to implement deregisterVideoRawDataObserver.

   - (void)deregisterVideoRawDataObserver:(ObserverVideoType)observerType {
     // Get the C++ handle of the Native SDK
     agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
     // Create IMediaEngine
     agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
     mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
     // Call registerVideoFrameObserver to unregister the video frame observer
     mediaEngine->registerVideoFrameObserver(NULL);
     s_videoFrameObserver.mediaDataPlugin = nil;
   }

API reference

• getNativeHandle
• registerVideoFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• onPreEncodeVideoFrame

Considerations

• The raw video data function is implemented by the C++ APIs of the SDK, and you need to mix the Objective-C and C++ in a .mm file. See AgoraMediaDataPlugin.mm for reference.
• Ensure that you call getNativeHandle to get the C++ handle each time before calling registerVideoFrameObserver.
• The raw video data function is resource intensive and may affect performance.

See also

Refer to Raw Audio Data if you want to implement the raw audio data function in your project.