This page shows you how to get raw video data for pre- and post-processing.
During the video transmission process, you can pre- and post-process the captured video data to achieve the desired playback effect.
Agora provides the raw data function for you to process the video data as per your application scenario. This function enables you to pre-process the captured video frames before sending it to the encoder, or to post-process the decoded video frames.
Before using the raw data function, ensure that you have implemented the basic real-time communication functions in your project. See Start a Video Call or Start Interactive Live Video Streaming for details.
To implement the raw video data function in your project, refer to the following steps.
Before joining a channel, create an IVideoFrameObserver
object and then call registerVideoFrameObserver
to register video frame observer.
// Register or unregister a video frame observer object
BOOL CAgoraOriginalVideoDlg::RegisterVideoFrameObserver(BOOL bEnable, IVideoFrameObserver *videoFrameObserver)
{
// Create an AutoPtr instance using the IMediaEngine class as template
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
// The AutoPtr instance calls the queryInterface method to obtain the pointer of the IMediaEngine instance through IID.
// The AutoPtr instance access the pointer of the IMediaEngine instance through the arrow operator and call registerVideoFrameObserver through the IMediaEngine instance.
mediaEngine.queryInterface(m_rtcEngine, AGORA_IID_MEDIA_ENGINE);
int nRet = 0;
agora::base::AParameter apm(*m_rtcEngine);
if (mediaEngine.get() == NULL)
return FALSE;
if (bEnable)
{
//Register a video frame observer object
nRet = mediaEngine->registerVideoFrameObserver(videoFrameObserver);
}
else
{
//Unregister a video frame observer object
nRet = mediaEngine->registerVideoFrameObserver(nullptr);
}
return nRet == 0 ? TRUE : FALSE;
}
Implement the onCaptureVideoFrame
, onRenderVideoFrame
, and onScreenCaptureVideoFrame
callbacks. Get the video frames captured the callbacks and process the data according to your needs.
// Get the original video data collected by the local camera through the onCaptureVideoFrame callback, perform grayscale processing, and send it back to the SDK
bool CGrayVideoProcFrameObserver::onCaptureVideoFrame(VideoFrame &videoFrame)
{
int nSize = videoFrame.height * videoFrame.width;
memset(videoFrame.uBuffer, 128, nSize / 4);
memset(videoFrame.vBuffer, 128, nSize / 4);
return true;
}
// Get the remote video data through the onRenderVideoFrame callback
bool CGrayVideoProcFrameObserver::onRenderVideoFrame(const char *channelId, rtc::uid_t remoteUid, VideoFrame &videoFrame)
{
return true;
}
bool CAverageFilterVideoProcFrameObserver::onCaptureVideoFrame(VideoFrame &videoFrame)
{
static int step = 1;
static bool flag = true;
if (flag)
{
step += 2;
}
else
{
step -= 2;
}
if (step >= 151)
{
flag = false;
step -= 4;
}
else if (step <= 0)
{
flag = true;
step += 4;
}
AverageFiltering((unsigned char *)videoFrame.yBuffer, videoFrame.width, videoFrame.height, step);
AverageFiltering((unsigned char *)videoFrame.uBuffer, videoFrame.width / 2, videoFrame.height / 2, step);
AverageFiltering((unsigned char *)videoFrame.vBuffer, videoFrame.width / 2, videoFrame.height / 2, step);
return true;
}
// Get the video data collected from the screen through the onScreenCaptureVideoFrame callback
bool CGrayVideoProcFrameObserver::onScreenCaptureVideoFrame(VideoFrame &videoFrame)
{
return true;
}
This section includes in depth information about the methods you used in this page, and links to related pages.
Agora provides an open-source sample project OriginalVideo on GitHub.