The default Agora video module interacts seamlessly with the devices your app runs on. The SDK enables you to add specialized video features to your app using a custom video source.
By default, SDK integrates the default video modules on the device your app runs on for real-time communication. However, there are scenarios where you may want to integrate a custom video capturer. For example:
To manage the capture and processing of video frames when using a custom video source, use methods outside the SDK.
If the format of the custom captured video is Texture and the remote user sees anomalies such as flickering and distortion in the local custom captured video, the best practice is to make a copy of the video data before sending the custom video data back to the SDK, and then send both the original video data and the copied video data back to the SDK. This eliminates the anomalies during the internal data encoding.
The following figure shows the call sequence you need to implement in your app for custom video source:
The following figure shows how the video data is transferred when you customize the video source in Push mode:
pushlVideoFrame
method.Before proceeding, ensure that you have implemented the basic real-time communication functions in your project. For details, see Start a Video Call or Start Live Interactive Video Streaming.
To implement a custom video source in your project, refer to the following steps.
Before calling joinChannel
, call setExternalVideoSource
to specify the custom video source.
// Configure the external video source.
BOOL CAgoraCaptureVideoDlg::EnableExtendVideoCapture(BOOL bEnable)
{
// Create an AutoPtr instance using the IMediaEngine class as template
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
// You can refer to the implementation of the AutoPtr class in AgoraBase.h file of the SDK
// The AutoPtr instance accesses the pointer of the IMediaEngine instance through the arrow operator and calls setExternalVideoSource through the IMediaEngine instance.
// m_rtcEngine is an instance of the IRtcEngine object
mmediaEngine.queryInterface(m_rtcEngine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
int nRet = 0;
agora::base::AParameter apm(m_rtcEngine);
if (mediaEngine.get() == NULL)
return FALSE;
if (bEnable) {
// Enable the custom video source
nRet = mediaEngine->setExternalVideoSource(true, false, false);
}
else {
// Disable the custom video source
nRet = mediaEngine->setExternalVideoSource(false, false);
}
return nRet == 0 ? TRUE : FALSE;
}
Set the video frame properties through the ExternalVideoFrame
instance, and call setVideoEncoderConfiguration
to set the video encoding properties, and start video custom capture.
// Set the video encoder configuration and start custom video source
void CAgoraCaptureVideoDlg::EnableCaputre(BOOL bEnable)
{
if (bEnable == (BOOL)!m_extenalCaptureVideo)return;
// Get device types from the UI
int nIndex = m_cmbVideoType.GetCurSel();
if (bEnable)
{
// Select video capture device per the UI selection. m_agVideoCaptureDevice is an instance from the capture module.
m_agVideoCaptureDevice.SelectMediaCap(nIndex == -1 ? 0 : nIndex);
// Create VIDEOINFOHEADER instance, which is from the Windows Media Device Manager SDK in MFC
VIDEOINFOHEADER videoInfo;
// Create a VideoEncoderConfiguration instance
VideoEncoderConfiguration config;
// Create capture filter
m_agVideoCaptureDevice.CreateCaptureFilter();
// Get the selected video capture device
m_agVideoCaptureDevice.GetCurrentVideoCap(&videoInfo);
// bmiHeader belongs to Win32 _BITMAPINFOHEADER and includes color and dimension information of the image bitmap.
// Set config parameters of the setVideoEncoderConfiguration.
config.dimensions.width = videoInfo.bmiHeader.biWidth;
config.dimensions.height = videoInfo.bmiHeader.biHeight;
// m_videoFrame is an instance of the ExternalVideoFrame structure in IVideoFrame. Set the external video frame properties
// The following parameters are obtained from the bmiHeader parameter in videoInfo.
m_videoFrame.stride = videoInfo.bmiHeader.biWidth;
m_videoFrame.height = videoInfo.bmiHeader.biHeight;
// Set the following parameters.
m_videoFrame.rotation = 0;
m_videoFrame.cropBottom = 0;
m_videoFrame.cropLeft = 0;
m_videoFrame.cropRight = 0;
m_videoFrame.cropTop = 0;
m_videoFrame.format = agora::media::base::VIDEO_PIXEL_I420;
m_videoFrame.type = agora::media::base::ExternalVideoFrame::VIDEO_BUFFER_TYPE::VIDEO_BUFFER_RAW_DATA;
// Set video encoding configurations
m_rtcEngine->setVideoEncoderConfiguration(config);
// Initialize the video renderer module
m_d3dRender.Init(m_localVideoWnd.GetSafeHwnd(),
videoInfo.bmiHeader.biWidth, videoInfo.bmiHeader.biHeight, true);
// Start video capture
m_agVideoCaptureDevice.Start();
// Enable video
m_rtcEngine->enableVideo();
// Start local preview
m_rtcEngine->startPreview();
}
else {
// Stop video capture
m_agVideoCaptureDevice.Stop();
// Remove video capture module
m_agVideoCaptureDevice.RemoveCaptureFilter();
if (m_rtcEngine)
{
// Remove video capture module
m_rtcEngine->disableVideo();
// Stop local preview
m_rtcEngine->stopPreview();
}
}
}
Call pushVideoFrame
to send the video frames to the SDK for later use.
void CAgoraCaptureVideoDlg::ThreadRun(CAgoraCaptureVideoDlg * self)
{
// Create an AutoPtr instance using the IMediaEngine as template
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
// Visit agora::AGORA_IID_MEDIA_ENGINE by calling the queryInterface method
mediaEngine.queryInterface(self->m_rtcEngine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
{
if (self->m_videoFrame.format == 24)
{
return;
}
else if(self->m_videoFrame.format == agora::media::base::VIDEO_PIXEL_I420) {
int bufSize = self->m_videoFrame.stride * self->m_videoFrame.height * 3 / 2;
int timestamp = GetTickCount();
// If there are available video frames from the capture module, set the timestamp to current time
if (CAgVideoBuffer::GetInstance()->readBuffer(self->m_buffer, bufSize, timestamp)) {
memcpy_s(self->m_imgBuffer, bufSize, self->m_buffer, bufSize);
self->m_videoFrame.timestamp = timestamp;
}
else
{
// If there are no available video frames from the capture module,
// sleep for 1 millisecond, skip the steps below, and continue with the loop
Sleep(1);
continue;
}
// Assign the buffer to the buffer parameter of m_videoFrame (ExternalVideoFrame) for pushVideoFrame
self->m_videoFrame.buffer = self->m_imgBuffer;
// Call pushVideoFrame to push the captured video frames to the SDK
mediaEngine->pushVideoFrame(&self->m_videoFrame);
}
}
}
This section includes in depth information about the methods you used in this page, and links to related pages.
Agora provides the following open-source sample project CustomVideoCapture on GitHub.