This page shows you how to use the APIs provided by the Agora SDK v4.0.0 to build a video filter extension for Android.
To build a video filter extension, you use the following APIs:
IExtensionVideoFilter
: This interface implements the function of receiving, processing, and delivering video data.IExtensionProvider
: This interface encapsulates the functions in IExtensionVideoFilter
into an extension.Before proceeding, ensure that your development environment meets the following requirements:
Follow these steps to integrate the SDK into your project:
rtc/sdk/low_level_api/include
directory and save them under the directory of your project file.This section describes the APIs required for implementing a video filter extension.
To implement a video filter, you need to implement the IExtensionVideoFilter
class. You can find the class in the NGIAgoraMediaNode.h
file. IExtensionVideoFilter
contains the following methods:
getProcessMode
start
stop
getVideoFormatWanted
adaptVideoFrame
pendVideoFrame
deliverVideoFrame
setProperty
getProperty
Sets how the SDK communicates with your video filter extension. The SDK triggers this callback first when loading the extension. After receiving the callback, you need to return mode
and independent_thread
to specify how the SDK communicates with the extension.
virtual void getProcessMode(ProcessMode& mode, bool& independent_thread) = 0;
Parameter | Description |
---|---|
mode | The mode for transferring video frames between the SDK and extension. You can set it to the following values:adaptVideoFrame .pendVideoFrame , and the extension returns processed video frames to the SDK through deliverVideoFrame . |
independent_thread | Whether to create an independent thread for the extension: |
You can set the value of mode
and independent_thread
as follows:
mode
to Async
and independent_thread
to false;
if your extension does not use complicated YUV algorithm, Agora recommends setting mode
to Sync
and independent_thread
to false
. mode
to Sync
and independent_thread
to true
.virtual int start(agora::agora_refptr<Control> control) = 0;
The SDK triggers this callback when the IExtensionVideoFilter
instance is created. You can initialize OpenGL in this callback.
The SDK also passes a Control
object to the extension in this method. The Control
class provides methods for the extension to interact with the SDK. You can implement the methods in the Control
class based on your actual needs:
class Control : public RefCountInterface {
public:
/**
* In asynchronous mode (mode is set to Async), the extension calls this methods to
* return the processed video frame to the SDK.
* Before calling this method, ensure that the SDK submits the video frame to
* the extension through pendVideoFrame.
*/
virtual ProcessResult deliverVideoFrame(agora::agora_refptr<IVideoFrame> frame) = 0;
/**
* If the extension needs a new memory pool, call this method to create a new
* IVideoFrame object for better memory management.
* For example, an image enhancement extension can call this method to save both
* the original frame and processed frame with more efficient memory management.
*/
virtual agora::agora_refptr<IVideoFrameMemoryPool> getMemoryPool() = 0;
/**
* Call this method to report an event to the SDK. The SDK then sends the event
* notification to the app.
*/
virtual int postEvent(const char* key, const char* value) = 0;
/**
* Call this method to print logs to the SDK.
*/
virtual void printLog(commons::LOG_LEVEL level, const char* format, ...) = 0;
/**
* If an unrecoverable error occurs within the extension, call this method to report
* the error and stop SDK from sending video frames to the extension. The SDK then
* passes the error message to the app.
*/
virtual void disableMe(int error, const char* msg) = 0;
};
virtual int stop() = 0;
The SDK triggers this callback when the IExtensionVideoFilter
instance is destroyed. You can release OpenGL in this callback.
Sets the type and format of the video frame sent to your extension. The SDK triggers this callback before sending a video frame to the extension. In the callback, you need to specify the type and format for the frame. You can change the type and format of subsequent frames when you receive the next callback.
virtual void getVideoFormatWanted(VideoFrameData::Type& type, RawPixelBuffer::Format& format) = 0;
Parameter | Description |
---|---|
type | The type of the video frame. Currently you can only set it to RawPixels , which means raw data. |
format | The format of the video frame. You can set it to the following values:Unknown : An unknown format.I420 : The I420 format.I422 : The I420 format.``NV21 : The NV21 format.NV12 : The NV12 format.RGBA : The RGBA format.ARGB : The AGRB format.BGRA : The BGRA format. |
Adapts the video frame. In synchronous mode (mode is set to Sync)
, the SDK and extension transfer video frames through this method. By calling this method, the SDK sends video frames to the extension with in,
and the extension returns the processed frames with out
.
virtual ProcessResult adaptVideoFrame(agora::agora_refptr<IVideoFrame> in, agora::agora_refptr<IVideoFrame>& out) {
return ProcessResult::kBypass;
}
Parameters
Parameter | Description |
---|---|
in | An input parameter. The video frame to be processed by the extension. |
out | An output parameter. The processed video frame. |
Returns
The result of processing the video frame:
Submits the video frame. In asynchronous mode (mode is set to Async
), the SDK submits the video frame to the extension through this method. After calling this method, the extension must return the processed video frame through deliverVideoFrame
in the Control
class.
virtual ProcessResult pendVideoFrame(agora::agora_refptr<IVideoFrame> frame) {
return ProcessResult::kBypass;
}
Parameters
Parameter | Description |
---|---|
frame | The video frame to be processed by the extension. |
Returns
The result of processing the video frame:
Sets the property of the video filter extension. When an app client calls setExtensionProperty
, the SDK triggers this callback. In the callback, you need to return the extension property.
int ExtensionVideoFilter::setProperty(const char *key, const void *buf, size_t buf_size)
Parameter | Description |
---|---|
key | The key of the property. |
buf | The buffer of the property in the JSON format. You can use the open source nlohmann/json library for the serialization and deserialization between the C++ struct and the JSON string. |
buf_size | The size of the buffer. |
Gets the property of the video filter extension. When the app client calls getExtensionProperty
, the SDK calls this method to get the extension property.
int ExtensionVideoFilter::getProperty(const char *key, void *buf, size_t buf_size)
Parameter | Description |
---|---|
key | The key of the property. |
property | The pointer to the property. |
buf_size | The size of the buffer. |
The following code sample shows how to use these APIs together to implement a video filter:
#include "ExtensionVideoFilter.h"
#include "../logutils.h"
#include <sstream>
namespace agora {
namespace extension {
ExtensionVideoFilter::ExtensionVideoFilter(agora_refptr<ByteDanceProcessor> byteDanceProcessor):threadPool_(1) {
byteDanceProcessor_ = byteDanceProcessor;
}
ExtensionVideoFilter::~ExtensionVideoFilter() {
byteDanceProcessor_->releaseOpenGL();
}
// Set how the SDK communicates with your video filter extension.
void ExtensionVideoFilter::getProcessMode(ProcessMode& mode, bool& independent_thread) {
mode = ProcessMode::kSync;
independent_thread = false;
mode_ = mode;
}
// Set the type and format of the video frame sent to your extension.
void ExtensionVideoFilter::getVideoFormatWanted(rtc::VideoFrameData::Type& type,
rtc::RawPixelBuffer::Format& format) {
type = rtc::VideoFrameData::Type::kRawPixels;
format = rtc::RawPixelBuffer::Format::kI420;
}
// Save the Control object and initialize OpenGL.
int ExtensionVideoFilter::start(agora::agora_refptr<Control> control) {
PRINTF_INFO("ExtensionVideoFilter::start");
if (!byteDanceProcessor_) {
return -1;
}
if (control) {
control_ = control;
byteDanceProcessor_->setExtensionControl(control);
}
if (mode_ == ProcessMode::kAsync){
invoker_id = threadPool_.RegisterInvoker("thread_videofilter");
auto res = threadPool_.PostTaskWithRes(invoker_id, [byteDanceProcessor=byteDanceProcessor_] {
return byteDanceProcessor->initOpenGL();
});
isInitOpenGL = res.get();
} else {
isInitOpenGL = byteDanceProcessor_->initOpenGL();
}
return 0;
}
// Release OpenGL.
int ExtensionVideoFilter::stop() {
PRINTF_INFO("ExtensionVideoFilter::stop");
if (byteDanceProcessor_) {
byteDanceProcessor_->releaseOpenGL();
isInitOpenGL = false;
}
return 0;
}
// When mode is set to Async, the SDK and extension transfer video frames through pendVideoFrame and deliverVideoFrame.
rtc::IExtensionVideoFilter::ProcessResult ExtensionVideoFilter::pendVideoFrame(agora::agora_refptr<rtc::IVideoFrame> frame) {
if (!frame || !isInitOpenGL) {
return kBypass;
}
bool isAsyncMode = (mode_ == ProcessMode::kAsync);
if (isAsyncMode && byteDanceProcessor_ && control_ && invoker_id >= 0) {
threadPool_.PostTask(invoker_id, [videoFrame=frame, byteDanceProcessor=byteDanceProcessor_, control=control_] {
rtc::VideoFrameData srcData;
videoFrame->getVideoFrameData(srcData);
byteDanceProcessor->processFrame(srcData);
control->deliverVideoFrame(videoFrame);
});
return kSuccess;
}
return kBypass;
}
// When mode is set to Sync, the SDK and extension transfer video frames through adaptVideoFrame.
rtc::IExtensionVideoFilter::ProcessResult ExtensionVideoFilter::adaptVideoFrame(agora::agora_refptr<rtc::IVideoFrame> src,
agora::agora_refptr<rtc::IVideoFrame>& dst) {
if (!isInitOpenGL) {
return kBypass;
}
bool isSyncMode = (mode_ == ProcessMode::kSync);
if (isSyncMode && byteDanceProcessor_) {
rtc::VideoFrameData srcData;
src->getVideoFrameData(srcData);
byteDanceProcessor_->processFrame(srcData);
dst = src;
return kSuccess;
}
return kBypass;
}
// Set the property of the video filter.
int ExtensionVideoFilter::setProperty(const char *key, const void *buf,
size_t buf_size) {
PRINTF_INFO("setProperty %s %s", key, buf);
std::string stringParameter((char*)buf);
byteDanceProcessor_->setParameters(stringParameter);
return 0;
}
// Get the property of the video filter.
int ExtensionVideoFilter::getProperty(const char *key, void *buf, size_t buf_size) {
return -1;
}
}
}
To encapsulate the video filter into an extension, you need to implement the IExtensionProvider
interface. You can find the interface in the NGIAgoraExtensionProvider.h
file. The following methods from this interface must be implemented:
Enumerates your extensions that can be encapsulated. The SDK triggers this callback when loading the extension. In the callback, you need to return information about all of your extensions that can be encapsulated.
virtual void enumerateExtensions(ExtensionMetaInfo* extension_list,
int& extension_count) {
(void) extension_list;
extension_count = 0;
}
Parameter | Description |
---|---|
extension_list |
Extension information, including extension type and name. For details, see the definition of ExtensionMetaInfo. |
extension_count |
The total number of the extensions that can be encapsulated. |
The definition of ExtensionExtensionMetaInfo is as follows:
// EXTENSION_TYPE represents where the extension is located in the media transmission pipeline
enum EXTENSION_TYPE {
// Audio processing filter
AUDIO_FILTER,
// Video preprocessing filter
VIDEO_PRE_PROCESSING_FILTER,
// Video postprocessing filter
VIDEO_POST_PROCESSING_FILTER,
// Reserved for future use
AUDIO_SINK,
// Reserved for future use
VIDEO_SINK,
// Reserved for future use
UNKNOWN,
};
// Extension information, including extension type and name
struct ExtensionMetaInfo {
EXTENSION_TYPE type;
const char * extension_name;
};
If you specify VIDEO_PRE_PROCESSING_FILTER
or VIDEO_POST_PROCESSING_FILTER
as EXTENSION_TYPE, the SDK calls the createVideoFilter
method after the customer creates the IExtensionVideoProvider
object when initializing RtcEngine
.
Creates an video filter. You need to pass the IExtensionVideoFilter
instance in this method.
virtual agora_refptr < IExtensionVideoFilter > createVideoFilter(const char * name) {
return NULL;
}
After the IExtensionVideoFilter
instance is created, the extension processes video frames with methods in the IExtensionVideoFilter
class.
The following code sample shows how to use these APIs to encapsulate the video filter:
#include "ExtensionProvider.h"
#include "../logutils.h"
#include "VideoProcessor.h"
#include "plugin_source_code/JniHelper.h"
namespace agora {
namespace extension {
ExtensionProvider::ExtensionProvider() {
PRINTF_INFO("ExtensionProvider create");
byteDanceProcessor_ = new agora::RefCountedObject<ByteDanceProcessor>();
audioProcessor_ = new agora::RefCountedObject<AdjustVolumeAudioProcessor>();
}
ExtensionProvider::~ExtensionProvider() {
PRINTF_INFO("ExtensionProvider destroy");
byteDanceProcessor_.reset();
audioProcessor_.reset();
}
// Enumerate your extensions that can be encapsulated
void ExtensionProvider::enumerateExtensions(ExtensionMetaInfo* extension_list,
int& extension_count) {
extension_count = 1;
ExtensionMetaInfo i;
i.type = EXTENSION_TYPE::VIDEO_PRE_PROCESSING_FILTER;
i.extension_name = agora::extension::VIDEO_FILTER_NAME;
extension_list[0] = i;
}
// Create a video filter
agora_refptr<agora::rtc::IExtensionVideoFilter> ExtensionProvider::createVideoFilter(const char* name) {
PRINTF_INFO("ExtensionProvider::createVideoFilter %s", name);
auto videoFilter = new agora::RefCountedObject<agora::extension::ExtensionVideoFilter>(byteDanceProcessor_);
return videoFilter;
}
void ExtensionProvider::setExtensionControl(rtc::IExtensionControl* control){
}
}
}
After encapsulating the filter into an extension, you need to register and package it into a .aar
or .so
file, and submit it together with a file that contains the extension name, vendor name and filter name to Agora.
Register the extension with the macro REGISTER_AGORA_EXTENSION_PROVIDER
, which is in the AgoraExtensionProviderEntry.h
file.
Use this macro at the entrance of the extension implementation. When the SDK loads the extension, this macro automatically registers it to the SDK. For example:
REGISTER_AGORA_EXTENSION_PROVIDER(ByteDance, agora::extension::ExtensionProvider);
libagora-rtc-sdk-jni.so
fileIn CMakeLists.txt
, specify the path to save the libagora-rtc-sdk-jni.so
file in the downloaded SDK package according to the following table:
File | Path |
---|---|
64-bit libagora-rtc-sdk-jni.so |
AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v8a |
32-bit libagora-rtc-sdk-jni.so |
AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v7a |
Create a .java
or .md
file to provide the following information:
EXTENSION_NAME
: The name of the target link library used in CMakeLists.txt
. For example, for a .so
file named libagora-bytedance.so
, the EXTENSION_NAME
should be agora-bytedance
.EXTENSION_VENDOR_NAME
: The name of the extension provider, which is used for registering in the agora-bytedance.cpp
file.EXTENSION_FILTER_NAME
: The name of the filter, which is defined in ExtensionProvider.h
.Agora provides an Android sample project agora-simple-filter for developing audio and video filter extensions.