This page shows you how to use the APIs provided by the Agora SDK v4.0.0 to build an audio filter extension for Android.
To build an audio filter extension, you use the following APIs:
IAudioFilter
: This interface implements the function of receiving, processing, and delivering audio data.IExtensionProvider
: This interface encapsulates the functions in IAudioFilter
into an extension.Before proceeding, ensure that your development environment meets the following requirements:
Follow these steps to integrate the SDK into your project:
rtc/sdk/low_level_api/include
directory and save them under the directory of your project file.This section describes the APIs required for implementing an audio filter extension.
Use the IAudioFilter
interface to implement an audio filter. You can find the interface in the NGIAgoraMediaNode.h
file. You need to implement this interface first, and you must implement at least the following methods from this interface:
adaptAudioFrame
setEnabled
isEnabled
setProperty
getProperty
getName
getPreferredSampleRate
(Optional)getPreferredChannelNumbers
(Optional)Adapts the audio frame. This is the core method of the IAudioFilter
interface. By calling this method, the SDK processes audio frames from inAudioFrame
and returns the adapted frames with adaptedFrame
. This method supports audio data in the PCM format only.
virtual bool adaptAudioFrame(const media::base::AudioPcmFrame& inAudioFrame,
media::base::AudioPcmFrame& adaptedFrame) = 0;
Parameter | Description |
---|---|
inAudioFrame |
An input parameter. The pointer to the audio frames to be processed. |
adaptedFrame |
An output parameter. The pointer to the processed audio frames. |
Enables or disables the audio filter.
virtual void setEnabled(bool enable) {}
Parameter | Description |
---|---|
enable |
Whether to enable the audio filter: |
Checks whether the audio filter is enabled.
virtual bool isEnabled() { return true; }
Returns
Whether the audio filter is enabled:
Sets the property of the audio filter. When an app client calls setExtensionProperty,
the SDK triggers this callback. In the callback, you need to return the property of the audio filter.
size_t setProperty(const char* key, const void* buf, size_t buf_size)
Parameter | Description |
---|---|
key |
The key of the property. |
buf |
The buffer of the property in the JSON format. You can use the open source nlohmann/json library for the serialization and deserialization between the C++ struct and the JSON string. |
buf_size |
The size of the buffer. |
Gets the property of the audio filter. When the app client calls getExtensionProperty
, the SDK calls this method to get the property of the audio filter.
size_t getProperty(const char* key, char* property, size_t buf_size)
Parameter | Description |
---|---|
key |
The key of the property. |
property |
The pointer to the property. |
buf_size |
The size of the buffer. |
Retrieves the vendor name. You need to set the VENDOR_NAME
in the return value of this method.
virtual const char * getName() const = 0;
Retrieves the preferred sample rate of the audio filter.
This method is optional. If you specify a sample rate in the return value of this method, the SDK resamples the audio data accordingly before sending it to the audio filter.
virtual int getPreferredSampleRate() { return 0; };
Retrieves the preferred number of channels of the audio filter.
This method is optional. If you specify a number in the return value of this method, the SDK resamples the audio data accordingly before sending it to the audio filter.
virtual int getPreferredChannelNumbers() { return 0; };
The following code sample shows how to use these methods together to implement an audio filter extension:
// After receiving the audio frames to be processed, call adaptAudioFrame to process the audio frames.
bool ExtensionAudioFilter::adaptAudioFrame(const media::base::AudioPcmFrame &inAudioPcmFrame,
media::base::AudioPcmFrame &adaptedPcmFrame) {
return audioProcess_->processFrame(inAudioPcmFrame, adaptedPcmFrame) == 0;
}
// Call setProperty to set the property of the audio filter.
int ExtensionAudioFilter::setProperty(const char* key, const void* buf, int buf_size) {
std::string str_volume = "100";
if (std::string(key) == "volume") {
str_volume = std::string(static_cast<const char*>(buf), buf_size);
}
int int volume_ = atoi(str_volume.c_str());
audioProcessor_->setVolume(int_volume_);
return ERR_OK;
}
// Call getProperty to get the property of the audio filter.
int ExtensionAudioFilter::getProperty(const char* key, void* buf, int buf_size) const override {return ERR_OK; }
// Call setEnabled to enable the audio filter.
void ExtensionAudioFilter::setEnabled(bool enable) override { enabled_ = enable; }
// Call isEnabled to check whether the audio filter is enabled.
bool ExtensionAudioFilter::isEnabled() const override {return enabled_; }
// Set the vendor name in the return value of getName.
const char* getName() const override { return filterName_.c_str(); }
// (Optional) Specify the audio sample rate in the return value of getPreferredSampleRate
const char* ExtensionAudioFilter::getPreferredSampleRate() override { return 48000; }
// (Optional) Specify the number of audio channels in the return value of getPreferredChannelNumbers
int ExtensionAudioFilter::getPreferredChannelNumbers() override { return 2; };
To encapsulate the audio filter into an extension, you need to implement the IExtensionProvider
interface. You can find the interface in the NGIAgoraExtensionProvider.h
file. The following methods from this interface must be implemented:
Enumerates your extensions that can be encapsulated. The SDK triggers this callback when loading the extension. In the callback, you need to return information about all of your extensions that can be encapsulated.
virtual void enumerateExtensions(ExtensionMetaInfo* extension_list,
int& extension_count) {
(void) extension_list;
extension_count = 0;
}
Parameter | Description |
---|---|
extension_list |
Extension information, including extension type and name. For details, see the definition of ExtensionMetaInfo. |
extension_count |
The total number of the extensions that can be encapsulated. |
The definition of ExtensionExtensionMetaInfo is as follows:
// EXTENSION_TYPE represents where the extension is located in the media transmission pipeline
enum EXTENSION_TYPE {
// Audio processing filter
AUDIO_FILTER,
// Video preprocessing filter
VIDEO_PRE_PROCESSING_FILTER,
// Video postprocessing filter
VIDEO_POST_PROCESSING_FILTER,
// Reserved for future use
AUDIO_SINK,
// Reserved for future use
VIDEO_SINK,
// Reserved for future use
UNKNOWN,
};
// Extension information, including extension type and name
struct ExtensionMetaInfo {
EXTENSION_TYPE type;
const char* extension_name;
};
If you specify AUDIO_FILTER
as EXTENSION_TYPE, after the customer creates the IExtensionProvider
object when initializing RtcEngine
, the SDK calls the createAudioFilter
method, and you need to return the IAudioFilter
instance in this method.
Creates an audio filter. You need to pass the IAudioFilter
instance in this method.
virtual agora_refptr<IAudioFilter> createAudioFilter()
After creating an audio filter object, the extension processes the input audio frames with methods in IAudioFilter
.
Sets the extension control.
virtual void setExtensionControl(IExtensionControl* control)
After calling this method, you need to maintain the IExtensionControl
object returned in this method. The IExtensionControl
object manages the interaction between the extension and the SDK by triggering callbacks and sending logs. For example, if you have called fireEvent
in IExtensionControl
:
void ByteDanceProcessor::dataCallback(const char* data){
if (control_ != nullptr) {
control_->fireEvent(id_, "beauty", data);
}
}
And if the app registers the IMediaExtensionObserver
class when initializing RtcEngine
, the SDK triggers the following callback on the app client:
@Override
public void onEvent(String vendor, String key, String value) {
...
}
The following code sample shows how to use these methods to encapsulate the audio filter:
void ExtensionProvider::enumerateExtensions(ExtensionMetaInfo* extension_list,
int& extension_count) {
extension_count = 2;
//Declare a Video Filter, and IExtensionProvider::createVideoFilter will be called
ExtensionMetaInfo i;
i.type = EXTENSION_TYPE::VIDEO_PRE_PROCESSING_FILTER;
i.extension_name = agora::extension::VIDEO_FILTER_NAME;
extension_list[0] = i;
//Declare an Audio Filter, and IExtensionProvider::createAudioFilter will be called
ExtensionMetaInfo j;
j.type = EXTENSION_TYPE::AUDIO_FILTER;
j.extension_name = agora::extension::AUDIO_FILTER_NAME;
extension_list[1] = j;
}
agora_refptr<agora::rtc::IAudioFilter> ExtensionProvider::createAudioFilter(const char* name) {
PRINTF_INFO("ExtensionProvider::createAudioFilter %s", name);
auto audioFilter = new agora::RefCountedObject<agora::extension::ExtensionAudioFilter>(name, audioProcessor_);
return audioFilter;
}
void ExtensionAudioProvider::setExtensionControl(rtc::IExtensionControl* control){
audioProcessor_->setExtensionControl(control);
}
After encapsulating the filter into an extension, you need to register and package it into a .aar
or .so
file, and submit it together with a file that contains the extension name, vendor name and filter name to Agora.
Register the extension with the macro REGISTER_AGORA_EXTENSION_PROVIDER
, which is in the AgoraExtensionProviderEntry.h
file.
Use this macro at the entrance of the extension implementation. When the SDK loads the extension, this macro automatically registers it to the SDK. For example:
REGISTER_AGORA_EXTENSION_PROVIDER(ByteDance, agora::extension::ExtensionProvider);
libagora-rtc-sdk-jni.so
fileIn CMakeLists.txt
, specify the path to save the libagora-rtc-sdk-jni.so
file in the downloaded SDK package according to the following table:
File | Path |
---|---|
64-bit libagora-rtc-sdk-jni.so |
AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v8a |
32-bit libagora-rtc-sdk-jni.so |
AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v7a |
Create a .java
or .md
file to provide the following information:
EXTENSION_NAME
: The name of the target link library used in CMakeLists.txt
. For example, for a .so
file named libagora-bytedance.so
, the EXTENSION_NAME
should be agora-bytedance
.EXTENSION_VENDOR_NAME
: The name of the extension provider, which is used for registering in the agora-bytedance.cpp
file.EXTENSION_FILTER_NAME
: The name of the filter, which is defined in ExtensionProvider.h
.Agora provides an Android sample project agora-simple-filter for developing audio and video filter extensions.