This page shows how to develop an audio extension that can be used together with the Agora Web SDK 4.x.
Agora provides the following abstract classes for developing an audio extension:
AudioExtension
: This class implements initialization-related functions, including creating an extension, setting log reports, and setting event reports.AudioProcessor
: This class implements audio processing capabilities, including receiving, processing, and delivering audio data.Ticker
: This class helps manage periodic tasks.Logger
: This class uploads logs to the Agora SDK.Reporter
: This class reports events to the Agora SDK.Before proceeding, ensure that your development environment meets the following requirements:
Integrate the extension development module (agora-rte-extension ) into your project via npm:
To install the required module, run the following command:
npm install --save agora-rte-extension
To import the required module, add the following code to your .js
file:
import {AudioExtension, AudioProcessor} from 'agora-rte-extension'
This section introduces the APIs you must implement in order to develop an audio extension. Optionally, based on the actual needs, you can implement certain auxiliary APIs to optimize the design and performance of your extension. For details, refer to See also.
Extension initialization is implemented through the AudioExtension
class. You need to implement the createProcessor
method of the AudioExtension
class:
createProcessor
abstract class AudioExtension<T extends AudioProcessor> {
createProcessor(): T;
}
Creates an AudioProcessor
instance.
The SDK calls this method when the app client calls extension.createProcessor
. You need to return the created AudioProcessor
instance in this method.
To implement audio processing, you need to implement the following steps:
onNode
method of theAudioProcessor
class and the getAudioContext
method of theAudioProcessorContext
class.output
method of theAudioProcessor
class.onNode
abstract onNode?(node: AudioNode, context: IAudioProcessorContext): void;
Reports that audio data from the previous node is received.
Parameters
node
: The previous AudioNode
. AudioNode
is an interface provided by the Web Audio API. For details, see Web Audio API interfaces.
context
: The context of the current audio processing pipeline.
output
output(track: MediaStreamTrack | AudioNode, context: IProcessorContext): void;
Outputs the processed audio data.
Parameters
track
: The processed audio data. Agora recommends using AudioNode
.
context
: The context of the current audio processing pipeline.
getAudioContext
getAudioContext(): Promise<AudioContext>;
Gets the AudioContext
of the current audio processing pipeline. AudioContext
is an interface provided by the Web Audio API. For details, see Web Audio API interfaces.
Agora provides a Web sample project ExtensionDemo for developing audio and video extensions.
The following code sample shows how to develop an audio extension:
class YourExtension extends AudioExtension<YourProcessor> {
// Create a Processor
protected _createProcessor(): YourProcessor {
return new YourProcessor();
}
}
class CustomAudioProcessor extends AudioProcessor {
// Receive audio data from the previous AudioNode
onNode(node: AudioNode, context: IAudioProcessorContext) {
// Create an AudioNode
const audioContext = context.getAudioContext();
const gainNode = audioContext.createGain();
// Connect to the previous AudioNode
node.connect(gainNode);
}
process() {
// TODO: Add the audio processing logic for your extension
// Output the processed audio
this.output(gainNode, context);
}
}
This section describes the auxiliary APIs for developing an audio extension.
name: string;
The name of theProcessor
.
enabled :boolean;
Whether theProcessor
is enabled.
public readonly ID:string;
The identifier of theProcessor
.
public get Kind():'video' | 'audio';
The type of the Processor
, which is categorized as video or audio.
protected context?: IProcessorContext;
This property allows the Processor
to request and recapture the original media stream.
abstract onPiped?(context: IProcessorContext): void;
Reports that a LocalVideoTrack
connects to the current media processing pipeline.
Note: This callback is not triggered if only Processor
s are connected``. For example: The app client calls processorA.pipe(processorB)
.
abstract onUnPiped?(): void;
Reports that the Processor
disconnects from the media processing pipeline.
abstract onEnableChange?(enabled: boolean): void | Promise<void>;
Reports that the Processor
is enabled or disabled.
protected inputNode?:AudioNode;
Audio received by the Processor
.
protected outputNode?:AudioNode;
Audio output by the Processor
.
interface IAudioProcessorContext extends IProcessorContext {
getAudioContext(): Promise<AudioContext>;
}
This class allows the AudioProcessor
to request and recapture the original audio stream.
public constructor(type:"Timer" | "RAF" | "Oscillator", interval: number):Ticker;
The constructor of the Ticker
class.
Parameters
type
: The following types are supported:Timer
: Using setTimeout
as the internal timer
of the extension.RAF
: Using requestAnimationFrame
as the internal timer of the extension. In most cases, this type of Ticker
has the best rendering performance.Oscillator
: Using Web Audio's ``OscillatorNode
as the internal timer of the extension. This type of Ticker
keeps running when the browser page is not being viewed.interval
: The interval
between two recurring callbacks. The Ticker
tries its best to execute at this interval, but might not be 100% accurate.public add(fn: Function): void;
Adds a timer task.
public remove():void;
Removes the added timer task.
public start():void;
Starts the timer.
public stop():void;
Stops the timer.
interface IExtensionLogger {
debug(...args: any): void;
error(...args: any): void;
info(...args: any): void;
warning(...args: any): void;
}
This class provides four levels of logs.
If the app client chooses to upload the logs when calling AgoraRTC.registerExtension
, the status of the extension that implements the Logger
class is updated.
public reportApiInvoke<T>(params: ReportApiInvokeParams): AgoraApiExecutor<T>;
Reports events related to API calls to the SDK.
The definitions of ReportApiInvokeParams
and AgoraApiExecutor
are as follows:
interface ReportApiInvokeParams {
// The API name
name: string;
// Parameters or options related to the API
options: any;
// Whether to report the result of the API call
reportResult?: boolean;
// How long is counted as timeout for the API call
timeout?: number;
}
interface AgoraApiExecutor<T> {
// The API call succeeds
onSuccess: (result: T) => void;
// The API call fails
onError: (err: Error) => void;
}