This page shows how to develop a video extension that can be used together with the Agora Web SDK 4.x.
Agora provides the following abstract classes for developing a video extension:
Extension
: This class implements initialization-related functions, including creating an extension, setting log reports, and setting event reports.VideoProcessor:
This class implements video processing capabilities, including receiving, processing, and delivering video data.Ticker
: This class helps manage periodic tasks.Logger
: This class uploads logs to the Agora SDK.Reporter
: This class reports events to the Agora SDK.Before proceeding, ensure that your development environment meets the following requirements:
Integrate the extension development module (agora-rte-extension ) into your project via npm:
To install the required module, run the following command:
npm install --save agora-rte-extension
To import the required module, add the following code to your .js
file:
import {AudioExtension, AudioProcessor} from 'agora-rte-extension'
This section introduces the APIs you must implement in order to develop a video extension. Optionally, based on the actual needs, you can implement certain auxiliary APIs to optimize the design and performance of your extension. For details, refer to See also.
Extension initialization is implemented through the Extension
class. You need to implement the createProcessor
method of the Extension
class:
createProcessor
abstract class Extension<T extends BaseProcessor> {
createProcessor(): T;
}
Creates a VideoProcessor
instance.
The SDK calls this method when the app client calls extension.createProcessor
. You need to return the created VideoProcessor
instance in this method.
To implement video processing, you need to implement the following steps:
onTrack
method of theVideoProcessor
class.output
method of the VideoProcessor
class.onTrack
abstract onTrack?(track: MediaStreamTrack, context: IProcessorContext): void;
Reports that video data from the previous node is received.
Parameters
track
: The previous MediaStreamTrack
. MediaStreamTrack
is an interface provided by Web API. For details, see MediaStreamTrack.
context
: The context of the current video processing pipeline.
output
output(track: MediaStreamTrack, context: IProcessorContext): void;
Outputs the processed video data.
Parameters
track
: The processed MediaStreamTrack
.
context
: The context of the current video processing pipeline.
Agora provides a Web sample project ExtensionDemo for developing audio and video extensions.
The following code sample shows how to develop a video extension:
class YourExtension extends Extension<YourProcessor> {
// Create a Processor
protected _createProcessor(): YourProcessor {
return new YourProcessor();
}
}
class CustomVideoProcessor extends VideoProcessor {
// Receive video data from the previous MediaStreamTrack
onTrack(track: MediaStreamTrack, context: IProcessorContext) {
this.videoElement.srcObject = new MediaStream([track]);
this.videoElement.play();
this.loop();
}
// Continue to extract video frames as ImageData for video processing
loop() {
this.ctx.drawImage(this.videoElement, 0, 0);
const imageData = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
this.process(imageData);
requestAnimationFrame(() => this.loop());
}
process() {
//TODO: Add the video processing logic for your extension
}
doneProcessing() {
// Assemble MediaStreamTrack
const msStream = this.canvas.captureStream(30);
const outputTrack = msStream.getVideoTracks()[0];
// Output the processed video
if (this.context) {
this.output(outputTrack, this.context);
}
}
}
This section describes the auxiliary APIs for developing a video extension.
name: string;
The name of theProcessor
.
enabled :boolean;
Whether theProcessor
is enabled.
public readonly ID:string;
The identifier of theProcessor
.
public get Kind():'video' | 'audio';
The type of the Processor
, which is categorized as video or audio.
protected context?: IProcessorContext;
This property allows the Processor
to request and recapture the original media stream.
abstract onPiped?(context: IProcessorContext): void;
Reports that a LocalVideoTrack
connects to the current media processing pipeline.
Note: This callback is not triggered if only Processor
s are connected``. For example: The app client calls processorA.pipe(processorB)
.
abstract onUnPiped?(): void;
Reports that the Processor
disconnects from the media processing pipeline.
abstract onEnableChange?(enabled: boolean): void | Promise<void>;
Reports that the Processor
is enabled or disabled.
protected inputTrack?:MediaStreamTrack;
Video received by the extension from the previous Processor
or LocalVideoTrack
.
protected outputTrack?:MediaStreamTrack;
Video output by the Processor
. Calling the output
method automatically sets the outputTrack
property.
public constructor(type:"Timer" | "RAF" | "Oscillator", interval: number):Ticker;
The constructor of the Ticker
class.
Parameters
type
: The following types are supported:Timer
: Using setTimeout
as the internal timer
of the extension.RAF
: Using requestAnimationFrame
as the internal timer of the extension. In most cases, this type of Ticker
has the best rendering performance.Oscillator
: Using Web Audio's ``OscillatorNode
as the internal timer of the extension. This type of Ticker
keeps running when the browser page is not being viewed.interval
: The interval
between two recurring callbacks. The Ticker
tries its best to execute at this interval, but might not be 100% accurate.public add(fn: Function): void;
Adds a timer task.
public remove():void;
Removes the added timer task.
public start():void;
Starts the timer.
public stop():void;
Stops the timer.
interface IExtensionLogger {
debug(...args: any): void;
error(...args: any): void;
info(...args: any): void;
warning(...args: any): void;
}
This class provides four levels of logs.
If the app client chooses to upload the logs when calling AgoraRTC.registerExtension
, the status of the extension that implements the Logger
class is updated.
public reportApiInvoke<T>(params: ReportApiInvokeParams): AgoraApiExecutor<T>;
Reports events related to API calls to the SDK.
The definitions of ReportApiInvokeParams
and AgoraApiExecutor
are as follows:
interface ReportApiInvokeParams {
// The API name
name: string;
// Parameters or options related to the API
options: any;
// Whether to report the result of the API call
reportResult?: boolean;
// How long is counted as timeout for the API call
timeout?: number;
}
interface AgoraApiExecutor<T> {
// The API call succeeds
onSuccess: (result: T) => void;
// The API call fails
onError: (err: Error) => void;
}