The agora-extension-virtual-background
extension is used together with the Agora Web SDK (v4.10.0 or later) to implement the virtual background feature. This feature allows users to blur their actual background, or replace it with a solid color, an image, or a video. This extension is applicable to scenarios such as online conferences, online classes, and live streaming. It helps to protect personal privacy and reduce the audience getting distracted.
Feature | Example |
---|---|
Blurred background and image background | |
Video/Animated background |
This extension works best when there is only one user in the video captured by the camera.
The browser support for the virtual background extension is as follows:
The virtual background feature has high performance requirements. Make sure your computer meets the following requirements:
When using the extension on a laptop, Agora recommends choosing the high performance mode or the balanced mode. The battery saver mode might not suffice.
As of v1.0.0-beta-3, the extension supports setting a video as the virtual background. The video must be in a format supported by the HTML <video>
element. Additionally, Agora recommends the following tips:
If you use multiple media processing extensions at the same time, Agora recommends an Intel i5 quad-core CPU or better. When multiple extensions are enabled, if another app is consuming more system resources, the audio and video might freeze in your app.
A typical transmission pipeline in the Agora Web SDK consists of a chain of procedures, including capture, preprocessing, encoding, transmitting, decoding, post-processing, and playback. In the preprocessing stage, extensions can modify the audio and video data in the pipeline to implement features such as virtual background, noise cancellation, and image enhancement.
Integrate the virtual background extension, and implement the virtual background feature, as follows:
Refer to the appropriate Quickstart Guide to integrate the Web SDK (v4.10.0 or later) and implement the basic real-time communication functions in your project.
Integrate the virtual background extension into your project via npm:
To install the virtual background extension, run the following command :
npm install agora-extension-virtual-background
To import the virtual background extension, use any of the following methods:
Method one: Add the following code to the JavaScript file:
import VirtualBackgroundExtension from "agora-extension-virtual-background";
Method two: Use the Script tag in the HTML file. Once imported, the VirtualBackgroundExtension
object can be used directly in JavaScript files.
<script src="./agora-extension-virtual-background.js"></script>
Dynamically load the Wasm dependency: The virtual background extension depends on a Wasm file. Find the Wasm file in node_modules/agora-extension-virtual-background/wasms
. You need to publish the Wasm file to a CDN or static resource server. You need to pass in the URL of the Wasm file when calling processor.init
in subsequent steps. The extension dynamically loads the Wasm file.
Register the virtual background extension: After calling AgoraRTC.createClient
to create a client object, new a VirtualBackgroundExtension
object and pass in the VirtualBackgroundExtension
object when calling AgoraRTC.registerExtensions
.
// Create a client object
var client = AgoraRTC.createClient({mode: "rtc", codec: "vp8"});
// Create a VirtualBackgroundExtension instance
const extension = new VirtualBackgroundExtension();
// Register the extension
AgoraRTC.registerExtensions([extension]);
Initialize the virtual background extension:
Call extension.createProcessor
to create a VirtualBackgroundProcessor
instance.
processor = extension.createProcessor();
Call processor.init
to initialize the extension. In this method, you need to pass in the URL of the Wasm module. If resource loading or extension initialization fails, this method throws an exception. Catch this exception, and handle it accordingly.
await processor.init("./assets/wasms");
After creating a local camera track, call videoTrack.pipe
and specify videoTrack.processorDestination
to inject the extension into the SDK's media processing pipeline.
localTracks.videoTrack.pipe(processor).pipe(localTracks.videoTrack.processorDestination);
Call processor.setOptions
to choose the virtual background type and make settings. Agora supports the following settings:
Set a solid color as the background: Set the type
parameter as "color"
, and set the color
parameter.
processor.setOptions({type: 'color', color: '#00ff00'});
Set an image as the background: Set the type
parameter as "img"
, and set the source
parameter to specify the image object.
processor.setOptions({type: 'img', source: HTMLImageElement});
Blur the user's original background: Set the type
parameter as "blur"
and set the blurDegree
parameter to choose the blurring degree, low (1
), medium (2
), or high (3
).
processor.setOptions({type: 'blur', blurDegree: 2});
Set a video as the background: Set the type
parameter as "video"
, and set the source
parameter to specify the video object.
processor.setOptions({type: 'video', source: HTMLVideoElement});
Call processor.enable
to enable the virtual background feature.
await processor.enable();
If you do not call setOptions
before calling this method, the default behavior of the SDK is to blur users' actual background with the blurring degree set as 1. After enabling the virtual background feature, to switch the virtual background type, just call setOptions
.
According to your business scenario, call videoTrack.unpipe
to remove the extension's processor from the current video track.
localTracks.videoTrack.unpipe()
import AgoraRTC from "agora-rtc-sdk-ng";
import VirtualBackgroundExtension from "agora-extension-virtual-background";
// Create a client object
var client = AgoraRTC.createClient({mode: "rtc", codec: "vp8"});
// Create a VirtualBackgroundExtension instance
const extension = new VirtualBackgroundExtension();
// Register the extension
AgoraRTC.registerExtensions([extension]);
let processor = null;
var localTracks = {
videoTrack: null,
audioTrack: null
};
// Initialization
async function getProcessorInstance() {
if (!processor && localTracks.videoTrack) {
// Create a VirtualBackgroundProcessor instance
processor = extension.createProcessor();
try {
// Initialize the extension and pass in the URL of the Wasm file
await processor.init("./assets/wasms");
} catch(e) {
console.log("Fail to load WASM resource!");return null;
}
// Inject the extension into the video processing pipeline in the SDK
localTracks.videoTrack.pipe(processor).pipe(localTracks.videoTrack.processorDestination);
}
return processor;
}
// Join a channel
async function join() {
// Add event listeners
client.on("user-published", handleUserPublished);
client.on("user-unpublished", handleUserUnpublished);
[options.uid, localTracks.audioTrack, localTracks.videoTrack] = await Promise.all([
// Join a channel
client.join(options.appid, options.channel, options.token || null),
// Create a local microphone track and camera track
localTracks.audioTrack || AgoraRTC.createMicrophoneAudioTrack(),
localTracks.videoTrack || AgoraRTC.createCameraVideoTrack({encoderConfig: '720p_4'})
]);
// Play local tracks
localTracks.videoTrack.play("local-player");
}
// Set a solid color as the background
async function setBackgroundColor() {
if (localTracks.videoTrack) {
document.getElementById("loading").style.display = "block";
let processor = await getProcessorInstance();
try {
processor.setOptions({type: 'color', color: '#00ff00'});
await processor.enable();
} finally {
document.getElementById("loading").style.display = "none";
}
virtualBackgroundEnabled = true;
}
}
// Blur the user's actual background
async function setBackgroundBlurring() {
if (localTracks.videoTrack) {
document.getElementById("loading").style.display = "block";
let processor = await getProcessorInstance();
try {
processor.setOptions({type: 'blur', blurDegree: 2});
await processor.enable();
} finally {
document.getElementById("loading").style.display = "none";
}
virtualBackgroundEnabled = true;
}
}
// Set an image as the background
async function setBackgroundImage() {
const imgElement = document.createElement('img');
imgElement.onload = async() => {
document.getElementById("loading").style.display = "block";
let processor = await getProcessorInstance();
try {
processor.setOptions({type: 'img', source: imgElement});
await processor.enable();
} finally {
document.getElementById("loading").style.display = "none";
}
virtualBackgroundEnabled = true;
}
imgElement.src = '/images/background.png';
}
createProcessor(): IVirtualBackgroundProcessor;
Creates an VirtualBackgroundProcessor
object.
init(wasmDir: string): Promise<void>;
Initializes the extension.
If the initialization of the extension fails due to the failure to access the Wasm file, this method throws an exception. Agora recommends that you disable the virtual background feature catching the exception.
If the Wasm file is deployed on third-party services such as CDN and OSS across domains, you need to enable cross-domain access. For example, when deploying Nginx servers across domains, to enable cross-domain access, add the following configurations:
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' "true";
Parameters:
wasmDir
: The URL where the virtual background WASM module is located (without the WASM
filename).setOptions(options: VirtualBackgroundEffectOptions): void;
Chooses the virtual background type, and sets parameters.
Parameters:
options
: The virtual background options. See VirtualBackgroundEffectOptions.enable(): void | Promise<void>;
Enables the virtual background feature.
If you do not call setOptions
before calling this method, the default behavior of the SDK is to blur users' actual background with the blurring degree set as 1.
disable(): void | Promise<void>;
Disables the virtual background feature.
onoverload?: () => void;
When the system performance cannot meet the processing requirements, the SDK triggers onoverload
.
Agora recommends calling disable
in this event to disable virtual background and adding a UI prompt.
Virtual background types and settings. Used in the setOptions method.
export type VirtualBackgroundEffectOptions = {
type: string,
color?: string;
source?: HTMLImageElement | HTMLVideoElement;
blurDegree?: Number;
};
Properties:
type
: String. Choose the virtual background type:
"color"
: Sets a solid color as the background."img"
: Sets an image as the background."blur"
: Blurs the user's original background."video"
: Sets a video as the animated background.color
: String. When you set type
as "color"
, set this parameter to specify the color. The value must be a valid CSS color such as "white"
, "#00ff00"
, or "RGB(255, 0, 0)"
.
source
: The HTMLImageElement or HTMLVideoElement object. When you set type
as "img"
or "video"
, you can set a custom background image or video through this parameter.
crossOrigin
property of the HTMLImageElement
object as "anonymous"
. HTMLImageElement
object, Agora recommends calling setOptions
in the onload
callback of the HTMLImageElement
object, otherwise, the background momentarily turns black.mute
and loop
HTML attributes.blurDegree
: Number. When you set type
as "blur"
, set this parameter to choose the blurring degree:1
: Low2
: Medium3
: High