With an AI-powered noise reduction algorithm, the AI Denoiser extension is used together with the Agora Web SDK (v4.10.0 or later) to suppress hundreds of types of noise and reduce distortion for human voices when multiple people speak simultaneously. In scenarios such as online meetings, online chat rooms, video consultations with doctors, and online gaming, the AI Denoiser extension makes virtual communication as smooth as face-to-face communication.
The AI Denoiser extension suppresses the following types of noise:
The AI Denoiser extension supports the following noise reduction strategies:
Currently, the AI Denoiser extension has the following limitations:
If the sample rate of the input signal is not 16 KHz, the extension downsamples the signal to 16 KHz, removes noise, and resamples the output signal to the original sample rate. This means the audio data above 8 KHz is removed in the output signal.
In some unknown scenarios, the extension could cause audio quality to decrease by a certain degree.
When multiple people speak at the same time, the audio quality of lowest human voices could be decreased by a certain degree.
If only some of the audio tracks on the current web page enable the extension, the audio tracks that do not enable the extension could be affected because the extension turns on AEC and AGC and turns off NS in the browser.
While the extension supports Safari later than 14.1, Agora does not recommend using the extension on Safari because of Safari's performance issues.
Currently, the extension does not support browsers on mobile devices.
If you use multiple media processing extensions at the same time, Agora recommends an Intel i5 quad-core CPU or better. When multiple extensions are enabled, if another app is consuming more system resources, the audio and video might freeze in your app.
A typical transmission pipeline in the Agora Web SDK consists of a chain of procedures, including capture, preprocessing, encoding, transmitting, decoding, post-processing, and playback. In the preprocessing stage, extensions can modify the audio and video data in the pipeline to implement features such as virtual background and noise cancellation.
To use the AI Denoiser extension, you must meet the following requirements:
Follow these steps to integrate the AI Denoiser extension and implement the noise reduction feature:
Refer to the appropriate Quickstart Guide to integrate the Web SDK (v4.10.0 or later) and implement the basic real-time communication functions in your project.
Integrate the AI Denoiser extension (agora-extension-ai-denoiser) into your project via npm:
npm install agora-extension-ai-denoiser
.js
file:import {AIDenoiserExtension} from "agora-extension-ai-denoiser";
Dynamically load the Wasm and JS dependencies: The AI Denoiser extension depends on a few Wasm and JS files. To ensure that the browser can load and execute these files, you need to follow these steps:
node_modules/agora-extension-ai-denoiser/external
directory to the CDN or static resource server, and put them under one public path. In subsequent steps, you need to pass in the public path URL to create anAIDenoiserExtension
instance. The extension then dynamically loads these files.'wasm-unsafe-eval'
in the script-src
options. For example:<meta http-equiv="Content-Security-Policy" content="script-src 'self' 'wasm-unsafe-eval'">
'unsafe-eval'
in the script-src
options.Register the AI Denoiser extension: Call the AgoraRTC.registerExtensions
method, and pass in the created AIDenoiserExtension
instance. Optionally, listen for the callback reporting that the Wasm and JS files fail to load.
AIDenoiserExtension
instance only.// Create an AIDenoiserExtension instance, and pass in the host URL of the Wasm and JS files
const denoiser = new AIDenoiserExtension({assetsPath:'./external'});
// Register the extension
AgoraRTC.registerExtensions([denoiser]);
// (Optional) Listen for the callback reporting that the Wasm and JS files fail to load
denoiser.onloaderror = (e) => {
// If the Wasm and JS files fail to load, you can disable the plugin, for example:
// openDenoiserButton.enabled = false;
console.log(e);
}
Create an IAIDenoiserProcessor
instance: Call the createProcessor
method to create a processor
, and set whether to enable the extension by default. Optionally, you can listen for the callback reporting that the noise reduction process takes too long.
// Create a processor
const processor = denoiser.createProcessor();
// Enable the extension by default
processor.enable();
// Disable the extension by default
// processor.disable();
// (Optional) Listen for the callback reporting that the noise reduction process takes too long
processor.onoverload = async () => {
console.log("overload!!!");
// If noise reduction takes too long, turn off the extension
await processor.disable();
}
Inject the extension to the audio processing pipeline: Call the pipe
method and specify the processorDestination
property.
// Create a local video track
const audioTrack = await AgoraRTC.createMicrophoneAudioTrack();
// Inject the extension to the audio processing pipeline
audioTrack.pipe(processor).pipe(audioTrack.processorDestination);
await processor.enable();
Enable or disable the extension as needed: Call the enable
or disable
methods.
() => {
if (processor.enabled) {
await processor.disable();
} else {
await processor.enable();
}
}
Dump audio data from the noise reduction process: Call the dump
method, and listen for the ondump
and ondumpend
callbacks.
processor.ondump = (blob, name) => {
// Dump the audio data to a local folder in WAV format
const objectURL = URL.createObjectURL(blob);
const tag = document.createElement("a");
tag.download = name + ".wav";
tag.href = objectURL;
tag.click();
}
processor.ondumpend = () => {
console.log("dump ended!!");
}
processor.dump();
createProcessor(): IAIDenoiserProcessor;
Creates an IAIDenoiserProcessor
instance.
onloaderror?: () => void;
Reports when the Wasm and JS files fail to load.
get kind(): 'video' | 'audio';
The processor type, which is categorized as video or audio.
get enabled(): boolean;
Whether the extension is enabled:
enable(): void | Promise<void>;
Enables the AI Denoiser extension.
disable(): void | Promise<void>;
Disables the AI Denoiser extension.
dump(): void;
Dumps audio data in order to help troubleshoot noise reduction issues.
Calling this method triggers the ondump
callback nine times, returning nine WAV files (see the description below) for the audio data processed by the extension 30 seconds before and 60 seconds after the method call, then triggers the ondumpend
callback to inform you that the audio data ends dumping.
Audio file description
Notes
ondump
callback is triggered nine times, the dumping process ends immediately and the ondumpend
callback is triggered. In this case, fewer than nine audio data files are returned.ondump?: (blob: Blob, name: string) => void;
Reports that audio data is dumped.
This callback returns the following parameters:
blob
: The audio data file.name
: The name
of the audio data file.ondumpend?: () => void;
Reports that audio data ends dumping.
onoverload?: () => void;
Reports that the noise reduction process takes too long. This callback could be triggered by mistake and is for reference only.
export interface AIDenoiserExtensionOptions {
assetsPath: string
}
Options for initializing the AI Denoiser extension:
assetsPath
: The host URL of the .wasm and .js files required by the AI Denoiser extension.