Agora provides spatial audio effects to give users an immersive audio experience in scenarios such as esports competitions and online conferences.
For the local user to hear a remote user's spatial audio effects, you need to calculate the relative positions of the local and remote users on your own, and then call updateSelfPosition
and updateRemotePosition
to update the spatial coordinates of the local and remote users respectively.
For the user to hear the spatial audio effects when playing an audio file locally, you need to call updateSelfPosition
and updatePlayerPositionInfo
to update the spatial coordinates of the local user and the media player.
To integrate the spatial audio extension and implement the spatial audio effect feature, follow these steps :
Refer to the appropriate Quickstart Guide to integrate the Web SDK (v4.13.0 or later) and implement the basic real-time communication functions in your project.
Integrate the spatial audio extension (agora-extension-spatial-audio) into your project via npm.
npm install agora-extension-spatial-audio
.js
file:import {SpatialAudioExtension} from "agora-extension-spatial-audio";
Dynamically load the Wasm and JS dependencies: The spatial audio extension depends on a few Wasm and JS files. To ensure that the browser can load and execute these files, you need to follow these steps:
Publish the files located in the node_modules/agora-extension-spatial-audio/external
directory to the CDN or static resource server, and put them under one public path. In subsequent steps, you need to pass in the public path URL to create an SpatialAudioExtension
instance. The extension then dynamically loads these files.
If you have enabled the Content Security Policy (CSP), because Wasm files are not allowed to load in Chrome and Edge by default, you need to configure the CSP as follows:
For versions later than Chrome 97 and Edge 97 (Chrome 97 and Edge 97 included): Add wasm-unsafe-eval
in the script-src
options. For example:
<meta http-equiv="Content-Security-Policy" content="script-src 'self' 'wasm-unsafe-eval'">
For versions earlier than Chrome 97 and Edge 97: Add unsafe-eval
in the script-src
options.
Register the spatial audio extension: Call the AgoraRTC.registerExtensions
method, and pass in the created SpatialAudioExtension
instance.
const extension = new SpatialAudioExtension({
assetsPath: yourDeployPath, // for example, yourDeployPath = './resources/external/'
});
AgoraRTC.registerExtensions([extension]);
When subscribing to a remote user, implement the user's spatial audio effect as follows:
SpatialAudioProcessor
instance: Call the createProcessor
method to create.pipe
method and specify the processorDestination
property.async function subscribe(user, mediaType) {
await client.subscribe(user, mediaType);
console.log("subscribe success");
if (mediaType === "audio") {
const processor = extension.createProcessor();
const track = user.audioTrack;
track.pipe(processor).pipe(track.processorDestination);
track.play();
}
}
(Optional) When playing audio files such as background music, accompaniment, and NPC voice locally, implement the spatial audio effect as follows:
SpatialAudioProcessor
instance: Call the createProcessor
method to create.pipe
method and specify the processorDestination
property.var localPlayerSound = [
// './resources/1.mp3',
"./resources/2.mp3",
// './resources/3.mp3',
// './resources/4.mp3',
];
async function localPlayerStart() {
for (let i = 0; i < localPlayerSound.length; i++) {
const processor = extension.createProcessor();
const track = await AgoraRTC.createBufferSourceAudioTrack({source: localPlayerSound[i]});
track.startProcessAudioBuffer({loop: true});
track.pipe(processor).pipe(track.processorDestination);
track.play();
}
}
The API call sequence is as follows:
createProcessor(): SpatialAudioProcessor;
Create a SpatialAudioProcessor
instance.
public updateSelfPosition(position: position, forward: forward, right: right, up: up): SpatialAudioErrorCode
Updates the spatial position of the local user.
After a successful call of this method, the SDK updates the spatial audio effect parameters according to the relative position changes of the local user, all remote users, and the media player (if any).
Parameters:
position
: The coordinates in the world coordinate system. This parameter is an array of length 3, and the three values represent the front, right, and top coordinates in turn.forward
: The unit vector of the x axis in the world coordinate system. This parameter is an array of length 3, and the three values represent the front, right, and top coordinates in turn.right
: The unit vector of the y axis in the coordinate system. This parameter is an array of length 3, and the three values represent the front, right, and top coordinates in turn.up
: The unit vector of the z axis in the coordinate system. This parameter is an array of length 3, and the three values represent the front, right, and top coordinates in turn.Return values
See SpatialAudioErrorCode.
const mockLocalUserNewPosition = {
// For example, mockLocalUserNewPosition can be produced by dragging the local user's avatar to walk in a 3D scene
position: [1, 1, 1],
forward: [1, 0, 0],
right: [0, 1, 0],
up: [0, 0, 1],
};
extension.updateSelfPosition(
mockLocalUserNewPosition.position,
mockLocalUserNewPosition.forward,
mockLocalUserNewPosition.right,
mockLocalUserNewPosition.up,
);
public clearRemotePositions(): SpatialAudioErrorCode
Removes the spatial positions of all remote users.and local media players.
After a successful call of this method, the spatial audio effects of all remote users and local media players are removed, but the local user can still hear their original sounds.
After leaving the channel, to avoid wasting resources, you can also call this method to delete the spatial positions of all remote users.
Return values
See SpatialAudioErrorCode.
public updateSpatialBlur(enable: boolean): void
Updates the audio blurring setting.
After a successful call of this method, the audio blurring setting of the remote user or local media player is updated.
Parameters:
enable
: Whether to enable audio blurring:true
: Enable blurring.false
: (Default) Disable blurring.public updateSpatialAirAbsorb(enable: boolean): void
Updates the air absorption setting.
After a successful call of this method, the air absorption setting of the remote user or local media player is updated.
Parameters:
enable
: Whether to enable air absorption. This function simulates the energy attenuation of audio when the audio transmits in the air:true
: (Default) Enable air absorption.false
: Disable air absorption.public updatePlayerPositionInfo(info: RemoteVoicePositionInfo): SpatialAudioErrorCode;
Updates the spatial position of the specified local media player.
After a successful call of this method, the SDK calculates the spatial audio parameters based on the relative position changes of the local user and the media player.
Parameters:
info
: The spatial position of the media player. See RemoteVoicePositionInfo for details.Return values
See SpatialAudioErrorCode.
const mockLocalPlayerNewPosition = {
position: [1, 1, 1],
forward: [1, 0, 0],
};
processor.updatePlayerPositionInfo(mockLocalPlayerNewPosition);
public updateRemotePosition(info: RemoteVoicePositionInfo): SpatialAudioErrorCode;
Updates the spatial position of the specified remote user.
After a successful call of this method, the SDK calculates the spatial audio parameters based on the relative position changes of the local and remote users.
Parameters:
info
: The spatial position of the remote user. See RemoteVoicePositionInfo for details.Return values
See SpatialAudioErrorCode.
const processors = new Map();
client.on("user-published", async (user, mediaType) => {
await client.subscribe(user, mediaType);
console.log("subscribe success");
if (mediaType === "audio") {
const processor = extension.createProcessor();
processors.set(user.uid, processor);
const track = user.audioTrack;
track.pipe(processor).pipe(track.processorDestination);
track.play();
}
});
// The following code mocks the case of receiving the remote user's position change message from your server
yourOwnEventHandler.on("updatePosition", (uid, data) => {
const processor = processors.get(uid);
processor.updateRemotePosition({
position: data.position,
forward: data.forward,
});
});
public removeRemotePosition(): SpatialAudioErrorCode
Removes the spatial position of the specified remote user. or local media player.
After a successful call of this method, the spatial audio effects of the specified remote user or local media player are removed, but the local user can still hear their original sounds.
After leaving the channel, to avoid wasting resources, you can also call this method to delete the spatial position of the specified remote user or media player.
Return values
See SpatialAudioErrorCode.
yourOwnEventHandler.on("removePosition", uid => {
const processor = processors.get(uid);
processor.removeRemotePosition();
});
export interface RemoteVoicePositionInfo {
position: position;
forward: forward;
};
The spatial position of the remote user or the media player. Used in updatePlayerPositionInfo and updateRemotePosition.
Properties:
position
: The coordinates in the world coordinate system. This parameter is an array of length 3, and the three values represent the front, right, and top coordinates in turn.forward
: The unit vector of the x axis in the world coordinate system. This parameter is an array of length 3, and the three values represent the front, right, and top coordinates in turn.Error codes related to the spatial audio extension:
success
: The method call succeeds.EXTENSION_NOT_REG
: The extension is not registered.PROCESSOR_NOT_CREATE
: The Processor instance has not been created.PROCESSOR_NOT_ENABLED
: The Processor instance is not enabled.INVALID_PARMS
: Parameters are invalid.UNKNOWN_ERROR
: An unknown error occurs.