Spatial Audio

Agora provides spatial audio effects to give users an immersive audio experience in scenarios such as esports competitions and online conferences.

• By updating the user's spatial position, you can feel the changes in the distance, position, and orientation of other users in real time.
• By updating the spatial position of the media player, you can add a sense of space to media resources such as background sounds and musical accompaniment.
• By setting the parameters, you can experience multiple audio effects such as audio blurring and air absorption.

Understand the tech

Spatial audio effects for users

Agora provides users with local Cartesian coordinate system calculation solution:

This solution uses the AgoraLocalSpatialAudioKit class to implement spatial audio effects, and calculates the relative position of the local user and the remote user through the SDK. You need to call updateSelfPosition and updateRemotePosition to update the spatial coordinates of the local and remote users respectively, so that the local user can hear the remote user's spatial audio effects.

Spatial audio effects for media player

Agora provides media player with local Cartesian coordinate system calculation solution:

This solution calculates the relative positions of the local user and the media player through the SDK. You need to call updateSelfPosition and updatePlayerPositionInfo in the AgoraLocalSpatialAudioKit classes respectively to update the spatial coordinates of the local user and the media player, so that the local user can hear the spatial audio effect of the media player.

Prerequisites

• Before proceeding, ensure that you have implemented basic real-time functions in your project. See Start a Video Call or Start Interactive Live Streaming for details.
• When user uses a Bluetooth headset in a spatial audio effect scenario, it is necessary to ensure that the A2DP transmission protocol is used, that is, only the Bluetooth headset is used for audio playback, and other devices are used for audio capture. Otherwise, users cannot hear the spatial audio when they use the microphone.

Implementation

Spatial audio effects for users

This solution implements spatial audio effect through the AgoraLocalSpatialAudioKit class. The API call sequence and operation steps are as follows:

1. Before calling other Agora APIs, call sharedEngineWithAppId and fill in your App ID to initialize the AgoraRtcEngineKit object.

2. Before calling other APIs of the AgoraLocalSpatialAudioKit class, call sharedLocalSpatialAudioWithConfig to initialize the AgoraLocalSpatialAudioKit object.

3. Call setAudioProfile to set the profile parameter to AgoraAudioProfileDefault and the scenario to AgoraAudioScenarioGameStreaming.

4. Call joinChannelByToken with the mediaOptions parameter to join channel (using RTC Token). Set channelProfile to AgoraChannelProfileLiveBroadcasting and clientRoleType to AgoraClientRoleBroadcaster in AgoraRtcChannelMediaOptions.

5. Agora subscribes to the audio streams of all remote users by default. You need to call muteAllRemoteAudioStreams(true) of AgoraRtcEngineKit to unsubscribe all remote users, otherwise the audio reception range you set in step 6 will be invalid.

6. Call the following method to set the audio reception range:

   1. Call setMaxAudioRecvCount to set the maximum number of audio streams that a user can receive in a specified audio reception range.
   2. Call setAudioRecvRange to set the audio reception range (meters) of the local user.
   3. Call setDistanceUnit to set the length (meters) of the game engine unit distance.

7. You need to call updateSelfPosition and updateRemotePosition to update the spatial coordinates of the local and the remote users successively, so that the local user can hear the remote user's spatial audio effects.

8. If you do not need to experience spatial audio effects, call clearRemotePositions to delete the spatial position information of all remote users. After the deletion, the local user cannot hear all remote users.

9. Call destroy of AgoraLocalSpatialAudioKit to destroy the AgoraLocalSpatialAudioKit object.

   The AgoraLocalSpatialAudioKit object must be destroyed before calling destroy of the AgoraRtcEngineKit.

10. Call leaveChannel and destroy of the AgoraRtcEngineKit class to leave the channel and destroy the AgoraRtcEngineKit object.

Spatial audio effects for media player

This solution implements spatial audio effect through the updateSelfPosition and updatePlayerPositionInfo in the AgoraLocalSpatialAudioKit class. Taking the AgoraLocalSpatialAudioKit class as an example, the operation steps are as follows:

1. Before calling other Agora APIs, call sharedEngineWithAppId and fill in your App ID to initialize the AgoraRtcEngineKit object.

2. Before calling other APIs of the AgoraLocalSpatialAudioKit class, call sharedLocalSpatialAudioWithConfig to initialize the AgoraLocalSpatialAudioKit object.

3. Call setAudioProfile to set the profile to AgoraAudioProfileDefault and the scenario to AgoraAudioScenarioGameStreaming.

4. Call joinChannelByToken with the mediaOptions parameter to join channel (using RTC Token). Set channelProfile to AgoraChannelProfileLiveBroadcasting and clientRoleType to AgoraClientRoleBroadcaster in AgoraRtcChannelMediaOptions.

5. Agora subscribes to the audio streams of all remote users by default. You need to call muteAllRemoteAudioStreams(true) of AgoraRtcEngineKit to unsubscribe all remote users, otherwise the audio reception range you set in step 6 will be invalid.

6. Call the following method to set the audio reception range:

   1. Call setMaxAudioRecvCount to set the maximum number of audio streams that a user can receive in a specified audio reception range.
   2. Call setAudioRecvRange to set the audio reception range (meters) of the local user.
   3. Call setDistanceUnit to set the length (meters) of the game engine unit distance.

7. You need to call updateSelfPosition and updateRemotePosition to update the spatial coordinates of the local and the remote users successively, so that the local user can hear the remote user's spatial audio effects.

8. Call destroy of AgoraLocalSpatialAudioKit to destroy the AgoraLocalSpatialAudioKit object.

   The AgoraLocalSpatialAudioKit object must be destroyed before calling destroy of the AgoraRtcEngineKit.

9. Call destroy of the AgoraRtcEngineKit to destroy the AgoraRtcEngineKit object.

Sample code

This section shows sample code that uses the local Cartesian coordinate system calculation solution to implement the spatial audio effects for the media player:

// Swift
class SpatialAudioMain: BaseViewController {
    var agoraKit: AgoraRtcEngineKit!
    var mediaPlayer: AgoraRtcMediaPlayerProtocol!
    var localSpatial: AgoraLocalSpatialAudioKit!

        override func viewDidLoad() {
        // Initialize the AgoraRtcEngineKit
        agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: KeyCenter.AppId, delegate: self)
        // Set the channel profile as liveBroadcasting
        config.channelProfile = .liveBroadcasting
        // Set the client role as the host.
        agoraKit.setClientRole(.broadcaster)
        // Set audio profile and scenario
        agoraKit.setAudioProfile(.default, scenario: .gameStreaming)
        guard let filePath = Bundle.main.path(forResource: "audiomixing", ofType: "mp3") else {return}
        // Create a media player
        mediaPlayer = agoraKit.createMediaPlayer(with: self)
        // Opens a media resource
        mediaPlayer.open(filePath, startPos: 0)

        // Initialize the AgoraLocalSpatialAudioKit
        let localSpatialConfig = AgoraLocalSpatialAudioConfig()
        localSpatialConfig.rtcEngine = agoraKit
        localSpatial = AgoraLocalSpatialAudioKit.sharedLocalSpatialAudio(with: localSpatialConfig)
        // Set the audio reception range (meters)
        localSpatial.setAudioRecvRange(50)
        // Set the length (meters) of the game engine distance per unit.
        localSpatial.setDistanceUnit(1)
        // Updates the spatial position of the local user.
        let pos = [NSNumber(0.0), NSNumber(0), NSNumber(0.0)]
        let forward = [NSNumber(1.0), NSNumber(0.0), NSNumber(0.0)]
        let right = [NSNumber(0.0), NSNumber(1.0), NSNumber(0.0)]
        let up = [NSNumber(0.0), NSNumber(0.0), NSNumber(1.0)]
        self.localSpatial.updateSelfPosition(pos, axisForward: forward, axisRight: right, axisUp: up)
    }

        override func willMove(toParent parent: UIViewController?) {
        if parent == nil {
            // Destroy the AgoraLocalSpatialAudioKit
            AgoraLocalSpatialAudioKit.destroy()
            // Destroy the AgoraRtcEngineKit object.
            AgoraRtcEngineKit.destroy()
        }
    }

    // Updates the spatial position of the media player.
    func updateRemoteUserSpatialAudioPositon() {
        let maxR = UIScreen.main.bounds.height / 2.0
        let maxSpatailDistance = 30.0
        let spatialDistance = currentDistance * maxSpatailDistance / maxR

                let posForward = spatialDistance * cos(currentAngle);
        let posRight = spatialDistance * sin(currentAngle);
        let position = [NSNumber(value: posForward), NSNumber(value: posRight), NSNumber(0.0)]
        let forward = [NSNumber(1.0), NSNumber(0.0), NSNumber(0.0)]

                let positionInfo = AgoraRemoteVoicePositionInfo()
        positionInfo.position = position
        positionInfo.forward = forward
        localSpatial.updatePlayerPositionInfo(mediaPlayer.getMediaPlayerId(), positionInfo);
    }
}

Documentation

Sample project

Agora provides an open source sample project SpatialAudio on GitHub. You can download the sample project to try it out or refer to the source code.

API reference

• AgoraLocalSpatialAudioKit