Spatial Audio

Last updated 2022/09/28 09:58:26

Agora provides spatial audio effects to give users an immersive audio experience in scenarios such as esports competitions and online conferences.

By updating the user's spatial position, you can feel the changes in the distance, position, and orientation of other users in real time.
By updating the spatial position of the media player, you can add a sense of space to media resources such as background sounds and musical accompaniment.
By setting the parameters, you can experience multiple audio effects such as audio blurring and air absorption.

The spatial audio function is currently in the experimental stage. Contact sales-us@agora.io to activate this function. If you need technical support, contact technical support.

Understand the tech

Spatial audio effects for users

Agora provides users local Cartesian coordinate system calculation solution:

This solution uses the ILocalSpatialAudioEngine class to implement spatial audio effects and calculates the relative positions of the local and remote users through the SDK. You need to call updateSelfPosition and updateRemotePosition to update the spatial coordinates of the local and remote users respectively, so that the local user can hear the remote user's spatial audio effects.

Spatial audio effects for media player

Agora provides media player with local Cartesian coordinate system calculation solution:

This solution calculates the relative positions of the local user and the media player through the SDK. You need to call updateSelfPosition and updatePlayerPositionInfo in the ILocalSpatialAudioEngine class to update the spatial coordinates of the local user and the media player, respectively, so that the local user can hear the spatial audio effect of the media player.

Prerequisites

Before proceeding, ensure that you have implemented basic real-time functions in your project. See Start a Video Call or Start Interactive Live Streaming.
When user uses a Bluetooth headset in a spatial audio effect scenario, it is necessary to ensure that the A2DP transmission protocol is used, that is, only the Bluetooth headset is used for audio playback, and other devices are used for audio capture. Otherwise, users cannot hear the spatial audio when they use the microphone.

Implementation

Spatial audio effects for users

This solution implements spatial audio effects through the ILocalSpatialAudioEngine class. The API call sequence and operation steps are as follows:

Before calling other Agora APIs, call create of the RtcEngine class, and fill in your App ID to initialize the RtcEngine object.
Before calling other APIs of the ILocalSpatialAudioEngine class, call create and initialize of the ILocalSpatialAudioEngine class to initialize the ILocalSpatialAudioEngine object.
Call setAudioProfile to set the profile to AUDIO_PROFILE_DEFAULT and the scenario to AUDIO_SCENARIO_GAME_STREAMING.
Call joinChannel with the options parameter to join the channel (using RTC Token). Set channelProfile to CHANNEL_PROFILE_LIVE_BROADCASTING and clientRoleType to CLIENT_ROLE_BROADCASTER in ChannelMediaOptions.
Agora subscribes to the audio streams of all remote users by default. You need to call muteAllRemoteAudioStreams(true) of RtcEngine to unsubscribe all remote users; otherwise, the audio reception range you set in step 6 is invalid.
Call the following method to set the audio reception range:
1. Call setMaxAudioRecvCount to set the maximum number of audio streams that a user can receive in a specified audio reception range.
2. Call setAudioRecvRange to set the audio reception range (meters) of the local user.
3. Call setDistanceUnit to set the length (meters) of the game engine unit distance.
You need to call updateSelfPosition and updateRemotePosition to update the spatial coordinates of the local and the remote users successively, so that the local user can hear the remote user's spatial audio effects.
If you do not need to experience spatial audio effects, call clearRemotePositions to delete the spatial position information of all remote users. After the deletion, the local user cannot hear all remote users.
Call destroy of ILocalSpatialAudioEngine to destroy the ILocalSpatialAudioEngine object.

The ILocalSpatialAudioEngine object must be destroyed before calling destroy of the RtcEngine.
Call leaveChannel and destroy of the RtcEngine class to leave the channel and destroy the RtcEngine object.

Spatial audio effects for media player

This solution implements spatial audio effects through the updateSelfPosition and updatePlayerPositionInfo in the ILocalSpatialAudioEngine class. Taking the ILocalSpatialAudioEngine class as an example, the operation steps are as follows:

Before calling other Agora APIs, call create of the RtcEngine class, and fill in your App ID to initialize the RtcEngine object.
Before calling other APIs of the ILocalSpatialAudioEngine class, call create and initialize of the ILocalSpatialAudioEngine class to initialize the ILocalSpatialAudioEngine object.
Call setAudioProfile to set the profile to AUDIO_PROFILE_DEFAULT and the scenario to AUDIO_SCENARIO_GAME_STREAMING.
Call joinChannel with the options parameter to join the channel (using RTC Token). Set channelProfile to CHANNEL_PROFILE_LIVE_BROADCASTING and clientRoleType to CLIENT_ROLE_BROADCASTER in ChannelMediaOptions.
Agora subscribes to the audio streams of all remote users by default. You need to call muteAllRemoteAudioStreams(true) of RtcEngine to unsubscribe all remote users; otherwise, the audio reception range you set in step 6 is invalid.
Call the following method to set the audio reception range:
1. Call setMaxAudioRecvCount to set the maximum number of audio streams that a user can receive in a specified audio reception range.
2. Call setAudioRecvRange to set the audio reception range (meters) of the local user.
3. Call setDistanceUnit to set the length (meters) of the game engine unit distance.
You need to call updateSelfPosition and updateRemotePosition to update the spatial coordinates of the local and the remote users successively, so that the local user can hear the remote user's spatial audio effects.
Call destroy of ILocalSpatialAudioEngine to destroy the ILocalSpatialAudioEngine object.

The ILocalSpatialAudioEngine object must be destroyed before calling destroy of the RtcEngine.
Call destroy of the RtcEngine to destroy the RtcEngine object.

Sample code

This section shows sample code that uses the local Cartesian coordinate system calculation solution to implement the spatial audio effects for the media player:

import static io.agora.api.example.common.model.Examples.ADVANCED;
import static io.agora.mediaplayer.Constants.MediaPlayerState.PLAYER_STATE_IDLE;
import static io.agora.mediaplayer.Constants.MediaPlayerState.PLAYER_STATE_OPEN_COMPLETED;
import static io.agora.mediaplayer.Constants.MediaPlayerState.PLAYER_STATE_PLAYBACK_COMPLETED;
import static io.agora.mediaplayer.Constants.MediaPlayerState.PLAYER_STATE_STOPPED;

import io.agora.api.example.common.Constant;
import io.agora.mediaplayer.Constants;
import io.agora.mediaplayer.IMediaPlayer;
import io.agora.mediaplayer.IMediaPlayerObserver;
import io.agora.mediaplayer.data.PlayerUpdatedInfo;
import io.agora.mediaplayer.data.SrcInfo;
import io.agora.rtc2.IRtcEngineEventHandler;
import io.agora.rtc2.RtcEngine;
import io.agora.rtc2.SpatialAudioParams;
import io.agora.spatialaudio.ILocalSpatialAudioEngine;
import io.agora.spatialaudio.LocalSpatialAudioConfig;
import io.agora.spatialaudio.RemoteVoicePositionInfo;

public class SpatialSound extends BaseFragment {
    private RtcEngine engine;
    private IMediaPlayer mediaPlayer;
    private ILocalSpatialAudioEngine localSpatial;
    private RemoteVoicePositionInfo positionInfo = new RemoteVoicePositionInfo();
    private int speakerUid;

        @Nullable
    @Override

        private void startRecord() {
        startTv.setVisibility(View.GONE);
        // Opens a media resource
        mediaPlayer.open(Constant.URL_PLAY_AUDIO_FILES, 0);

        LocalSpatialAudioConfig localSpatialAudioConfig = new LocalSpatialAudioConfig();
        localSpatialAudioConfig.mRtcEngine = engine;
        // Creates an ILocalSpatialAudioEngine
        localSpatial = ILocalSpatialAudioEngine.create();
        // Initializes the ILocalSpatialAudioEngine
        localSpatial.initialize(localSpatialAudioConfig);
        // Sets the audio reception range (meters)
        localSpatial.setAudioRecvRange(50);
        // Sets the length (in meters) of the game engine distance per unit.
        localSpatial.setDistanceUnit(1);
        // Updates the spatial position of the local user.
        float[] pos = new float[]{0.0F, 0.0F, 0.0F};
        float[] forward = new float[]{1.0F, 0.0F, 0.0F};
        float[] right = new float[]{0.0F, 1.0F, 0.0F};
        float[] up = new float[]{0.0F, 0.0F, 1.0F};
        localSpatial.updateSelfPosition(pos, forward, right, up);
    }

        private void updateSpatialSoundParam() {
        float transX = speakerIv.getTranslationX();
        float transY = speakerIv.getTranslationY();
        double viewDistance = Math.sqrt(Math.pow(transX, 2) + Math.pow(transY, 2));
        double viewMaxDistance = Math.sqrt(Math.pow((rootView.getWidth() - speakerIv.getWidth()) / 2.0f, 2) + Math.pow((rootView.getHeight() - speakerIv.getHeight()) / 2.0f, 2));
        double spkMaxDistance = 3;
        double spkMinDistance = 1;

        double spkDistance = spkMaxDistance * (viewDistance / viewMaxDistance);
        if (spkDistance < spkMinDistance) {
            spkDistance = spkMinDistance;
        }
        if (spkDistance > spkMaxDistance) {
            spkDistance = spkMaxDistance;
        }
        double degree = getDegree((int) transX, (int) transY);
        if (transX > 0) {
            degree = 360 - degree;
        }

        double posForward = spkDistance * Math.cos(degree);
        double posRight = spkDistance * Math.sin(degree);

        // Updates the spatial position of the media player.
        RemoteVoicePositionInfo positionInfo = new RemoteVoicePositionInfo();
        positionInfo.forward = new float[]{1.0F, 0.0F, 0.0F};
        positionInfo.position = new float[]{(float) posForward, (float) posRight, 0.0F};
        localSpatial.updatePlayerPositionInfo(mediaPlayer.getMediaPlayerId(), positionInfo);
    }

        private int getDegree(int point1X, int point1Y) {
        int vertexPointX = 0, vertexPointY = 0, point0X = 0;
        int point0Y = -10;
        int vector = (point0X - vertexPointX) * (point1X - vertexPointX) + (point0Y - vertexPointY) * (point1Y - vertexPointY);
        double sqrt = Math.sqrt(
                (Math.abs((point0X - vertexPointX) * (point0X - vertexPointX)) + Math.abs((point0Y - vertexPointY) * (point0Y - vertexPointY)))
                        * (Math.abs((point1X - vertexPointX) * (point1X - vertexPointX)) + Math.abs((point1Y - vertexPointY) * (point1Y - vertexPointY)))
        );
        double radian = Math.acos(vector / sqrt);
        return (int) (180 * radian / Math.PI);
    }

        @Override
    public void onDestroy() {
        super.onDestroy();
        // Stops playing the media resource
        mediaPlayer.stop();
        handler.removeCallbacksAndMessages(null);
        handler.post(RtcEngine::destroy);
        engine = null;
    }

        @Override
    public void onActivityCreated(@Nullable Bundle savedInstanceState) {
        super.onActivityCreated(savedInstanceState);
        Context context = getContext();

        if (channel != null) {
            return;
        }
        try {
            String appId = getString(R.string.agora_app_id);
            // Creates and initializes an RtcEngine
            engine = RtcEngine.create(getContext().getApplicationContext(), appId, iRtcEngineEventHandler);
            // Creates an IMediaPlayer
            mediaPlayer = engine.createMediaPlayer();
            // Registers a media player observer.
            mediaPlayer.registerPlayerObserver(iMediaPlayerObserver);
        } catch (Exception e) {
            e.printStackTrace();
            getActivity().onBackPressed();
        }
    }

        private IMediaPlayerObserver iMediaPlayerObserver = new IMediaPlayerObserver() {
        @Override
        // Reports the playback state change.
        public void onPlayerStateChanged(io.agora.mediaplayer.Constants.MediaPlayerState mediaPlayerState, io.agora.mediaplayer.Constants.MediaPlayerError mediaPlayerError) {
            Log.e(TAG, "onPlayerStateChanged mediaPlayerState " + mediaPlayerState);
            if (mediaPlayerState.equals(PLAYER_STATE_OPEN_COMPLETED)) {
                // Sets the loop playback of media resources
                mediaPlayer.setLoopCount(-1);
                // Stops playing the media resource
                mediaPlayer.play();
            }
        }
    };
}

Documentation

Sample project

Agora provides an open source sample project called SpatialSound on GitHub. You can download this sample project to try it out or refer to the source code.

API reference

ILocalSpatialAudioEngine

Is this page helpful?

Yes