Custom Audio Source and Renderer

Introduction

The Agora SDK uses default audio modules for capturing and rendering in real-time communications.
However, these modules might not meet your development requirements, such as in the following scenarios:

• Your app has its own audio module.
• You want to use a pre-processing library for the audio data.
• You need flexible device resource allocation to avoid conflicts with other services.

This article introduces how to use the methods provided by the Agora SDK to implement custom audio capture and rendering to meet your needs.

Sample project

Agora provides an open-sourced API-Example iOS sample project on GitHub that includes the following files that implement custom audio source and renderer functions:

Custom audio source

Implementation

Use the following steps to implement the custom audio source:

1. Before joining a channel, call enableExternalAudioSourceWithSampleRate to notify the SDK to use the external audio source.
2. Record and process the audio data on your own.
3. Send the audio data back to the SDK using pushExternalAudioFrame.

The API call sequence is as follows:

Sample code

1. Enable the external audio source

Call enableExternalAudioSourceWithSampleRate to notify the SDK to use customized audio source.

/**
 * Enables the external audio source.
 * @param sampleRate The sample rate of the audio data.
 * @param channel The number of channels of the audio data.
 */
agoraKit.enableExternalAudioSource(withSampleRate: sampleRate, channelsPerFrame: channel)

You need to define a method that specifies the sample rate and number of channels for capturing audio data. In the sample project, we define a setUpAudioSessionWithSampleRate method, and use the native methods of iOS to capture audio data.

/**
 * Enables external audio capture.
 * @param sampleRate The sample rate of the audio data.
 * @param channels The number of audio channels.
 * @param audioCRMode The audio capture mode.
 * @param ioType iOS The ioType for audio playback.
 */
[self.audioController setUpAudioSessionWithSampleRate:sampleRate channelCount:channels audioCRMode:audioCRMode IOType:ioType];

2. Join a channel and start capturing

Call joinChannelByToken of the SDK to join a channel and use the startWork method to notify the app to start capturing audio data. In the sample code, we define a startWork method to enable the external audio source.

// Call joinChannelByToken to join a channel.
let result = agoraKit.joinChannel(byToken: KeyCenter.Token, channelId: channelName, info: nil, uid: 0) {[unowned self] (channel, uid, elapsed) -> Void in
        self.isJoined = true
        LogUtils.log(message: "Join \(channel) with uid \(uid) elapsed \(elapsed)ms", level: .info)

        // Call startWork to start capturing audio data.
        self.exAudio.startWork()
        try? AVAudioSession.sharedInstance().setPreferredSampleRate(Double(sampleRate))

    }
    if result != 0 {
        // A common error is invalid parameter.
        self.showAlert(title: "Error", message: "joinChannel call failed: \(result), please check your params")
    }

3. Send the captured audio data to the SDK

After the system gets the audio data, call pushExternalAudioFrameRawData of the SDK to push the external audio data to the SDK.

You need a method to receive the captured audio data. In the sample code, we define an AudioController class, which contains a didCaptureData callback that occurs when receiving an audio frame.

// Occurs when the captured audio data is received. You can get the captured data in this callback.
- (void)audioController:(AudioController *)controller didCaptureData:(unsigned char *)data bytesLength:(int)bytesLength {

    if (self.audioCRMode != AudioCRModeExterCaptureSDKRender) {
        // Call pushExternalAudioFrameRawData to send the captured data to the SDK.
        [self.agoraKit pushExternalAudioFrameRawData:data samples:bytesLength / 2 timestamp:0];
    }
}

4. Stop capturing audio data

Disable the external audio source to stop capturing when the user leaves the channel.

// Stops audio data capture when the user leaves the channel
if isJoined {
    exAudio.stopWork()
    // Call leaveChannel to leave the channel.
    agoraKit.leaveChannel { (stats) -> Void in
            LogUtils.log(message: "left channel, duration: \(stats.duration)", level: .info)
        }
    }

API reference

• enableExternalAudioSourceWithSampleRate
• disableExternalAudioSource
• pushExternalAudioFrameRawData
• pushExternalAudioFrameSampleBuffer

Custom audio renderer

Implementation

Choose either of the following methods to implement a custom audio renderer:

Method one

1. Before joining a channel, call registerAudioFrameObserver to register the audio frame observer, and implement the IAudioFrameObserver class.
   After registering the observer, you can get the raw audio data of all remote users through the onPlaybackAudioFrame callback.
2. Call setParameters("{\"che.audio.external_render\": true}") to disable the audio rendering of the SDK.
3. Call setParameters("{\"che.audio.keep.audiosession\": true}") to disable the SDK control of the Audio Session.
4. Use your own audio renderer to render the audio data in the onPlaybackAudioFrame callback.

The API call sequence is as follows:

Method two

1. Call enableExternalAudioSink to enable the external audio sink before joining a channel.
2. After you join the channel, call pullPlaybackAudioFrameRawData or pullPlaybackAudioFrameSampleBufferByLengthInByte to get the remote audio data according to the format of the audio data.
3. Render and play the remote audio data on your own.

The API call sequence is as follows:

Sample code

1. Set up external audio

Define the setupExternalAudioWithAgoraKit method: First set the sample rate, number of audio channels, capture and render mode, and the playback channel, and then call registerAudioFrameObserver to register the audio frame observer.

// Define setupExternalAudioWithAgoraKit to set the external audio
- (void)setupExternalAudioWithAgoraKit:(AgoraRtcEngineKit *)agoraKit sampleRate:(uint)sampleRate channels:(uint)channels audioCRMode:(AudioCRMode)audioCRMode IOType:(IOUnitType)ioType {

  // Implement AudioController to trigger callbacks
  self.audioController = [AudioController audioController];
  self.audioController.delegate = self;

    // Register audio frame observer
    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)agoraKit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);  

    if (mediaEngine) {
        s_audioFrameObserver = new ExternalAudioFrameObserver();
        s_audioFrameObserver -> sampleRate = sampleRate;
        s_audioFrameObserver -> sampleRate_play = channels;
        mediaEngine->registerAudioFrameObserver(s_audioFrameObserver);
    }    
}

Call setupExternalAudioWithAudioKit and use the external renderer.

exAudio.setupExternalAudio(withAgoraKit: agoraKit, sampleRate: UInt32(sampleRate), channels: UInt32(channel), audioCRMode: .sdkCaptureExterRender, ioType: .remoteIO)

2. Disable the SDK control of the audio module

To avoid the SDK affecting the custom audio rendering, you need to disable the audio rendering of the SDK and the SDK control of the Audio Session.

// Disable the audio rendering of the SDK.
agoraKit.setParameters("{\"che.audio.external_render\": true}")
// Disable the SDK control of the Audio Session.
agoraKit.setParameters("{\"che.audio.keep.audiosession\": true}")

3. Get the audio data for rendering

After registering the audio frame observer, the SDK returns the raw audio data of remote users in the onPlaybackAudioFrame callback. Render the audio data in this callback.

virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
{
// Implement your own audio renderer
}

4. Join a channel and start rendering

Call joinChannelByToken to join a channel, and start rendering after joining the channel successfully.

// Join a channel.
let result = agoraKit.joinChannel(byToken: KeyCenter.Token, channelId: channelName, info: nil, uid: 0) {[unowned self] (channel, uid, elapsed) -> Void in
        self.isJoined = true
        LogUtils.log(message: "Join \(channel) with uid \(uid) elapsed \(elapsed)ms", level: .info)

        // Start rendering audio.
        self.exAudio.startWork()

    }
    if result != 0 {
        // Report errors if joining failed, usually due to invalid parameters.
        self.showAlert(title: "Error", message: "joinChannel call failed: \(result), please check your params")
    }

5. Stop rendering

Define stopWork and cancelRegister.

- (void)stopWork {
    [self.audioController stopWork];
    [self cancelRegister];
}


- (void)cancelRegister {
    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
    // Unregister the audio frame observer.
    mediaEngine->registerAudioFrameObserver(NULL);
}

Stop rendering the audio when the user leaves the channel.

if isJoined {
// Stop rendering the audio
    exAudio.stopWork()
    // Leave the channel
    agoraKit.leaveChannel { (stats) -> Void in
            LogUtils.log(message: "left channel, duration: \(stats.duration)", level: .info)
        }
    }

API reference

• enableExternalAudioSink
• disableExternalAudioSink
• pullPlaybackAudioFrameRawData
• pullPlaybackAudioFrameSampleBufferByLengthInByte
• getNativeHandle
• registerAudioFrameObserver
• onPlaybackAudioFrame

Considerations

Customizing the audio source and sink requires you to manage audio data recording and playback on your own.

• When customizing the audio source, you need to record and process the audio data on your own.
• When customizing the audio sink, you need to process and play back the audio data on your own.
• If you use the onPlaybackAudioFrame callback to get the raw audio data, ensure that you call setParameters to disable the audio rendering of the SDK.