Raw Audio Data

Introduction

During real-time communications, you can pre- and post-process the audio and video data and modify them for desired playback effects.

Agora provides the raw data function for you to process the audio data according to your scenarios. This function enables you to pre-process the captured audio signal before sending it to the encoder, or to post-process the decoded audio signal.

Sample project

Agora provides the following open-source sample projects on GitHub that implement the raw audio data function:

• Process raw audio data using iOS/macOS APIs
  • iOS: RawAudioData
  • macOS: RawAudioData
• Process raw audio data using C++ APIs
  • iOS: RawMediaData
  • macOS: RawMediaData

You can view the source code on Github or download the project to try it out.

Implementation

Before proceeding, ensure that you have implemented the basic real-time communication functions in your project.

Data transfer

The following diagram shows the data transfer while pre- or post-processing raw audio data.

With onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, or onMixedAudioFrame , you can:

• Get raw audio frames.
• Process the raw frames and return to the SDK.

Process raw audio data using iOS/macOS APIs

As of v3.4.5, you can call iOS/macOS APIs to capture raw audio data and process the data.

1. Call setAudioDataFrame to set the audio frame delegate. You can call this method either before or after joining the channel, but it only captures the audio data after the audio frame delegate is successfully registered.
2. After you successfully register the audio frame delegate, the SDK triggers the getRecordAudioParams, getPlaybackAudioParams, or getMixedAudioParams callbacks. You can set the desired audio data format in the return values of these callbacks.
3. The SDK triggers the getObservedAudioFramePosition and isMultipleChannelFrameWanted callbacks when capturing each audio frame. In the return values of these callbacks, you can set the audio observation positions and whether to capture the audio data from multiple channels.
4. According to the return values of getObservedAudioFramePosition and isMultipleChannelFrameWanted, the SDK triggers onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing/onPlaybackAudioFrameBeforeMixingEx, or onMixedAudioFrame callbacks to send you the captured raw audio data.
5. Process the captured audio data according to your scenarios. You can send the processed audio data with the onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing/onPlaybackAudioFrameBeforeMixingEx, or onMixedAudioFrame callbacks according to your scenarios.

API call sequence

The following diagram shows how to implement the raw audio data function in your project:

Sample code

// swift

class RawAudioDataMain: BaseViewController {
  var localVideo = Bundle.loadVideoView(type: .local, audioOnly: true)
  var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: true)

  @IBOutlet weak var container: AGEVideoContainer!
  // Define the agoraKit variable
  var agoraKit: AgoraRtcEngineKit!

    ...
    // Initialize agoraKit, and register related callbacks
  agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

  // Call setAudioDataFrame to set the audio frame delegate. You need to implement the AgoraAudioDataFrameProtocol in this method.
  agoraKit.setAudioDataFrame(self)

    ...

  // Implement the extension of the AgoraAudioDataFrameProtocol in the current class
  extension RawAudioDataMain: AgoraAudioDataFrameProtocol{

  // Implement the getObservedAudioFramePosition callback, and set the audio observation position in this callback
  func getObservedAudioFramePosition() -> AgoraAudioFramePosition {
      return .record
  }

  // Implement the onRecordAudioFrame callback
  func onRecordAudioFrame(_ frame: AgoraAudioFrame) -> Bool {
        return true
    }

  // Implement the getRecordAudioParams callback, and set the audio recording format in the return value of this callback.
  func getRecordAudioParams() -> AgoraAudioParam {
      let param = AgoraAudioParam()
      param.channel = 1
      param.mode = .readOnly
      param.sampleRate = 44100
      param.samplesPerCall = 1024
      return param
  }
}

API reference

• setAudioDataFrame
• onRecordAudioFrame
• onPlaybackAudioFrame
• onMixedAudioFrame
• onPlaybackAudioFrameBeforeMixing
• onPlaybackAudioFrameBeforeMixingEx
• getObservedAudioFramePosition
• getRecordAudioParams
• getMixedAudioParams
• getPlaybackAudioParams

Process raw audio data using C++ APIs

Follow these steps to implement the raw data functions using C++ APIs:

1. Call the registerAudioFrameObserver method to register a audio observer object before joining the channel. You need to implement an IAudioFrameObserver class in this method.
2. After you successfully register the observer object, the SDK triggers the onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, or onMixedAudioFrame callback to send the raw audio data at the set time interval.
3. After acquiring the raw data, you can process the captured raw data based on your scenario and send the processed data to the SDK via the callbacks mentioned in step 2.

API call sequence

The following diagram shows how to implement the raw data functions in your project:

Sample code

In addition to the API call sequence diagram, you can refer to the following code samples.

1. Initialize AgoraRtcEngineKit

Call sharedEngineWithConfig to initialize AgoraRtcEngineKit.

// Swift
// Initialize AgoraRtcEngineKit
let config = AgoraRtcEngineConfig()
agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

2. Register an audio frame observer

// Swift
let audioType:ObserverAudioType = ObserverAudioType(rawValue: ObserverAudioType.recordAudio.rawValue | ObserverAudioType.playbackAudioFrameBeforeMixing.rawValue | ObserverAudioType.mixedAudio.rawValue | ObserverAudioType.playbackAudio.rawValue);
agoraMediaDataPlugin?.registerAudioRawDataObserver(audioType)
agoraMediaDataPlugin?.audioDelegate = self

The Agora SDK provides only C++ methods and callbacks for implementing the raw audio data function. Therefore, you need to call C++ methods on iOS or macOS to register an audio frame observer.

// Import the C++ header file
#import <AgoraRtcKit/IAgoraMediaEngine.h>
#import <AgoraRtcKit/IAgoraRtcEngine.h>

- (void)registerAudioRawDataObserver:(ObserverAudioType)observerType {
    // Gets the C++ event handler of the RTC Native SDK
    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
    // Creates an IMediaEngine instance
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    // Ensure that you call queryInterface in IMediaEngine to set the agora::AGORA_IID_MEDIA_ENGINE interface, or you cannot implement registerAudioFrameObserver
    mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);

    NSInteger oldValue = self.observerAudioType;
    self.observerAudioType |= observerType;

    if (mediaEngine && oldValue == 0)
    {
        // Register an audio frame observer
        mediaEngine->registerAudioFrameObserver(&s_audioFrameObserver);
        s_audioFrameObserver.mediaDataPlugin = self;
    }
}

3. Set the audio frame capture parameters

If you want to modify the audio sampling rate, use mode, number of channels, or sampling interval of the captured audio data, you can call the following methods:

// Swift
// Sets the audio data format returned in onRecordAudioFrame
agoraKit.setRecordingAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)
// Sets the audio data format returned in onMixedAudioFrame
agoraKit.setMixedAudioFrameParametersWithSampleRate(44100, samplesPerCall: 4410)
// Sets the audio data format returned in onPlaybackAudioFrame
agoraKit.setPlaybackAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)

4. Join a channel

Call joinChannelByToken to join an RTC channel.

// Swift
agoraKit.joinChannel(byToken: KeyCenter.Token, channelId: channelName, info: nil, uid: 0) {[unowned self] (channel, uid, elapsed)}

5. Receive the captured audio data

After joining a channel, you can receive the captured audio data from the callbacks in the IAudioFrameObserver class. You can also use these callbacks to send the processed audio data back to the SDK.

// Swift
// Gets the raw audio data of the local user, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, didRecord audioRawData: AgoraAudioRawDate) -> AgoraAudioRawData {
  return audioRawData
}

// Gets the raw audio data of all remote users, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, willPlaybackAudioRawData audioRawData: AgoraRawData) -> AgoraAudioRawData {
  return audioRawData
}

// Gets the raw audio data of a specified remote user, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, willPlaybackBeforeMixing audioRawData: AgoraAudioRawData, ofUid uid: uint) -> AgoraAudioRawData {
  return audioRawData
}

// Gets the raw audio data of the local user and all remote users, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, didMixedAudioRawData audioRawData: AgoraAudioRawData) -> AgoraAudioRawData {
  return audioRawData
}

Call the C++ methods in the .mm file to implement the callbacks.

// Swift
class AgoraMediaDataPluginAudioFrameObserver : public agora::media::IAudioFrameObserver
{
public:
    AgoraMediaDataPlugin *mediaDataPlugin;

    // Defines the format of the raw audio data
    AgoraAudioRawData* getAudioRawDataWithAudioFrame(AudioFrame& audioFrame)
    {
        AgoraAudioRawData *data = [[AgoraAudioRawData alloc] init];
        data.samples = audioFrame.samples;
        data.bytesPerSample = audioFrame.bytesPerSample;
        data.channels = audioFrame.channels;
        data.samplesPerSec = audioFrame.samplesPerSec;
        data.renderTimeMs = audioFrame.renderTimeMs;
        data.buffer = (char *)audioFrame.buffer;
        data.bufferSize = audioFrame.samples * audioFrame.bytesPerSample;
        return data;
    }

    // Defines the format of the processed audio data
    void modifiedAudioFrameWithNewAudioRawData(AudioFrame& audioFrame, AgoraAudioRawData *audioRawData)
    {
        audioFrame.samples = audioRawData.samples;
        audioFrame.bytesPerSample = audioRawData.bytesPerSample;
        audioFrame.channels = audioRawData.channels;
        audioFrame.samplesPerSec = audioRawData.samplesPerSec;
        audioFrame.renderTimeMs = audioRawData.renderTimeMs;
    }

    // Receives the raw audio data of the local user from onRecordAudioFrame
    virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override
    {

        if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 0) == 0)) return true;
        @autoreleasepool {
            if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:didRecordAudioRawData:)]) {
                AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
                AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin didRecordAudioRawData:data];
                modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
            }
        }

        // Sets the return value as true, meaning to send the data back to the SDK
        return true;
    }

    // Receives the raw audio data of all remote users from onPlaybackAudioFrame
    virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
    {

        if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 1) == 0)) return true;
        @autoreleasepool {
            if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:willPlaybackAudioRawData:)]) {
                AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
                AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin willPlaybackAudioRawData:data];
                modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
            }
        }
        // Sets the return value as true, meaning to send the data back to the SDK
        return true;
    }

    // Receives the raw audio data of all remote users from onPlaybackAudioFrameBeforeMixing  
    virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override
    {

        if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 2) == 0)) return true;
        @autoreleasepool {
            if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:willPlaybackBeforeMixingAudioRawData:ofUid:)]) {
                AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
                AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin willPlaybackBeforeMixingAudioRawData:data ofUid:uid];
                modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
            }
        }
        // Sets the return value as true, meaning to send the data back to the SDK
        return true;
    }

    // Receives the raw audio data of the local user and all remote users from onMixedAudioFrame
    virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
    {

        if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 3) == 0)) return true;
        @autoreleasepool {
            if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:didMixedAudioRawData:)]) {
                AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
                AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin didMixedAudioRawData:data];
                modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
            }
        }
        // Sets the return value as true, meaning to send the data back to the SDK
        return true;
    }
};

6. Stop registering the audio frame observer

Call registerAudioFrameObserver(NULL) to stop registering the audio frame observer.

- (void)deregisterAudioRawDataObserver:(ObserverAudioType)observerType {
    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);

    self.observerAudioType ^= observerType;

    if (mediaEngine && self.observerAudioType == 0)
    {
        mediaEngine->registerAudioFrameObserver(NULL);
        s_audioFrameObserver.mediaDataPlugin = nil;
    }
}

API Reference

• registerAudioFrameObserver
• setRecordingAudioFrameParameters
• setPlaybackAudioFrameParameters
• setMixedAudioFrameParameters
• onRecordAudioFrame
• onPlaybackAudioFrame
• onPlaybackAudioFrameBeforeMixing
• onMixedAudioFrame

Considerations

• The raw audio data function is implemented by the C++ APIs of the SDK, and you need to mix the Objective-C and C++ methods in an .mm file. See AgoraMediaDataPlugin.mm for reference.

• Ensure that you call getNativeHandler to get the C++ handle before calling the C++ method.

• The raw audio data function is resource intensive and may affect performance.