Audio Frame Observer
Introduces methods and callbacks related to audio frame observers.
getEarMonitoringAudioParams
Sets the audio format for the onEarMonitoringAudioFrame callback.
virtual AudioParams getEarMonitoringAudioParams() = 0;
- Since
- v4.0.1
You need to register the callback when calling the registerAudioFrameObserver method. After you successfully register the audio observer, the SDK triggers this callback, and you can set the audio format in the return value of this callback.
The SDK triggers the onEarMonitoringAudioFrame callback with the AudioParams calculated sampling interval you set in the return value. The calculation formula is Sample interval (sec) = samplePerCall/(sampleRate × channel).
Ensure that the sample interval ≥ 0.01 (s).
Returns
The audio data of in-ear monitoring, see AudioParams.
getMixedAudioParams
Sets the audio format for the onMixedAudioFrame callback.
virtual AudioParams getMixedAudioParams() = 0;
You need to register the callback when calling the registerAudioFrameObserver method. After you successfully register the audio observer, the SDK triggers this callback, and you can set the audio format in the return value of this callback.
The SDK triggers the onMixedAudioFrame callback with the AudioParams calculated sampling interval you set in the return value. The calculation formula is Sample interval (sec) = samplePerCall/(sampleRate × channel).
Ensure that the sample interval ≥ 0.01 (s).
Returns
The mixed captured and playback audio data. See AudioParams.
getObservedAudioFramePosition
Sets the frame position for the video observer.
virtual int getObservedAudioFramePosition() = 0;
You can set one or more positions you need to observe by modifying the return value of getObservedAudioFramePosition based on your scenario requirements:
When the annotation observes multiple locations, the | (or operator) is required. To conserve system resources, you can reduce the number of frame positions that you want to observe.
Returns
- AUDIO_FRAME_POSITION_PLAYBACK(0x0001): This position can observe the playback audio mixed by all remote users, corresponding to the onPlaybackAudioFrame callback.
- AUDIO_FRAME_POSITION_RECORD(0x0002): This position can observe the collected local user's audio, corresponding to the onRecordAudioFrame callback.
- AUDIO_FRAME_POSITION_MIXED(0x0004): This position can observe the playback audio mixed by the loacl user and all remote users, corresponding to the onMixedAudioFrame callback.
- AUDIO_FRAME_POSITION_BEFORE_MIXING(0x0008): This position can observe the audio of a single remote user before mixing, corresponding to the onPlaybackAudioFrameBeforeMixing callback.
- AUDIO_FRAME_POSITION_EAR_MONITORING(0x0010): This position can observe the audio of a single remote user before mixing, corresponding to the onEarMonitoringAudioFrame callback.
getPlaybackAudioParams
Sets the audio format for the onPlaybackAudioFrame callback.
virtual AudioParams getPlaybackAudioParams() = 0;
You need to register the callback when calling the registerAudioFrameObserver method. After you successfully register the audio observer, the SDK triggers this callback, and you can set the audio format in the return value of this callback.
The SDK triggers the onPlaybackAudioFrame callback with the AudioParams calculated sampling interval you set in the return value. The calculation formula is Sample interval (sec) = samplePerCall/(sampleRate × channel).
Ensure that the sample interval ≥ 0.01 (s).
Returns
The audio data for playback, see AudioParams.
getRecordAudioParams
Sets the audio format for the onRecordAudioFrame callback.
virtual AudioParams getRecordAudioParams() = 0;
You need to register the callback when calling the registerAudioFrameObserver method. After you successfully register the audio observer, the SDK triggers this callback, and you can set the audio format in the return value of this callback.
The SDK triggers the onRecordAudioFrame callback with the AudioParams calculated sampling interval you set in the return value. The calculation formula is Sample interval (sec) = samplePerCall/(sampleRate × channel).
Ensure that the sample interval ≥ 0.01 (s).
Returns
The captured audio data, see AudioParams.
onEarMonitoringAudioFrame
Gets the in-ear monitoring audio frame.
virtual bool onEarMonitoringAudioFrame(AudioFrame& audioFrame) = 0;
- Method 1: After calling setEarMonitoringAudioFrameParameters to set the audio data format and registerAudioFrameObserver to register the audio frame observer object, the SDK calculates the sampling interval according to the parameters set in the methods, and triggers the onEarMonitoringAudioFrame callback according to the sampling interval.
- Method 2: After calling registerAudioFrameObserver to register the audio frame observer object, set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK then calculates the sampling interval according to the return value of the getEarMonitoringAudioParams callback, and triggers the onEarMonitoringAudioFrame callback according to the sampling interval.
- The priority of method 1 is higher than that of method 2. If method 1 is used to set the audio data format, the setting of method 2 is invalid.
Parameters
- audioFrame
- The raw audio data. See AudioFrame.
Returns
Reserved for future use.
onFrame
Occurs each time the player receives an audio frame.
virtual void onFrame(AudioPcmFrame* frame) = 0;
After registering the audio frame observer, the callback occurs every time the player receives an audio frame, reporting the detailed information of the audio frame.
Parameters
- frame
- The audio frame information. See AudioPcmFrame.
onLocalAudioSpectrum
Gets the statistics of a local audio spectrum.
virtual bool onLocalAudioSpectrum(const AudioSpectrumData& data) = 0;
After successfully calling registerAudioSpectrumObserver to implement the onLocalAudioSpectrum callback in IAudioSpectrumObserver and calling enableAudioSpectrumMonitor to enable audio spectrum monitoring, the SDK will trigger the callback as the time interval you set to report the received remote audio data spectrum.
Parameters
- data
- The audio spectrum data of the local user. See AudioSpectrumData.
Returns
true
: Spectrum data is received.false
: No spectrum data is received.
OnMixedAudioEncodedFrame
Gets the mixed and encoded audio data of the local and all remote users.
virtual void OnMixedAudioEncodedFrame(const uint8_t* frameBuffer, int length, const EncodedAudioFrameInfo& audioEncodedFrameInfo) = 0;
After calling registerAudioEncodedFrameObserver and setting the audio profile as AUDIO_ENCODED_FRAME_OBSERVER_POSITION_MIXED, you can get the mixed and encoded audio data of the local and all remote users through this callback.
Parameters
- frameBuffer
- The audio buffer.
- length
- The data length (byte).
- audioEncodedFrameInfo
- Audio information after encoding. See EncodedAudioFrameInfo.
onMixedAudioFrame
Retrieves the mixed captured and playback audio frame.
virtual bool onMixedAudioFrame(AudioFrame& audioFrame) = 0;
- Method 1: After calling setMixedAudioFrameParameters to set the audio data format and registerAudioFrameObserver to register the audio frame observer object, the SDK calculates the sampling interval according to the parameters set in the methods, and triggers the onMixedAudioFrame callback according to the sampling interval.
- Method 2: After calling registerAudioFrameObserver to register the audio frame observer object, set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK then calculates the sampling interval according to the return value of the getMixedAudioParams callback, and triggers the onMixedAudioFrame callback according to the sampling interval.
- The priority of method 1 is higher than that of method 2. If method 1 is used to set the audio data format, the setting of method 2 is invalid.
Parameters
- audioFrame
- The raw audio data. See AudioFrame.
Returns
Reserved for future use.
OnPlaybackAudioEncodedFrame
Gets the encoded audio data of all remote users.
virtual void OnPlaybackAudioEncodedFrame(const uint8_t* frameBuffer, int length, const EncodedAudioFrameInfo& audioEncodedFrameInfo) = 0;
After calling registerAudioEncodedFrameObserver and setting the encoded audio as AUDIO_ENCODED_FRAME_OBSERVER_POSITION_PLAYBACK, you can get encoded audio data of all remote users through this callback.
Parameters
- frameBuffer
- The audio buffer.
- length
- The data length (byte).
- audioEncodedFrameInfo
- Audio information after encoding. See EncodedAudioFrameInfo.
onPlaybackAudioFrame
Gets the raw audio frame for playback.
virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) = 0;
- Method 1: After calling setPlaybackAudioFrameParameters to set the audio data format and registerAudioFrameObserver to register the audio frame observer object, the SDK calculates the sampling interval according to the parameters set in the methods, and triggers the onPlaybackAudioFrame callback according to the sampling interval.
- Method 2: After calling registerAudioFrameObserver to register the audio frame observer object, set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK then calculates the sampling interval according to the return value of the getPlaybackAudioParams callback, and triggers the onPlaybackAudioFrame callback according to the sampling interval.
- The priority of method 1 is higher than that of method 2. If method 1 is used to set the audio data format, the setting of method 2 is invalid.
Parameters
- audioFrame
- The raw audio data. See AudioFrame.
Returns
Reserved for future use.
onPlaybackAudioFrameBeforeMixing
Retrieves the audio frame of a specified user before mixing.
virtual bool onPlaybackAudioFrameBeforeMixing(const char* channelId, rtc::uid_t uid, AudioFrame& audioFrame) = 0;
Parameters
- channelId
- The channel ID.
- uid
- The user ID of the specified user.
- audioFrame
- The raw audio data. See AudioFrame.
Returns
Reserved for future use.
OnRecordAudioEncodedFrame
Gets the encoded audio data of the local user.
virtual void OnRecordAudioEncodedFrame(const uint8_t* frameBuffer, int length, const EncodedAudioFrameInfo& audioEncodedFrameInfo) = 0;
After calling registerAudioEncodedFrameObserver and setting the encoded audio as AUDIO_ENCODED_FRAME_OBSERVER_POSITION_RECORD, you can get the encoded audio data of the local user from this callback.
Parameters
- frameBuffer
- The audio buffer.
- length
- The data length (byte).
- audioEncodedFrameInfo
- Audio information after encoding. See EncodedAudioFrameInfo.
onRecordAudioFrame
Gets the captured audio frame.
virtual bool onRecordAudioFrame(AudioFrame& audioFrame) = 0;
- Method 1: After calling setRecordingAudioFrameParameters to set the audio data format and registerAudioFrameObserver to register the audio frame observer object, the SDK calculates the sampling interval according to the parameters set in the methods, and triggers the onRecordAudioFrame callback according to the sampling interval.
- Method 2: After calling registerAudioFrameObserver to register the audio frame observer object, set the audio data format in the return value of the getObservedAudioFramePosition callback. The SDK then calculates the sampling interval according to the return value of the getRecordAudioParams callback, and triggers the onRecordAudioFrame callback according to the sampling interval.
- The priority of method 1 is higher than that of method 2. If method 1 is used to set the audio data format, the setting of method 2 is invalid.
Parameters
- audioFrame
- The raw audio data. See AudioFrame.
Returns
Reserved for future use.
onRemoteAudioSpectrum
Gets the remote audio spectrum.
virtual bool onRemoteAudioSpectrum(const UserAudioSpectrumInfo * spectrums, unsigned int spectrumNumber) = 0;
After successfully calling registerAudioSpectrumObserver to implement the onRemoteAudioSpectrum callback in the IAudioSpectrumObserver and calling enableAudioSpectrumMonitor to enable audio spectrum monitoring, the SDK will trigger the callback as the time interval you set to report the received remote audio data spectrum.
Parameters
- spectrums
- The audio spectrum information of the remote user, see UserAudioSpectrumInfo. The number of arrays is the number of remote users monitored by the SDK. If the array is null, it means that no audio spectrum of remote users is detected.
- spectrumNumber
- The number of remote users.
Returns
true
: Spectrum data is received.false
: No spectrum data is received.
registerAudioEncodedFrameObserver
Registers an encoded audio observer.
virtual int registerAudioEncodedFrameObserver(const AudioEncodedFrameObserverConfig& config, IAudioEncodedFrameObserver *observer) = 0;
- Call this method after joining a channel.
- You can call this method or the startAudioRecording method to set the audio content and audio quality. Agora does not recommend using this method and startAudioRecording at the same time; otherwise, only the method called later takes effect.
Parameters
- config
- Observer settings for the encoded audio. See AudioEncodedFrameObserverConfig.
- observer
- The encoded audio observer. See IAudioEncodedFrameObserver.
Returns
- 0: Success.
- < 0: Failure.
registerAudioFrameObserver
Registers an audio frame observer object.
virtual int registerAudioFrameObserver(IAudioFrameObserver* observer) = 0;
Call this method to register an audio frame observer object (register a callback). When you need the SDK to trigger the onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame or onEarMonitoringAudioFrame callback, you need to use this method to register the callbacks.
Parameters
- observer
-
The observer object instance. See IAudioFrameObserver. Set the value as NULL to release the instance. Agora recommends calling after receiving onLeaveChannel to release the audio observer object.
Returns
- 0: Success.
- < 0: Failure.
registerAudioSpectrumObserver
Registers an audio spectrum observer.
virtual int registerAudioSpectrumObserver(agora::media::IAudioSpectrumObserver * observer) = 0;
After successfully registering the audio spectrum observer and calling enableAudioSpectrumMonitor to enable the audio spectrum monitoring, the SDK reports the callback that you implement in the IAudioSpectrumObserver class at the time interval you set.
Parameters
- observer
-
The audio spectrum observer. See IAudioSpectrumObserver.
Returns
- 0: Success.
- < 0: Failure.
setMixedAudioFrameParameters
Sets the audio data format reported by onMixedAudioFrame.
virtual int setMixedAudioFrameParameters(int sampleRate, int channel, int samplesPerCall) = 0;
Parameters
- sampleRate
-
The sample rate (Hz) of the audio data, which can be set as
8000
,16000
,32000
,44100
, or48000
.
- channel
-
The number of channels of the audio data, which can be set as
1
(Mono) or2
(Stereo).
- samplesPerCall
-
Sets the number of samples. In Media Push scenarios, set it as
1024
.
The SDK triggers the onMixedAudioFrame callback according to the sample interval. Sample interval (sec) = samplePerCall
/(sampleRate
× channel
) Ensure that the value of sample interval is more than or equal to 0.01.
Returns
- 0: Success.
- < 0: Failure.
setPlaybackAudioFrameBeforeMixingParameters
Sets the audio data format reported by onPlaybackAudioFrameBeforeMixing.
virtual int setPlaybackAudioFrameBeforeMixingParameters(int sampleRate, int channel) = 0;
Parameters
- sampleRate
-
The sample rate (Hz) of the audio data, which can be set as
8000
,16000
,32000
,44100
, or48000
.
- channel
-
The number of channels of the external audio source, which can be set as
1
(Mono) or2
(Stereo).
Returns
- 0: Success.
- < 0: Failure.
setPlaybackAudioFrameParameters
Sets the audio data format for playback.
virtual int setPlaybackAudioFrameParameters(int sampleRate,
int channel,
RAW_AUDIO_FRAME_OP_MODE_TYPE mode,
int samplesPerCall) = 0;
Sets the data format for the onPlaybackAudioFrame callback.
- Ensure that you call this method before joining a channel.
- The SDK calculates the sampling interval based on the samplesPerCall, sampleRate and channel parameters set in this method.Sample interval = samplePerCall/(sampleRate × channel). Ensure that the sample interval ≥ 0.01 (s). The SDK triggers the onPlaybackAudioFrame callback according to the sampling interval.
Parameters
- sampleRate
- The sample rate returned in the onPlaybackAudioFrame callback, which can be set as 8000, 16000, 32000, 44100, or 48000 Hz.
- channel
-
The number of channels returned in the onPlaybackAudioFrame callback:
- 1: Mono.
- 2: Stereo.
- mode
-
The use mode of the audio frame. See RAW_AUDIO_FRAME_OP_MODE_TYPE.
- samplesPerCall
- The number of data samples returned in the onPlaybackAudioFrame callback, such as 1024 for the Media Push.
Returns
- 0: Success.
- < 0: Failure.
setRecordingAudioFrameParameters
Sets the format of the captured raw audio data.
virtual int setRecordingAudioFrameParameters(int sampleRate,
int channel,
RAW_AUDIO_FRAME_OP_MODE_TYPE mode,
int samplesPerCall) = 0;
Sets the audio format for the onRecordAudioFrame callback.
- Ensure that you call this method before joining a channel.
- The SDK calculates the sampling interval based on the samplesPerCall, sampleRate and channel parameters set in this method.Sample interval = samplePerCall/(sampleRate × channel). Ensure that the sample interval ≥ 0.01 (s). The SDK triggers the onRecordAudioFrame callback according to the sampling interval.
Parameters
- sampleRate
- The sample rate returned in the onRecordAudioFrame callback, which can be set as 8000, 16000, 32000, 44100, or 48000 Hz.
- channel
-
The number of channels returned in the onRecordAudioFrame callback:
- 1: Mono.
- 2: Stereo.
- mode
-
The use mode of the audio frame. See RAW_AUDIO_FRAME_OP_MODE_TYPE.
- samplesPerCall
- The number of data samples returned in the onRecordAudioFrame callback, such as 1024 for the Media Push.
Returns
- 0: Success.
- < 0: Failure.
unregisterAudioSpectrumObserver
Unregisters the audio spectrum observer.
virtual int unregisterAudioSpectrumObserver(agora::media::IAudioSpectrumObserver * observer) = 0;
After calling registerAudioSpectrumObserver, if you want to disable audio spectrum monitoring, you can call this method.
Parameters
- observer
- The audio spectrum observer. See IAudioSpectrumObserver.
Returns
- 0: Success.
- < 0: Failure.