IMediaEngine
The IMediaEngine class.
pullAudioFrame
Pulls the remote audio data.
virtual int pullAudioFrame(IAudioFrameObserver::AudioFrame* frame) = 0;
Before calling this method, you need to call setExternalAudioSink to notify the app to enable and set the external rendering.
After a successful method call, the app pulls the decoded and mixed audio data for playback.
- This method only supports pulling data from custom audio source. If you need to pull the data captured by the SDK, do not call this method.
- Call this method after joining a channel.
- Once you enable the external audio sink, the app will not retrieve any audio data from the onPlaybackAudioFrame callback.
- The difference between this method and the onPlaybackAudioFrame callback is as follows:
- The SDK sends the audio data to the app through the onPlaybackAudioFrame callback. Any delay in processing the audio frames may result in audio jitter.
- After a successful method call, the app automatically pulls the audio data from the SDK. After setting the audio data parameters, the SDK adjusts the frame buffer and avoids problems caused by jitter in the external audio playback.
Parameters
- frame
- Pointers to AudioFrame.
Returns
- 0: Success.
- < 0: Failure.
pushAudioFrame
Pushes the external audio frame.
virtual int pushAudioFrame(MEDIA_SOURCE_TYPE type, IAudioFrameObserver::AudioFrame* frame, bool wrap = false, int sourceId = 0) = 0;
Parameters
- type
- The type of the audio recording device. See MEDIA_SOURCE_TYPE.
- frame
- The external audio frame. See AudioFrame.
- wrap
- Whether to use the placeholder. Agora recommends using the default value.
true
: Use the placeholder.false
: (Default) Do not use the placeholder.
- sourceId
- The ID of external audio source. If you want to publish a custom external audio source, set this parameter to the ID of the corresponding custom audio track you want to publish.
Returns
- 0: Success.
- < 0: Failure.
pushVideoFrame
Pushes the external raw video frame to the SDK.
virtual int pushVideoFrame(base::ExternalVideoFrame* frame, unsigned int videoTrackId = 0) = 0;
To push the unencoded external raw video frame to the SDK, call createCustomVideoTrack to get the video track ID, set customVideoTrackId as the video track ID you want to publish in the ChannelMediaOptions of each channel, and set publishCustomVideoTrack as true
.
Parameters
- frame
-
The external raw video frame to be pushed. See ExternalVideoFrame.
- videoTrackId
- The video track ID returned by calling the createCustomVideoTrack method. The default value is 0.
Returns
- 0: Success.
- < 0: Failure.
registerAudioFrameObserver
Registers an audio frame observer object.
virtual int registerAudioFrameObserver(IAudioFrameObserver* observer) = 0;
Call this method to register an audio frame observer object (register a callback). When you need the SDK to trigger onMixedAudioFrame, onRecordAudioFrame, onPlaybackAudioFrame or onEarMonitoringAudioFrame callback, you need to use this method to register the callbacks.
Parameters
- observer
-
The observer object instance. See IAudioFrameObserver. Set the value as NULL to release the instance. Agora recommends calling after receiving onLeaveChannel to release the audio observer object.
Returns
- 0: Success.
- < 0: Failure.
registerVideoEncodedFrameObserver
Registers a receiver object for the encoded video image.
virtual int registerVideoEncodedFrameObserver(IVideoEncodedFrameObserver* observer) = 0;
If you only want to observe encoded video frames (such as h.264 format) without decoding and rendering the video, Agora recommends that you implement one IVideoEncodedFrameObserver class through this method.
- Call registerVideoFrameObserver to register the raw video frame observer before joining the channel.
- Call registerVideoEncodedFrameObserver to register the encoded video frame observer before joining the channel.
- After joining the channel, get the user IDs of group B users through onUserJoined, and then call to set the encodedFrameOnly of this group of users to
true
. - Call muteAllRemoteVideoStreams to start receiving the video streams of all remote users. Then:
- The raw video data of group A users can be obtained through the callback in IVideoFrameObserver, and the SDK renders the data by default.
- The encoded video data of group B users can be obtained through the callback in IVideoEncodedFrameObserver.
- Call this method before joining a channel.
Parameters
- observer
- The video frame observer object. See IVideoEncodedFrameObserver.
Returns
- 0: Success.
- < 0: Failure.
registerVideoFrameObserver
Registers a video frame observer object.
virtual int registerVideoFrameObserver(IVideoFrameObserver* observer) = 0;
If you want to observe raw video frames (such as YUV or RGBA format), Agora recommends that you implement one IVideoFrameObserver class with this method.
After registering the class, you need to register the callbacks in the class as required. After you successfully register the video frame observer, the SDK triggers the registered callbacks each time a video frame is received.
- Call registerVideoFrameObserver to register the raw video frame observer before joining the channel.
- Call registerVideoEncodedFrameObserver to register the encoded video frame observer before joining the channel.
- After joining the channel, get the user IDs of group B users through onUserJoined, and then call to set the encodedFrameOnly of this group of users to
true
. - Call muteAllRemoteVideoStreams to start receiving the video streams of all remote users. Then:
- The raw video data of group A users can be obtained through the callback in IVideoFrameObserver, and the SDK renders the data by default.
- The encoded video data of group B users can be obtained through the callback in IVideoEncodedFrameObserver.
- Ensure that you call this method before joining a channel.
- When handling the video data returned in the callbacks, pay attention to the changes in the width and height parameters, which may be adapted under the following circumstances:
- When network conditions deteriorate, the video resolution decreases incrementally.
- If the user adjusts the video profile, the resolution of the video returned in the callbacks also changes.
Parameters
- observer
- The observer object instance. See IVideoFrameObserver. To release the instance, set the value as NULL.
Returns
- 0: Success.
- < 0: Failure.
setExternalAudioSink
Sets the external audio sink.
virtual int setExternalAudioSink(bool enabled, int sampleRate, int channels) = 0;
This method applies to scenarios where you want to use external audio data for playback. After you set the external audio sink, you can call pullAudioFrame to pull remote audio frames. The app can process the remote audio and play it with the audio effects that you want.
Parameters
- enabled
-
Whether to enable or disable the external audio sink:
true
: Enables the external audio sink.false
: (Default) Disables the external audio sink.
- sampleRate
-
The sample rate (Hz) of the external audio sink, which can be set as 16000, 32000, 44100, or 48000.
- channels
- The number of audio channels of the external audio sink:
- 1: Mono.
- 2: Stereo.
Returns
- 0: Success.
- < 0: Failure.
setExternalAudioSource
Sets the external captured audio parameters and chooses whether to publish the audio to the remote user.
virtual int setExternalAudioSource(bool enabled,
int sampleRate,
int channels,
int sourceNumber,
bool localPlayback = false,
bool publish = true) = 0;
Parameters
- enabled
-
Whether to enable the external audio source:
true
: Enable the external audio source.false
: (Default) Disable the external audio source.
- sampleRate
- The sample rate (Hz) of the external audio source, which can be set as
8000
,16000
,32000
,44100
, or48000
. - channels
- The number of channels of the external audio source, which can be set as
1
(Mono) or2
(Stereo). - sourceNumber
- The number of external audio sources. The value of this parameter should be larger than 0. The SDK creates a corresponding number of custom audio tracks based on this parameter value and names the audio tracks starting from 0. In ChannelMediaOptions, you can set publishCustomAudioSourceId to the audio track ID you want to publish.
- localPlayback
-
Whether to play the external audio source:
true
: Play the external audio source.false
: (Default) Do not play the external source.
- publish
-
Whether to publish audio to the remote users:
true
: (Default) Publish audio to the remote users.false
: Do not publish audio to the remote users.
Returns
- 0: Success.
- < 0: Failure.
setExternalVideoSource
Configures the external video source.
virtual int setExternalVideoSource(
bool enabled, bool useTexture, EXTERNAL_VIDEO_SOURCE_TYPE sourceType = VIDEO_FRAME,
rtc::SenderOptions encodedVideoOption = rtc::SenderOptions()) = 0;
Parameters
- enabled
- Whether to use the external video source:
true
: Use the external video source. The SDK prepares to accept the external video frame.false
: (Default) Do not use the external video source.
- useTexture
- Whether to use the external video frame in the Texture format.
true
: Use the external video frame in the Texture format.false
: (Default) Do not use the external video frame in the Texture format.
- sourceType
- Whether to encode the external video frame, see EXTERNAL_VIDEO_SOURCE_TYPE.
- encodedVideoOption
- Video encoding options. This parameter needs to be set if sourceType is ENCODED_VIDEO_FRAME. To set this parameter, contact contact technical support.
Returns
- 0: Success.
- < 0: Failure.