This page introduces how to use the Cloud Gateway to send media streams to the client and receive media streams from the client.
Before you begin, make sure you have downloaded the latest Cloud Gateway. See Integrate the C++ SDK.
Complete the following steps before sending and receiving media streams.
You need to call createAgoraService
and initialize
to create and initialize an AgoraService
object. The IAgoraService
object persists as long as the server app keeps running.
The SDK supports user IDs in both int format and string format. This page only shows user IDs in int format (the character set can only be digits). To learn more about user IDs in string format, see Using User IDs in String Format.
// Creates an IAgoraService object
auto service = createAgoraService();
// Initializes the IAgoraService object
agora::base::AgoraServiceConfiguration scfg;
// Sets Agora App ID
scfg.appId = appid;
// Enables the audio processing module
scfg.enableAudioProcessor = enableAudioProcessor;
// Disables the audio device module (Normally we do not directly connect audio capture or playback devices to a server)
scfg.enableAudioDevice = enableAudioDevice;
// Whether to enable video
scfg.enableVideo = enableVideo;
// Disables user IDs in string format (the character can be digits, letters, or special symbols) so that user ID can only contain digits
scfg.useStringUid = enableuseStringUid;
if (service->initialize(scfg) != agora::ERR_OK) {
return nullptr;
}
After initializing the SDK, you can refer to the following steps to connect to the Agora RTC channel.
Call createRtcConnection
to create an IRtcConnection
object to connect to the Agora RTC channel:
// Creates an IRtcConnection object
agora::rtc::RtcConnectionConfiguration ccfg;
ccfg.autoSubscribeAudio = false;
ccfg.autoSubscribeVideo = false;
ccfg.clientRoleType = agora::rtc::CLIENT_ROLE_BROADCASTER;
agora::agora_refptr<agora::rtc::IRtcConnection> connection = service->createRtcConnection(ccfg);
Call registerObserver
to listen to connection events. The SampleConnectionObserver
class in the sample code inherits the IRtcConnectionObserver
class and the INetworkObserver
class:
// Calls registerObserver to listen to connection events
auto connObserver = std::make_shared<SampleConnectionObserver>();
connection->registerObserver(connObserver.get());
Call connect
to connect to an Agora RTC channel:
// Calls connect to connect to an Agora RTC channel
if (connection->connect(options.appId.c_str(), options.channelId.c_str(),
options.userId.c_str())) {
AG_LOG(ERROR, "Failed to connect to Agora channel!");
return -1;
}
Refer to the following steps to send media streams to the client:
You can use the IMediaNodeFactory
object to create various types of media stream senders:
IAudioPcmDataSender
:Sends audio data in PCM format.IVideoFrameSender
:Sends video data in YUV format.IAudioEncodedFrameSender
:Sends encoded audio data.IVideoEncodedImageSender
:Sends encoded video data.Create an IMediaNodeFactory
object:
// Creates an IMediaNodeFactory object.
agora::agora_refptr<agora::rtc::IMediaNodeFactory> factory = service->createMediaNodeFactory();
if (!factory) {
AG_LOG(ERROR, "Failed to create media node factory!");
}
Per your own requirements, create an IAudioPcmDataSender
object, an IVideoFrameSender
object, an IAudioEncodedFrameSender
object, or an IVideoEncodedImageSender
object for sending audio in PCM format, video in YUV format, encoded audio, or encoded video:
// Creates a sender for PCM audio
agora::agora_refptr<agora::rtc::IAudioPcmDataSender> audioPcmDataSender =
factory->createAudioPcmDataSender();
if (!audioPcmDataSender) {
AG_LOG(ERROR, "Failed to create audio data sender!");
return -1;
}
// Creates a sender for YUV video
agora::agora_refptr<agora::rtc::IVideoFrameSender> videoFrameSender =
factory->createVideoFrameSender();
if (!videoFrameSender) {
AG_LOG(ERROR, "Failed to create video frame sender!");
return -1;
}
// Creates a sender for encoded audio
agora::agora_refptr<agora::rtc::IAudioEncodedFrameSender> audioFrameSender =
factory->createAudioEncodedFrameSender();
if (!audioFrameSender) {
AG_LOG(ERROR, "Failed to create audio encoded frame sender!");
return -1;
}
// Creates a sender for encoded video
agora::agora_refptr<agora::rtc::IVideoEncodedImageSender> videoEncodedFrameSender =
factory->createVideoEncodedImageSender();
if (!videoEncodedFrameSender) {
AG_LOG(ERROR, "Failed to create encoded video frame sender!");
return -1;
}
Create an ILocalAudioTrack
object and an ILocalVideoTrack
object, which correspond to the local audio track and local video track to be published to the Agora RTC channel:
// Creates a custom audio track that uses a PCM audio stream sender
agora::agora_refptr<agora::rtc::ILocalAudioTrack> customAudioTrack =
service->createCustomAudioTrack(audioPcmDataSender);
if (!customAudioTrack) {
AG_LOG(ERROR, "Failed to create audio track!");
return -1;
}
// Creates a custom audio track that uses an encoded audio stream sender
agora::agora_refptr<agora::rtc::ILocalAudioTrack> customAudioTrack =
service->createCustomAudioTrack(audioFrameSender, agora::base::MIX_DISABLED);
if (!customAudioTrack) {
AG_LOG(ERROR, "Failed to create audio track!");
return -1;
}
// Creates a custom video track that uses a YUV video stream sender
agora::agora_refptr<agora::rtc::ILocalVideoTrack> customVideoTrack =
service->createCustomVideoTrack(videoFrameSender);
if (!customVideoTrack) {
AG_LOG(ERROR, "Failed to create video track!");
return -1;
}
// Creates a custom video track that uses an encoded video stream sender
agora::agora_refptr<agora::rtc::ILocalVideoTrack> customVideoTrack =
service->createCustomVideoTrack(videoEncodedFrameSender);
if (!customVideoTrack) {
AG_LOG(ERROR, "Failed to create video track!");
return -1;
}
Use publish
methods in the ILocalUser
object to publish local audio and video tracks created in the previous step to the Agora RTC channel:
// Enables and publishes audio track
customAudioTrack->setEnabled(true);
connection->getLocalUser()->publishAudio(customAudioTrack);
// Enables and publishes video track
customVideoTrack->setEnabled(true);
connection->getLocalUser()->publishVideo(customVideoTrack);
Start the sending thread, which calls the send
methods of the audio/video sender:
// Audio sending thread
std::thread sendAudioThread(SampleSendAudioTask, options, audioFrameSender, std::ref(exitFlag));
// Video sending thread
std::thread sendVideoThread(SampleSendVideoH264Task, options, videoFrameSender,
std::ref(exitFlag));
sendAudioThread.join();
sendVideoThread.join();
Taking PCM data as an example, the following code shows the implementation of SampleSendAudioTask
:
static void SampleSendAudioTask(
const SampleOptions& options,
agora::agora_refptr<agora::rtc::IAudioPcmDataSender> audioFrameSender, bool& exitFlag) {
// Currently, the SDK only supports sending PCM data of 10 ms in length each time, so the sending interval of PCM data is 10 ms.
PacerInfo pacer = {0, 10, std::chrono::steady_clock::now()};
while (!exitFlag) {
sendOnePcmFrame(options, audioFrameSender);
waitBeforeNextSend(pacer); // sleep for a while before sending the next frame
}
}
The following code shows the implementation of sendOnePcmFrame
. You can see that sendOnePcmFrame
calls the sendAudioPcmData
method of audioFrameSender
to send PCM data:
static void sendOnePcmFrame(const SampleOptions& options,
agora::agora_refptr<agora::rtc::IAudioPcmDataSender> audioFrameSender) {
static FILE* file = nullptr;
const char* fileName = options.audioFile.c_str();
// Calculates the sample size based on the length of the sample
int sampleSize = sizeof(int16_t) * options.audio.numOfChannels;
int samplesPer10ms = options.audio.sampleRate / 100;
int sendBytes = sampleSize * samplesPer10ms;
if (!file) {
if (!(file = fopen(fileName, "rb"))) {
AG_LOG(ERROR, "Failed to open audio file %s", fileName);
return;
}
AG_LOG(INFO, "Open audio file %s successfully", fileName);
}
uint8_t frameBuf[sendBytes];
if (fread(frameBuf, 1, sizeof(frameBuf), file) != sizeof(frameBuf)) {
if (feof(file)) {
fclose(file);
file = nullptr;
AG_LOG(INFO, "End of audio file");
} else {
AG_LOG(ERROR, "Error reading audio data: %s", std::strerror(errno));
}
return;
}
// Calls the sendAudioPcmData method of audioFrameSender to send PCM data.
// Configures PCM data in sendAudioPcmData
if (audioFrameSender->sendAudioPcmData(frameBuf, 0, samplesPer10ms, agora::rtc::TWO_BYTES_PER_SAMPLE,
options.audio.numOfChannels,
options.audio.sampleRate) < 0) {
AG_LOG(ERROR, "Failed to send audio frame!");
}
}
Taking H.264 video data as an example, the following code shows the implementation of SampleSendVideoH264Task
:
static void SampleSendVideoH264Task(
const SampleOptions& options,
agora::agora_refptr<agora::rtc::IVideoEncodedImageSender> videoH264FrameSender,
bool& exitFlag) {
std::unique_ptr<HelperH264FileParser> h264FileParser(
new HelperH264FileParser(options.videoFile.c_str()));
h264FileParser->initialize();
// Calculate send interval based on frame rate. H264 frames are sent at this interval
PacerInfo pacer = {0, 1000 / options.video.frameRate, 0, std::chrono::steady_clock::now()};
while (!exitFlag) {
if (auto h264Frame = h264FileParser->getH264Frame()) {
sendOneH264Frame(options.video.frameRate, std::move(h264Frame), videoH264FrameSender);
waitBeforeNextSend(pacer); // sleep for a while before sending the next frame
}
};
}
The following code shows the implementation of sendOneH264Frame
. You can see that sendOneH264Frame
calls the sendEncodedVideoImage
method of videoH264FrameSender
to send H.264 data.
You can set the format of the encoded video data by using the
videoEncodedFrameInfo
parameter of thesendEncodedVideoImage
method.
static void sendOneH264Frame(
int frameRate, std::unique_ptr<HelperH264Frame> h264Frame,
agora::agora_refptr<agora::rtc::IVideoEncodedImageSender> videoH264FrameSender) {
agora::rtc::EncodedVideoFrameInfo videoEncodedFrameInfo;
videoEncodedFrameInfo.rotation = agora::rtc::VIDEO_ORIENTATION_0;
videoEncodedFrameInfo.codecType = agora::rtc::VIDEO_CODEC_H264;
videoEncodedFrameInfo.framesPerSecond = frameRate;
videoEncodedFrameInfo.frameType =
(h264Frame.get()->isKeyFrame ? agora::rtc::VIDEO_FRAME_TYPE::VIDEO_FRAME_TYPE_KEY_FRAME
: agora::rtc::VIDEO_FRAME_TYPE::VIDEO_FRAME_TYPE_DELTA_FRAME);
videoH264FrameSender->sendEncodedVideoImage(
reinterpret_cast<uint8_t*>(h264Frame.get()->buffer.get()), h264Frame.get()->bufferLen,
videoEncodedFrameInfo);
}
After finishing media sending tasks, you can refer to the following steps to disconnect from the channel and release resources.
Call unpublish
methods to stop publishing audio and video:
connection->getLocalUser()->unpublishAudio(customAudioTrack);
connection->getLocalUser()->unpublishVideo(customVideoTrack);
Call unregisterObserver
to unregister the connection observer:
connection->unregisterObserver(connObserver.get());
Call disconnect
to disconnect from the Agora RTC channel:
if (connection->disconnect()) {
AG_LOG(ERROR, "Failed to disconnect from Agora channel!");
return -1;
}
AG_LOG(INFO, "Disconnected from Agora channel successfully");
Release resources from created objects:
connObserver.reset();
audioPcmDataSender = nullptr;
videoFrameSender = nullptr;
customAudioTrack = nullptr;
customVideoTrack = nullptr;
factory = nullptr;
connection = nullptr;
Refer to the following steps to receive media streams from the client.
ILocalUserObserver
object to register video and audio framesInstantiate the video and audio frame observer object, and register the observer by using the member methods of the ILocalUserObserver
class. After registration is successful, the SDK triggers callbacks when audio or video frames are available. You can get audio and video frames from the callbacks.
The SampleLocalUserObserver
class in the sample code not only includes observer objects from the IVideoEncodedFrameObserver
class, the IAudioFrameObserverBase
class, and the IVideoFrameObserver2
class, but also inherits the ILocalUserObserver
class. The member methods of the ILocalUserObserver
class can be used to register video and audio frame observer objects.
// The SampleLocalUserObserver class inherits the ILocalUserObserver class
auto localUserObserver =
std::make_shared<SampleLocalUserObserver>(connection->getLocalUser());
// The PcmFrameObserver class inherits the IAudioFrameObserver class
auto pcmFrameObserver = std::make_shared<PcmFrameObserver>(options.audioFile);
if (connection->getLocalUser()->setPlaybackAudioFrameBeforeMixingParameters(
options.audio.numOfChannels, options.audio.sampleRate)) {
AG_LOG(ERROR, "Failed to set audio frame parameters!");
return -1;
}
// Instantiates the IAudioFrameObserver class
localUserObserver->setAudioFrameObserver(pcmFrameObserver.get());
// The H264FrameReceiver class inherits the IVideoEncodedImageReceiver class
auto h264FrameReceiver = std::make_shared<H264FrameReceiver>(options.videoFile);
// Instantiates the IVideoEncodedImageReceiver class
localUserObserver->setVideoEncodedImageReceiver(h264FrameReceiver.get());
setAudioFrameObserver
and setVideoEncodedImageReceiver
are member methods of the SampleLocalUserObserver
class and can be used to instantiate the IAudioFrameObserver
class and the IVideoEncodedImageReceiver
class.
void setAudioFrameObserver(agora::media::IAudioFrameObserver* observer) {
audio_frame_observer_ = observer;
}
void setVideoEncodedImageReceiver(agora::rtc::IVideoEncodedImageReceiver* receiver) {
video_encoded_receiver_ = receiver;
}
ILocalUserObserver
class to receive media streamsThe following code shows how to receive encoded video, video in YUV format, and audio in PCM format.
// Receives encoded video by using the OnEncodedVideoImageReceived callback, and saves the video data in a file
class H264FrameReceiver : public agora::rtc::IVideoEncodedImageReceiver {
public:
H264FrameReceiver(const std::string& outputFilePath)
: outputFilePath_(outputFilePath),
h264File_(nullptr),
fileCount(0),
fileSize_(0) {}
bool OnEncodedVideoImageReceived(
const uint8_t* imageBuffer, size_t length,
const agora::rtc::EncodedVideoFrameInfo& videoEncodedFrameInfo) override;
private:
std::string outputFilePath_;
FILE* h264File_;
int fileCount;
int fileSize_;
};
// Receives YUV video by using the onFrame callback, and saves the video data in a file
class YuvFrameObserver : public agora::rtc::IVideoFrameObserver2 {
public:
YuvFrameObserver(const std::string& outputFilePath)
: outputFilePath_(outputFilePath),
yuvFile_(nullptr),
fileCount(0),
fileSize_(0) {}
void onFrame(const char* channelId, agora::user_id_t remoteUid, const agora::media::base::VideoFrame* frame) override;
virtual ~YuvFrameObserver() = default;
private:
std::string outputFilePath_;
FILE* yuvFile_;
int fileCount;
int fileSize_;
};
// Receives PCM audio by using the onPlaybackAudioFrameBeforeMixing callback and save the audio data in a file
bool PcmFrameObserver::onPlaybackAudioFrameBeforeMixing(const char* channelId, agora::media::base::user_id_t userId, AudioFrame& audioFrame) {
// Creates files to save received audio data
if (!pcmFile_) {
std::string fileName = (++fileCount > 1)
? (outputFilePath_ + to_string(fileCount))
: outputFilePath_;
if (!(pcmFile_ = fopen(fileName.c_str(), "w"))) {
AG_LOG(ERROR, "Failed to create received audio file %s",
fileName.c_str());
return false;
}
AG_LOG(INFO, "Created file %s to save received PCM samples",
fileName.c_str());
}
// Writes PCM data to a file
size_t writeBytes =
audioFrame.samplesPerChannel * audioFrame.channels * sizeof(int16_t);
if (fwrite(audioFrame.buffer, 1, writeBytes, pcmFile_) != writeBytes) {
AG_LOG(ERROR, "Error writing decoded audio data: %s", std::strerror(errno));
return false;
}
fileSize_ += writeBytes;
if (fileSize_ >= DEFAULT_FILE_LIMIT) {
fclose(pcmFile_);
pcmFile_ = nullptr;
fileSize_ = 0;
}
return true;
}
When the media receiving task is complete, refer to the following steps to disconnect from the Agora RTC channel.
Call unregister methods to release the video and audio observer.
// Release video and audio observer
local_user_->unregisterAudioFrameObserver(audio_frame_observer_);
local_user_->unregisterVideoFrameObserver(video_frame_observer_);
Call disconnect
to disconnect from the Agora RTC channel.
// Disconnect from the Agora channel
if (connection->disconnect()) {
AG_LOG(ERROR, "Failed to disconnect from Agora channel!");
return -1;
}
AG_LOG(INFO, "Disconnected from Agora channel successfully");
Release the created objects.
// Releases the created objects.
localUserObserver.reset();
pcmFrameObserver.reset();
h264FrameReceiver.reset();
connection = nullptr;
When your server app stops running, refer to the following sample code to release the IAgoraService
object.
service->release();
service = nullptr;
Here you can find a link to the API reference, sequence diagrams, and developer considerations.
Refer to Cloud Gateway C++ API Reference to find detailed API and parameter descriptions.
The following diagrams show the API calling sequence for a variety of media stream scenarios.
Send YUV video and PCM audio data
Send encoded video and audio data
LIVE_BROADCASTING
.