The default Agora video module interacts seamlessly with the devices your app runs on. The SDK enables you to add specialized video features to your app using a custom video source.
By default, SDK integrates the default video modules on the device your app runs on for real-time communication. However, there are scenarios where you may want to integrate a custom video capturer. For example:
To manage the capture and processing of video frames when using a custom video source, use methods outside the SDK.
The Agora SDK provides the setExternalVideoSource
and pushExternalVideoFrame
methods to customize the video source. The API call sequence is as follows:
The following figure shows how the video data is transferred when you customize the video source in Push mode:
pushExternalVideoFrame
method.Before adjusting the audio volume, ensure that you have implemented the basic real-time communication functions in your project. For details, see Start a Call or Start Interactive Live Streaming.
To implement a custom video source in your project, refer to the following steps.
Before joining a channel, call setExternalVideoSource
to enable the custom video source. Once you enable it, you cannot use the methods of the SDK to capture video frames.
// Calls setExternalVideoSource to notify the SDK that the app uses the custom video source
agoraKit.setExternalVideoSource(true, useTexture: true, encodedFrame: true)
Implement the custom video source. Once the custom video source is enabled, you need to implement video capturing using APIs from outside the SDK. For example, the sample project defines the AgoraCameraSourcePush
class that captures video frames using the native methods of the system.
class AgoraCameraSourcePush: NSObject {
fileprivate var delegate: AgoraCameraSourcePushDelegate?
private var videoView: CustomVideoSourcePreview
private var currentCamera = Camera.defaultCamera()
private let captureSession: AVCaptureSession
private let captureQueue: DispatchQueue
private var currentOutput: AVCaptureVideoDataOutput? {
if let outputs = self.captureSession.outputs as? [AVCaptureVideoDataOutput] {
return outputs.first
} else {
return nil
}
}
// Initializes the custom video source
init(delegate: AgoraCameraSourcePushDelegate?, videoView: CustomVideoSourcePreview) {
self.delegate = delegate
self.videoView = videoView
captureSession = AVCaptureSession()
captureSession.usesApplicationAudioSession = false
let captureOutput = AVCaptureVideoDataOutput()
captureOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]
if captureSession.canAddOutput(captureOutput) {
captureSession.addOutput(captureOutput)
}
captureQueue = DispatchQueue(label: "MyCaptureQueue")
// Displays the captured video frames on the view
let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
videoView.insertCaptureVideoPreviewLayer(previewLayer: previewLayer)
}
deinit {
captureSession.stopRunning()
}
// Starts capturing video frames
func startCapture(ofCamera camera: Camera) {
guard let currentOutput = currentOutput else {
return
}
// Sets the camera as the capturing device
currentCamera = camera
currentOutput.setSampleBufferDelegate(self, queue: captureQueue)
captureQueue.async { [weak self] in
guard let strongSelf = self else {
return
}
strongSelf.changeCaptureDevice(toIndex: camera.rawValue, ofSession: strongSelf.captureSession)
strongSelf.captureSession.beginConfiguration()
if strongSelf.captureSession.canSetSessionPreset(AVCaptureSession.Preset.vga640x480) {
strongSelf.captureSession.sessionPreset = AVCaptureSession.Preset.vga640x480
}
strongSelf.captureSession.commitConfiguration()
strongSelf.captureSession.startRunning()
}
}
// Stops capturing video frames
func stopCapture() {
currentOutput?.setSampleBufferDelegate(nil, queue: nil)
captureQueue.async { [weak self] in
self?.captureSession.stopRunning()
}
}
// Switches the camera
func switchCamera() {
stopCapture()
currentCamera = currentCamera.next()
startCapture(ofCamera: currentCamera)
}
}
The AgoraCameraSourcePushDelegate
class is used to receive the captured video frames.
protocol AgoraCameraSourcePushDelegate {
func myVideoCapture(_ capture: AgoraCameraSourcePush, didOutputSampleBuffer pixelBuffer: CVPixelBuffer, rotation: Int, timeStamp: CMTime)
}
Implement the custom video renderer. The Agora SDK does not support rendering video frames captured in the push mode. Therefore, you need to implement a custom video renderer using methods from outside the SDK. In the sample project, we define a class called CustomVideoSourcePreview
using the native AVCaptureVideoPreviewLayer
class.
// Initializes localVideo
var localVideo = CustomVideoSourcePreview(frame: CGRect.zero)
// Defines the CustomVideoSourcePreview class
class CustomVideoSourcePreview : UIView {
private var previewLayer: AVCaptureVideoPreviewLayer?
func insertCaptureVideoPreviewLayer(previewLayer: AVCaptureVideoPreviewLayer) {
self.previewLayer?.removeFromSuperlayer()
previewLayer.frame = bounds
layer.insertSublayer(previewLayer, below: layer.sublayers?.first)
self.previewLayer = previewLayer
}
override func layoutSublayers(of layer: CALayer) {
super.layoutSublayers(of: layer)
previewLayer?.frame = bounds
}
}
Start capturing and rendering video frames. In the sample project, we create a customCamera
instance using the AgoraCameraSourcePush
class, and then we call startCapture
to start the capturing and rendering process.
// Initializes the AgoraCameraSourcePush class and sets the camera as the capturing device
customCamera = AgoraCameraSourcePush(delegate: self, videoView:localVideo)
// Calls startCapture of the AgoraCameraSourcePush class to start capturing video frames
customCamera?.startCapture(ofCamera: .defaultCamera())
Push the captured video frames to the SDK. Call the pushExternalVideoFrame
method to push the captured video frames to the SDK.
extension CustomVideoSourcePushMain:AgoraCameraSourcePushDelegate
{
func myVideoCapture(_ capture: AgoraCameraSourcePush, didOutputSampleBuffer pixelBuffer: CVPixelBuffer, rotation: Int, timeStamp: CMTime) {
let videoFrame = AgoraVideoFrame()
videoFrame.format = 12
videoFrame.textureBuf = pixelBuffer
videoFrame.time = timeStamp
videoFrame.rotation = Int32(rotation)
// Pushes the captured video frames to the SDK
agoraKit?.pushExternalVideoFrame(videoFrame)
}
}
This section includes in depth information about the methods you used in this page, and links to related pages.
Agora provides an open-source demo project on GitHub that implement the custom video source and renderer function.