Custom Video Source and Renderer

Introduction

The Agora SDK uses default audio and video modules for capturing and rendering in real-time communications.

However, these default modules might not meet your development requirements, such as in the following scenarios:

• Your app has its own video module.
• You want to use a non-camera source, such as recorded screen data.
• You need to process the captured video with a pre-processing library for functions such as image enhancement.
• You need flexible device resource allocation to avoid conflicts with other services.

This article describes how to use the Agora Native SDK to customize the video source and renderer.

Sample project

Agora provides open-source demo projects on GitHub that implement the custom video source and renderer function. You can download the projects to try them out or view the source code:

• iOS:
  • Customize the video source (Push mode): CustomVideoSourcePush
  • Customize the video source (mediaIO mode): CustomVideoSourceMediaIO
  • Customize the video renderer (mediaIO mode): CustomVideoRender
• macOS
  • Customize the video source (Push mode): CustomVideoSourcePush
  • Customize the video source (mediaIO mode): CustomVideoSourceMediaIO
  • Customize the video renderer (mediaIO mode): CustomVideoRender

Implementation

The following diagram shows how the video data is transferred when you customize the video source or video renderer:

Video frames captured by the SDK or a custom video source, or received from a remote user, can be rendered by either the SDK or a custom video renderer.

• You must implement the custom video source yourself using methods from outside the SDK.
• The Agora Native SDK provides two methods to send the captured video frames to the SDK:
  • The push mode: In this mode, you can call the pushExternalVideoFrame method to push the captured video frames to the SDK.
  • The mediaIO mode: In this mode, the SDK takes control of the capturing process with an AgoraVideoSourceProtocol class, and then it stores the captured video frames in an AgoraVideoFrameConsumer class. To send the captured frames to the SDK, you can call consumePixelBuffer or consumeRawData in the AgoraVideoFrameConsumer class.
• You must also implement the custom video renderer yourself using methods from outside the SDK. The SDK provides an AgoraVideoSinkProtocol class to control the rendering process, and the setLocalVideoRenderer and setRemoteVideoRenderer methods to display the video of the local and remote view.

Customize video source (push mode)

The Agora SDK provides the setExternalVideoSource and pushExternalVideoFrame methods to customize the video source. The API call sequence is as follows:

1. Enable the custom video source

Before joining a channel, call setExternalVideoSource to enable the custom video source. Once you enable it, you cannot use the methods of the SDK to capture video frames.

// Swift
// Calls setExternalVideoSource to notify the SDK that the app uses the custom video source
agoraKit.setExternalVideoSource(true, useTexture: true, pushMode: true)

2. Implement the custom video source

Once the custom video source is enabled, you need to implement video capturing using APIs from outside the SDK. In the sample project, we define a class called AgoraCameraSourcePush class that captures video frames using the native methods of the system.

// Swift
class AgoraCameraSourcePush: NSObject {

    fileprivate var delegate: AgoraCameraSourcePushDelegate?
    private var videoView: CustomVideoSourcePreview

    private var currentCamera = Camera.defaultCamera()
    private let captureSession: AVCaptureSession
    private let captureQueue: DispatchQueue
    private var currentOutput: AVCaptureVideoDataOutput? {
        if let outputs = self.captureSession.outputs as? [AVCaptureVideoDataOutput] {
            return outputs.first
        } else {
            return nil
        }
    }

    // Initializes the custom video source
    init(delegate: AgoraCameraSourcePushDelegate?, videoView: CustomVideoSourcePreview) {
        self.delegate = delegate
        self.videoView = videoView

        captureSession = AVCaptureSession()
        captureSession.usesApplicationAudioSession = false

        let captureOutput = AVCaptureVideoDataOutput()
        captureOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]
        if captureSession.canAddOutput(captureOutput) {
            captureSession.addOutput(captureOutput)
        }

        captureQueue = DispatchQueue(label: "MyCaptureQueue")

        // Displays the captured video frames on the view
        let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
        videoView.insertCaptureVideoPreviewLayer(previewLayer: previewLayer)
    }

    deinit {
        captureSession.stopRunning()
    }

    // Starts capturing video frames
    func startCapture(ofCamera camera: Camera) {
        guard let currentOutput = currentOutput else {
            return
        }

        // Sets the camera as the capturing device
        currentCamera = camera
        currentOutput.setSampleBufferDelegate(self, queue: captureQueue)

        captureQueue.async { [weak self] in
            guard let strongSelf = self else {
                return
            }
            strongSelf.changeCaptureDevice(toIndex: camera.rawValue, ofSession: strongSelf.captureSession)
            strongSelf.captureSession.beginConfiguration()
            if strongSelf.captureSession.canSetSessionPreset(AVCaptureSession.Preset.vga640x480) {
                strongSelf.captureSession.sessionPreset = AVCaptureSession.Preset.vga640x480
            }
            strongSelf.captureSession.commitConfiguration()
            strongSelf.captureSession.startRunning()
        }
    }

    // Stops capturing video frames
    func stopCapture() {
        currentOutput?.setSampleBufferDelegate(nil, queue: nil)
        captureQueue.async { [weak self] in
            self?.captureSession.stopRunning()
        }
    }

    // Switches the camera
    func switchCamera() {
        stopCapture()
        currentCamera = currentCamera.next()
        startCapture(ofCamera: currentCamera)
    }
}

We also define a class called AgoraCameraSourcePushDelegate to receive the captured video frames.

// Swift
protocol AgoraCameraSourcePushDelegate {
    func myVideoCapture(_ capture: AgoraCameraSourcePush, didOutputSampleBuffer pixelBuffer: CVPixelBuffer, rotation: Int, timeStamp: CMTime)
}

3. Implement the custom video renderer

The Agora SDK does not support rendering video frames captured in the push mode. Therefore, you need to implement a custom video renderer using methods from outside the SDK. In the sample project, we define a class called CustomVideoSourcePreview using the native AVCaptureVideoPreviewLayer class.

// Swift
// Initializes localVideo
var localVideo = CustomVideoSourcePreview(frame: CGRect.zero)

// Defines the CustomVideoSourcePreview class
class CustomVideoSourcePreview : UIView {

    private var previewLayer: AVCaptureVideoPreviewLayer?
    func insertCaptureVideoPreviewLayer(previewLayer: AVCaptureVideoPreviewLayer) {
        self.previewLayer?.removeFromSuperlayer()

        previewLayer.frame = bounds
        layer.insertSublayer(previewLayer, below: layer.sublayers?.first)
        self.previewLayer = previewLayer
    }

    override func layoutSublayers(of layer: CALayer) {
        super.layoutSublayers(of: layer)
        previewLayer?.frame = bounds
    }
}

4. Start capturing and rendering video frames

In the sample project, we create a customCamera instance using the AgoraCameraSourcePush class, and then we call startCapture to start the capturing and rendering process.

// Swift
// Initializes the AgoraCameraSourcePush class and sets the camera as the capturing device
customCamera = AgoraCameraSourcePush(delegate: self, videoView:localVideo)

// Calls startCapture of the AgoraCameraSourcePush class to start capturing video frames
customCamera?.startCapture(ofCamera: .defaultCamera())

5. Push the captured video frames to the SDK

Call the pushExternalVideoFrame method to push the captured video frames to the SDK.

// Swift
extension CustomVideoSourcePushMain:AgoraCameraSourcePushDelegate
{
    func myVideoCapture(_ capture: AgoraCameraSourcePush, didOutputSampleBuffer pixelBuffer: CVPixelBuffer, rotation: Int, timeStamp: CMTime) {
        let videoFrame = AgoraVideoFrame()
        videoFrame.format = 12
        videoFrame.textureBuf = pixelBuffer
        videoFrame.time = timeStamp
        videoFrame.rotation = Int32(rotation)

        // Pushes the captured video frames to the SDK
        agoraKit?.pushExternalVideoFrame(videoFrame)
    }

}

Custom video source and renderer (mediaIO)

The Agora SDK provides the following classes in the mediaIO interface:

• AgoraVideoSourceProtocol: Triggers callbacks that control video capturing.
• AgoraVideoSinkProtocol: Triggers callbacks that control video rendering.
• AgoraVideoFrameConsumer: Stores the captured video frames. In scenarios involving a custom video source, the AgoraVideoFrameConsumer class sends the captured video frames to the SDK. In scenarios involving a custom video renderer, the AgoraVideoFrameConsumer class sends the captured video frames to the renderer.

This section introduces how to implement a custom video source and renderer with the mediaIO interface.

Custom video source

The API call sequence for implementing the custom video source is as follows:

1. Create a customized video source instance

Create a customized video source instance with the AgoraVideoSourceProtocol class, and set the following callback logics:

• After receiving bufferType, specify the format of the captured video frames in the return value.
• After receiving shouldInitialize, initialize the custom video source, for example, by turning on the video capture device. The SDK maintains the AgoraVideoFrameConsumer instance contained in the callback.
• After receiving shouldStart, send video frames to the SDK using consumePixelBuffer or consumeRawData. You can modify the video frame parameters in AgoraVideoFrameConsumer according to your app scenario.
• After receiving shouldStop, stop sending video frames to the SDK.
• After receiving shouldDispose, release the custom video source, for example, by turning off the video capture device.

In the sample project, we implement a class called AgoraCameraSourceMediaIO based on AgoraVideoSourceProtocol, and we manage the video-capturing process with the callbacks contained in AgoraVideoSourceProtocol.

// Swift
// Uses AgoraVideoFrameConsumer to send and store captured video frames
var consumer: AgoraVideoFrameConsumer?

// Pushes the captured frames to the SDK
consumer?.consumePixelBuffer(pixelBuffer, withTimestamp: time, rotation: rotation)

extension AgoraCameraSourceMediaIO: AgoraVideoSourceProtocol {
    // Initializes the custom video source after receiving shouldInitialize
    func shouldInitialize() -> Bool {
        return initialize()
    }

    // Starts the custom video source after receiving shouldStart
    func shouldStart() {
        startCapture()
    }

    // Stops the custom video source after receiving shouldStop
    func shouldStop() {
        stopCapture()
    }

    // Releases the AgoraVideoConsumer instance after receiving shouldDispose
    func shouldDispose() {
        dispose()
    }

    // Specifies the format of the captured video frames as pixel buffer after receiving bufferType
    func bufferType() -> AgoraVideoBufferType {
        return .pixelBuffer
    }

    // Sets the content hint of the captured video frames as default
    func contentHint() -> AgoraVideoContentHint {
        return .none
    }

    // Sets the capture type as camera after receiving captureType
    func captureType() -> AgoraVideoCaptureType {
        return .camera
    }
}

2. Implement the custom video source

Refer to the following code samples when implementing the custom video source.

// Swift
private extension AgoraCameraSourceMediaIO {
    // Initializes capturing video frames
    func initialize() -> Bool {
        let captureSession = AVCaptureSession()
        captureSession.usesApplicationAudioSession = false
        let captureOutput = AVCaptureVideoDataOutput()
        captureOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]
        if captureSession.canAddOutput(captureOutput) {
            captureSession.addOutput(captureOutput)
        }

        self.captureSession = captureSession
        captureQueue = DispatchQueue(label: "Agora-Custom-Video-Capture-Queue")

        return true
    }

    // Starts capturing video frames
    func startCapture() {
        guard let currentOutput = currentOutput, let captureQueue = captureQueue else {
            return
        }

        currentOutput.setSampleBufferDelegate(self, queue: captureQueue)
        captureQueue.async { [weak self] in
            guard let strongSelf = self, let captureSession = strongSelf.captureSession else {
                return
            }
            strongSelf.changeCaptureDevice(toPosition: strongSelf.position, ofSession: captureSession)
            captureSession.beginConfiguration()
            if captureSession.canSetSessionPreset(.vga640x480) {
                captureSession.sessionPreset = .vga640x480
            }
            captureSession.commitConfiguration()
            captureSession.startRunning()
        }
    }

    // Stops capturing video frames
    func stopCapture() {
        currentOutput?.setSampleBufferDelegate(nil, queue: nil)
        captureQueue?.async { [weak self] in
            self?.captureSession?.stopRunning()
        }
    }

    func dispose() {
        captureQueue = nil
        captureSession = nil
    }
}

3. Call setVideoSource to set the custom video source

Call setVideoSource of the Agora SDK. Once setVideoSource is implemented, the shouldStart callback triggers the customized video capturing and calls consumePixelBuffer to send the captured frames to the SDK. After the SDK receives the captured video frames, you can call startPreview or setupLocalVideo to render the video frames.

// Swift
// Assigns AgoraCameraSourceMediaIO to customCamera
fileprivate let customCamera = AgoraCameraSourceMediaIO()

// Calls setVideoSource and passes customCamera to the SDK
agoraKit.setVideoSource(customCamera)

Custom video renderer

The API call sequence for implementing the custom video renderer is as follows:

1. Implement the custom video renderer

Implement AgoraVideoSinkProtocol and AgoraVideoFrameConsumer, and set the following callback logics:

• After receiving bufferType and pixelFormat, specify the format of rendered video frames in the return value.
• Control the video rendering process with the shouldInitialize, shouldStart, shouldStop, and shouldDispose callbacks.
• The SDK triggers renderPixelBuffer or renderRawData according to the video-frames format that you set in bufferType or pixelFormat.

In the sample project, we implement a class called AgoraMetalRender based on AgoraVideoSinkProtocol, and we control the video-rendering process with the callbacks in AgoraVideoSinkProtocol.

// Swift
extension AgoraMetalRender: AgoraVideoSinkProtocol {

    // Initializes the custom video renderer after receiving shouldInitialize
    func shouldInitialize() -> Bool {
        initializeRenderPipelineState()
        return true
    }

    // Starts rendering video frames after receiving shouldStart
    func shouldStart() {
    #if os(iOS) && (!arch(i386) undefined (!arch(i386) undefined (!arch(i386) && !arch(x86_64))
        guard CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly) == kCVReturnSuccess else {
            return
        }
        defer {
            CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
        }

        let isPlanar = CVPixelBufferIsPlanar(pixelBuffer)
        let width = isPlanar ? CVPixelBufferGetWidthOfPlane(pixelBuffer, 0) : CVPixelBufferGetWidth(pixelBuffer)
        let height = isPlanar ? CVPixelBufferGetHeightOfPlane(pixelBuffer, 0) : CVPixelBufferGetHeight(pixelBuffer)
        let size = CGSize(width: width, height: height)

        let mirror = mirrorDataSource?.renderViewShouldMirror(renderView: self) ?? false
        if let renderedCoordinates = rotation.renderedCoordinates(mirror: mirror,
                                                                  videoSize: size,
                                                                  viewSize: viewSize) {
            let byteLength = 4 * MemoryLayout.size(ofValue: renderedCoordinates[0])
            vertexBuffer = device?.makeBuffer(bytes: renderedCoordinates, length: byteLength, options: [])
        }

        if let yTexture = texture(pixelBuffer: pixelBuffer, textureCache: textureCache, planeIndex: 0, pixelFormat: .r8Unorm),
            let uvTexture = texture(pixelBuffer: pixelBuffer, textureCache: textureCache, planeIndex: 1, pixelFormat: .rg8Unorm) {
            self.textures = [yTexture, uvTexture]
        }
    #endif
    }
}

2. Call setLocalVideoRenderer to set the custom video renderer

Call setLocalVideoRenderer of the SDK to display the video in the local view.

// Swift
// Sets the local video renderer
if let customRender = localVideo.videoView {
    agoraKit.setLocalVideoRenderer(customRender)
}

Considerations

• Customizing the video source and renderer requires you to manage video capturing and rendering on your own.
  • When customizing the video source, you need to capture and process the video frames on your own.
  • When customizing the video sink, you need to process and render the video frames on your own.
• In scenarios involving the custom video renderer, the rotation parameter of the video frames in renderPixelBuffer or renderRawData may not be 0. This is probably due to the settings of video capturing, and you need to process the rotation information yourself.
• When you use MediaIO to capture CVPixelBuffer in the BGRA format on iOS, ensure that your video renderer is adapted to the BGRA format if the video is rendered by your custom video renderer instead of the SDK; otherwise, a black screen or screen flickering can occur.