How to solve SEI-related issues?
The Agora SEI
By default, Agora adds the encoding information of the current video to the transcoded H264/H265 SEI (Supplemental Enhancement Information) during Media Push. The encoding information is a JSON string. The following is the sample code:
{
"canvas": {
"w": 640,
"h": 360,
"bgnd": "#000000"
},
"regions": [
{
"uid": 1,
"alpha": 1.0,
"zorder": 1,
"volume": 50,
"x": 0,
"y": 0,
"w": 320,
"h": 360
},
{
"uid": 2,
"alpha": 1.0,
"zOrder": 1,
"volume": 89,
"x": 320,
"y": 0,
"w": 320,
"h": 360
}
],
"ver": "20190611",
"ts": 1535385600000,
"app_data": ""
}The definition of the parameters:
| Parameter | Description |
|---|---|
canvas |
The canvas information. It contains the following properties:w: Width (pixel) of the canvas. It corresponds to the width member in the LiveTranscoding class.h: Height (pixel) of the canvas. It corresponds to the height member in the LiveTranscoding class.bgnd: The background color (RGB) of the canvas, represented by a hexademical string. It corresponds to the backgoundColor member in the LiveTranscoding class. |
regions |
The layout information of the host. It corresponds to the transcodingUsers member in the LiveTranscoding class. It contains the following properties:suid: (Optional)The string user account of the host in this region. This parameter applies to scenarios where string user accounts are used to identify the host.uid: UID of the host in this region. It corresponds to the uid member in the TranscodingUser class.alpha: The transparency of the video frame of the host. The value range is [0.0, 1.0]. It corresponds to the alpha member in TranscodingUser .zorder: The layout position of the video frame of the host. The value range is [0, 100]. It corresponds to the zOrder member in TranscodingUser.volume: The volume (dB) of the host. The value range is [0, 100].x: The horizontal position of the video frame of the broadacaster from the top left corner of the Media Push. It corresponds to the x member in TranscodingUser.y: The vertical position of the video frame of the host from the top left corner of the Media Push. It corresponds to the y member in TranscodingUser.w: Width (pixel) of the video frame of the host. It corresponds to the width member in TranscodingUser.h: Height (pixel) of the video frame of the host. It corresponds to the height member in TranscodingUser. |
ver |
The version of the SEI protocol. The current version is 20190611. |
ts |
Timestamp (ms) of the current encoding information. |
app_data |
Extra user-defined information. It corresponds to the transcodingExtraInfo member in the LiveTranscoding class. |
The structure of SEI
The SEI information is as follows:
0000 0664bd7b 22617070 5f646174 61223a22 .d.{"app_data":"
0010 222c2263 616e7661 73223a7b 2262676e ","canvas":{"bgn
0020 64223a22 23666666 66666622 2c226822 d":"#ffffff","h"
0030 3a363430 2c227722 3a333630 7d2c2272 :640,"w":360},"r
0040 6567696f 6e73223a 5b7b2261 6c706861 egions":[{"alpha
0050 223a3235 352c2268 223a3634 302c2275 ":255,"h":640,"u
0060 6964223a 33313031 32373137 39312c22 id":3101271791,"
0070 766f6c75 6d65223a 32382c22 77223a33 volume":28,"w":3
0080 36302c22 78223a30 2c227922 3a302c22 60,"x":0,"y":0,"
0090 7a6f7264 6572223a 317d5d2c 22747322 zorder":1}],"ts"
00a0 3a313533 37393630 32333537 38332c22 :1537960235783,"
00b0 76657222 3a223230 31383038 3238227d ver":"20190611"}In which:
06: The SEI frame.64: The SEI frame type defined by the user. Here we define it as 100.bd: The length of the SEI frame. The following are some sample calculations rendered in decimal and hexadecimal:- If the length is 922, because 922 can be divided by 255 (
0xff) three times and the remainder is 157 (0x9d), thenbdisffffff9d. - If the length is 572, because 572 can be divided by 255 (
0xff) two times and the remainder is 62 (0x3e), thenbdisffff3e. - If the length is 234, because 234 divided by 255 (
0xff) gives 0 and the remainder is 234 (0xea), thenbdisea.
- If the length is 922, because 922 can be divided by 255 (
- Other digits: Content of the SEI frame.
FAQ
Q: If I use the SEI frame in Media Push, is it necessary to use the signaling system to send the layout information? Do I use either signaling or SEI?
A: Adding the H264/H265 SEI information to the video stream during Media Push is different from sending the uplink data. Only the app_data parameter of the SEI information relates to the uplink data.
Q: Does the signaling system send the layout information? Why does the SEI frame contain the layout information?
A: Agora puts the layout information in the SEI frame because the audience members in the LIVE_BROADCASTING profile also need the layout information of the current stream to set the window region of the hosts.
