Through the NERTC SDK, you may package user-defined information, including volume information, as part of the SEI into video streams and forward the streams to remote users for decoding and viewing. This can be used in scenarios such as audio and video synchronization.

Overview

In RTC streaming media, the user’s message delivery channel is separated from that of RTC or live streaming, thus it is typically hard to secure synchronicity of message and video data. NERTC incorporates timestamp and other user-defined data as part of Supplemental Enhancement Information (SEI), packages and forwards the data together with the video contents to remote users through the streaming media channel, thus achieving accurate synchronization of text data and RTC contents.

In a live streaming scenario, the interactive live streaming 2.0 automatically re-packages the SEI of RTC room members involved in interactive live streaming into CommsEase’s customized SEI data type and pushes the data to the subscriber. You can use CommsEase’s player SDK to automatically resolve the packaged user-defined data in the video streaming SEI.

SEI is typically used in the following scenarios:

• Online education, quiz and other scenarios requiring synchronization of the teacher’s voice and the question to be answered.
• EC shopping scenario requiring synchronization of product information and the host’s audio and video.
• Pan-entertainment online KTV scenario requiring synchronization of the lyrics, voice and video when singing in chorus.

Considerations

• The SEI information is forwarded through the mainstream channel by default. Whilst calling sendSEIMsg, you may also specify the forwarding of the SEI through the substream channel but make sure that the channel has been enabled before forwarding.
• In an audio-only scenario, the SEI frames are forwarded in the form of Fake Video, where data are not charged as video streamings.

Method

RTC live streaming scenario

Local:

1. After joining a room, the client may enable video streaming by calling enableLocalVideo.
2. After video streaming is enabled, call the sendSEIMsg API for forwarding the SEI.

Remote:

Upon receipt of the SEI fowarded from the local port, the onRecvSEIMsg callback will be triggered.

Audio-only communication scenario

In an audio-only communication scenario, if in need of forwarding the SEI to a remote port, the SDK will automatically resort to the Fake Video solution. This feature is implemented as follows:

1. The local port calls sendSEIMsg to forward the SEIinformation.

   In this case, the SDK automatically generates a Fake Video video streaming carrying the SEI which will be forwarded to the remote port. Fake Video resolution: 16×16, full black display.

2. After receiving the Fake Video signaling notice forwarded through the audio-only streaming, the remote port will determine whether the video streaming is Fake Video through the FAKE_PROFILE field of NERtcConstants.VideoProfile.

   If so, this video streaming shall be subscribed to, but the video needs not be shown.

3. After receiving the SEI forwarded from the local port, the remote port will trigger the onRecvSEIMsg callback.

In an audio-only communication scenario, after implementing the SEI frame forwarding through forwarding Fake Video, if you need to forward video data normally through enabling video streaming, you may enable video streaming by calling enableLocalVideo and call sendSEIMsg to continue forwarding the SEI. In this case, the SDK will automatically determine whether Fake Video has been enabled. If yes, Fake Video will be disabled before enabling video streaming.

Sample code

// Forward the SEI, through the mainstream by default
//int ret = NERtcEx.getInstance().sendSEIMsg(seiMsg);
//Alternatively you can specify the SEI forwarding through the mainstream or substream SEI
int ret = NERtcEx.getInstance().sendSEIMsg(seiMsg, nERtcVideoStreamType);
if (ret !=  NERtcConstants.ErrorCode.OK) {
    showToast("SEI forwarding failed, ret : " + ret);
}

//Receive the SEI, NERtcCallbackEx#onRecvSEIMsg(long userID, String seiMsg)
public class NERtcCallbackImpl implements NERtcCallbackEx{

//todo SDK other callback methods

@Override
public void onRecvSEIMsg(long userID, String seiMsg) {
  }
}

The SEI format of video streams in Interactive Live Streaming

In an interactive live streaming scenario, when the 2.0 server transcodes and pushes streams, the encoding information of the current video will be added to the transcoded H264/H265 SEI, including user-defined SEI uploaded through the client SDK, packaged as CommsEase’s customized SEI type. You can use the CommsEase’s player SDK to automatically resolve the packaged user-defined data in the video streaming SEI.

SEI format

The SEI format is in JSON-format strings, e.g.,:

{
    "canvas":{
        "w":640,
        "h":360,
        "bgnd":"#000000"
    },
    "regions":[
        {
            "uid":1,
            "alpha":1,
            "zorder":1,
            "volume":50,
            "x":0,
            "y":0,
            "w":320,
            "h":360
        },
        {
            "uid":2,
            "alpha":1,
            "zOrder":1,
            "volume":89,
            "x":320,
            "y":0,
            "w":320,
            "h":360
        }
    ],
    "rtc_sei":[
        {
            "uid":1,
            "mainSei":"xxxxx",
            "subSei":"xxxxx"
        },
        {
            "uid":2,
            "mainSei":"xxxxx",
            "subSei":"xxxxx"
        }
    ],
    "ver":"20190611",
    "ts":1535385600000,
    "app_data":"",
    "extraInfo":"abc"
}

Field description