Analysis of video stream scheme of picture transmission on cloud control platform

Time:2021-4-24

Introduction:This paper will summarize the video stream scheme of this cloud control platform picture transmission.

background
Arc is a cloud control platform based on in-depth customization of on-board equipment business, through which we can realize remote use of different types of on-board equipment. In order to let the remote users use the on-board equipment as in the local, the images of on-board equipment need to be sent back to the users in time. Therefore, image transmission capability is a core component of arc platform.

At first, we used minicap, which is widely used in the industry. The scheme obtains the screen data, compresses it to generate JPG image, and transmits it to the web for display frame by frame. Because the performance of the car machine is much worse than that of the mobile phone, the compressed image consumes a lot of CPU performance. On some low-end car machine devices, the compressed image can consume about 80% of the CPU, which is easy to cause the device to jam. At the same time, the image compression rate is not very high, the transmission consumes a lot of bandwidth, which causes the user to see the picture delay in low bandwidth.

Therefore, we need a solution to balance the image quality and CPU resource consumption. This paper will summarize the video stream scheme of this cloud control platform picture transmission.
Analysis of video stream scheme of picture transmission on cloud control platform

Ideas and methods
Analysis of video stream scheme of picture transmission on cloud control platform

Based on the basic transmission link of image data, in order not to consume the CPU resources of the device, we first thought that the image should not be compressed, but be transmitted to the server for processing. However, after investigation, the USB bandwidth transmission of the car machine can not meet the requirement of high-definition image transmission without compression. The original data of high-definition is very large, and only about three frames of data can be transmitted in one second.

Another idea is to use the hardware encoder on the device side to reduce the consumption of CPU resources. After research, Android 4.1 has basically brought its own H264 video encoder. Therefore, we decided to try to use the scheme of video stream. The video stream is encoded by hardware encoder on the device side, and transmitted to the web side for decoding and display through the server side.

Implementation plan
Analysis of video stream scheme of picture transmission on cloud control platform

The whole implementation scheme can be divided into the following three parts

Device side: responsible for screen acquisition and coding.
Server: deal with the transmission and control of video stream.
Web side: video stream decoding and display.

Picture acquisition and coding
The virtual display of Android is directly used for image acquisition. There are many ways to realize coding
Analysis of video stream scheme of picture transmission on cloud control platform

Because the Java solution can only support Android 5.0 or more machines, Android 4. X still accounts for a large proportion in the car market, which cannot be ignored. Therefore, we can only use CPP, which is compatible with Android 4.3 at least.

Transmission and control of video stream
RTMP / HLS / flvjs is the most common live broadcast scheme on the web. But these schemes have a delay of 1-3s at least. It has no impact on the general live broadcast platform, but for the cloud control platform with real-time interaction scenarios, it is required to achieve millisecond delay. Therefore, we finally decided to adopt the scheme of h264 bare stream transmission through socket, and the device side encoded H264 video stream was directly transmitted to the web side for playback.

At the same time, in order to improve the user experience, the flexible control is added to the transmission of video stream. By adding a cache queue in the server to monitor the front-end bandwidth load, and automatically adjust the frame rate and code rate according to the bandwidth status, the user’s sense of fluency is given priority.

Web display and decoding
The web side shows the scheme of using media source extension (MSE) + fragment MP4, encapsulating H264 bare stream into fragment MP4, decoding and playing through MSE API. The specific implementation refers to the open source jmuxer scheme.
**
Frame dropping and frame filling**
By default, the maximum frame rate generated by Android virtual display is 60fps, while we can feel fluent with our naked eye at 30fps. In order to save bandwidth, we define that the maximum output frame rate of video stream is 30fps. In the case of poor network bandwidth, we can also reduce the frame rate to avoid delay as much as possible. At the same time, Android mediacodec does not support controlling the frame rate, which is determined by the number of frames per second. Therefore, we need to control the frame rate by losing frames.

Win7 hardware decoder does not have low delay mode, it needs about 10 frames of data to start playing, while virtual display needs to implement frame supplement to eliminate decoding delay because the picture changes to generate image frames.
Analysis of video stream scheme of picture transmission on cloud control platform

We create an eglsurface to process the lost frame and supplement frame, control the frequency of eglstexture drawing into the eglsurface to lose frame through time interval, and supplement frame by repeatedly drawing in the last frame.

summary
The use of this scheme on arc platform can ensure the transmission quality and effectively improve the user’s sense of fluency. In theory, this scheme can also be applied to other similar cloud control platforms. If we do not need to support Android 4. X devices and use Java layer API to obtain video stream data, we can reduce the development and adaptation costs.

Original link
This article is the original content of Alibaba cloud and cannot be reproduced without permission.