Demo share – see how modelarts and hilens make the car run by itself


Author Li Tianyi

Summary:The hilens kit has been basically developed and can be deployed to hilens kit. The model selection is the yolov3 model based on darknet53, and the weight is the data set based on coco2014 training. The lane line detection is realized based on the traditional method of OpenCV, which can be developed and deployed through the whole process end cloud of modelarts AI gallery and hilens kit.

Click the portal to see the final video effect first →→ (PS: please ignore the background music)!

Demo share - see how modelarts and hilens make the car run by itself

Introduction to main process:

  1. (optional or ignored, depending on the quality of the camera. For the camera with large distortion, the distortion matrix and distortion coefficient of the camera shall be calculated first to correct the picture);
  2. Intercepting the region of interest and processing only the image region containing lane line information;
  3. Use perspective transformation for the region of interest;
  4. For different color lane lines, lane lines under different lighting conditions and lane lines with different clarity, different gradient thresholds are used according to different color spaces, and the color thresholds are processed differently. Each processing method is fused to obtain the binary diagram of lane line;
  5. Extracting pixels belonging to lane lines in binary image;
  6. Carry out histogram statistics on the pixels of the binary image, count the peak points on the left and right sides as the starting point coordinates of the left and right lane lines, and carry out curve fitting;
  7. The quadratic polynomial is used to fit the pixels of the left and right lane lines respectively (for the pixels with large noise, filtering can be carried out, or the random sampling consistency algorithm can be used for curve fitting);
  8. Calculate the curvature of the lane and the deviation position of the vehicle relative to the center of the lane;
  9. Effect display(feasible region display, curvature and position display).
  10. Detect the status of other vehicles on the road during driving and displayVehicle categoryConfidenceThe vehicle is detected through yolov3, and then the coordinates of the returned vehicle detection frame are perspective transformed with the current coordinates to obtain the approximate distance as the distance between vehiclesDistance dis


The main framework of this demo has been basically developed based on hilens kit. The model selection is yolov3 model based on darknet53, and the weight is the data set trained based on coco2014, while the lane line detection is realized based on the traditional method of OpenCV, so the amount of calculation is large, resulting in a slow overall speed.

About deployment and training – end cloud collaboration based on modelarts and hilens Kit:

1) Considering the deployment, it has been tested and deployed to hilens kit. By the way, it also supports running simulation tests in hilens stuido. It is more recommended in hilens studio. It is more convenient and does not need hardware support,Just install the pilot library on the hilens kit or hilens studio, how to install third-party libraries on hilens kit and hilens studio is very simple. Please refer to:

On the hilens Kit:…

On hilens Studio:…

2) If you want to train or optimize without hardware (such as GPU), it is recommended to use modelarts. One stop development and seamless connection to hilens kit. For the introduction of modelarts, please refer to:…

At the same time, it may also improve the running speed. Here are three algorithms. The pro test can be deployed to hilens kit reasoning. Oh, in the latestAI Gallery(original AI market) in addition to algorithms, there are also models, data sets and so on, which are very rich. You can explore them yourself and share your algorithms with other developers. Developers can create training by subscribing. It is very convenient. Compared with GitHub, it not only provides source code, but also provides hardware resources for training, Powerful Tesla V100 32GB version.

Demo share - see how modelarts and hilens make the car run by itself

(1)YOLOv3_Darknet53Yes, it is the famous yoov3, a classic target detection network. Later, yoov4 and yov5 (temporarily called this name), and AI gallery also launched yoov5. However, the pytorch framework does not support modules temporarily           Type conversion, so I gave it up for the time being.

        aboutYOLOv3_ The operation method and algorithm interface of darknet53 are introduced in detail, which will not be repeated here. You can refer to it. Note that for the model conversion part, please refer to the model conversion part in this blog:…

The alternate links of the algorithm are: -ccbd-420f-8999-2ea7b4bdd691

     (2)YOLOv3_Resnet18(GPU), if we only want to do vehicle detection, or in order to simplify the model and improve the speed, we can choose yolov3 with resnet18 as the backbone network, which is lighter and faster.

Also link to: -3e7a-4729-850b-069101d6e95c

     (3)YOLOv3_Resnet18(Ascend 910), which is not different from the version (2) above. It is mainly changed from GPU to ascend 910 training. You can choose freely.

The link is: -7eec-4977-8b66-3a7703e9fd22

Well, the algorithm is partially solved. What about the data?The AI Gallery of modelarts also thought of it. It provides a dataset. You can download it to your own OBS and import it for free (note that OBS requires a certain cost). Here I also provide 5164 pictures of people and cars in the open source dataset for everyone to use. Link: id=a337272b-6918-4282-83d6-2b15e2a4c716

Now that you have data sets and algorithms, it’s time for training. It can be completed well on modelarts. It can be completed quickly with the powerful Tesla V100 32GB or ascend 910.

The above overall operation process can be completed with reference to the following blog posts:…

Operation effect:

It takes about 300ms to complete one frame of processing as shown in the video, that is, the FPS is about 3, which is not slow and needs to be optimized. I think there is still a lot of room for optimization, such as:

(1) Preprocessing and post-processing take a lot of time, and the real reasoning is very fast. It can be accelerated from this aspect. I don’t know whether there is an improvement in starting multithreading. It should be;

(2) It should be able to give better play to the special hardware. Remember that there are dvpp and AIPP, which can be used for color gamut conversion;

(3) The post-processing and parsing of yolov3 is troublesome. Here, we use 80 types of coco, but we can only use two types of person and car, or add a little. At the same time, NMS is also time-consuming. If it can be implemented by operators and integrated into om model, it will speed up a lot;

(4) Python is used here. If C + + is used, it should be improved and the execution efficiency will be much higher.

(5) To optimize and speed up, we should analyze the time-consuming parts according to the hardware performance to maximize the hardware performance. Targeted and customized optimization is needed. I personally have a shallow understanding and hope to have the guidance of familiar students.


  1. The models and codes used in this demo are from open source code. It seems that it is a project involved in a course of teacher Wu Enda, but we can’t find the source. If you know, please reply below. Thank you.
  2. The model accuracy and running speed of this demo are not guaranteed and are only for learning and communication.
  3. Please give us more advice on other matters not covered.
  4. The code will be open source. Please look forward to it. Thank you.

This article is shared from the Huawei cloud community “demo sharing – when autopilot meets modelarts, modelarts AI gallery and hilens kit development”, original author: Tianyi_ Li 。

Click focus to learn about Huawei cloud’s new technologies for the first time~