Waymo – automatic driving long tail challenge (2019)

Time:2020-9-30

Although waymo has accumulated more than 10 million miles on open roads, waymo’s engineers still find that there are endless new autonomous driving scenarios to be solved.

1. Examples of long tail scenarios for autonomous driving

Scene oneA cyclist holds a stop sign sign sign in his hand. We don’t know when it’s going to lift the sign. A driverless car must understand this scenario. Even if he raises the Stop Sign sign, the autopilot should not stop.

Waymo - automatic driving long tail challenge (2019)

Scene 2:The oncoming vehicles loaded with plastic pipes are scattered all over the place. The autopilot must learn to deal with this sudden situation and avoid their influence on the driverless vehicle.

Waymo - automatic driving long tail challenge (2019)

Scene 3:Due to road construction and other factors, the pavement is covered with cone barrels. The unmanned vehicle must recognize these scenes correctly and realize reasonable driving in the scene full of cone barrels on the road.

Waymo - automatic driving long tail challenge (2019)

Scene 4:The traffic light is green, and the unmanned vehicle has the right of way. Although our unmanned vehicle arrives at the intersection first, it must give way to the special vehicles that will arrive later.

Waymo - automatic driving long tail challenge (2019)

Scene 5:When there is a green light at the intersection, the unmanned vehicle is ready to turn left. When encountering the social vehicles running through the red light at high speed, the unmanned vehicle needs to identify this scene and stop in time to avoid the illegal vehicles.

Waymo - automatic driving long tail challenge (2019)

2. Autopilot core modules – perception, prediction and planning

Perception, prediction and planning modules are the core modules of automatic driving, and each module has great challenges.

2.1 Perception

Perception input: the input information of the sensor (LIDAR) and the prior information of the scene.

Perception output: road traffic objects (pedestrians, vehicles, etc.), semantic segmentation and understanding of road scenes.

Waymo - automatic driving long tail challenge (2019)

Perception itself is a very complex and difficult problem, it must be able to identify various forms of different types of objects. As shown in the picture on the left below, a group of pedestrians in dinosaur suits must be able to recognize them correctly.

Waymo - automatic driving long tail challenge (2019)

The appearance performance of the same object at different times and seasons will also vary greatly, which will bring great challenges to perception.

Waymo - automatic driving long tail challenge (2019)

It is very difficult to understand the segmentation of various complex scenes. As shown in the figure below, the first on the left: a man carrying a box; the third from the left: a man riding a horse. Perception must be able to correctly segment and recognize these scenes, without any recognition errors caused by occlusion.

Waymo - automatic driving long tail challenge (2019)

2.2 Prediction

Perception predicts the next step behavior of the detected objects, so as to assist automatic driving vehicles to make reasonable behavior decisions.

Waymo - automatic driving long tail challenge (2019)

Perception should consider the historical behavior of objects. For example, the vehicle will not make a 90 degree turn in a short period of time. Therefore, we can assume that the vehicle is still moving in accordance with the current direction and speed in a short period of time; we should have a higher semantic understanding of the scene; we should be able to pay attention to the attribute differences and visual cues of different objects, such as the probability that the vehicle will drive on the lane, The pedestrian will walk the zebra crossing, and the direction of the vehicle can roughly reflect its intention. If the pedestrian makes a parking gesture, the probability is to cross the road; to be able to solve the behavior interaction between the predicted object and other objects.

As shown in the figure below, there is a stationary vehicle on the roadside. Cyclists will intrude into the driveway when approaching the stationary vehicle. Perception module needs to correctly understand these scenarios and generate reasonable prediction curve.

Waymo - automatic driving long tail challenge (2019)

How to accurately predict the behavior of social vehicles is still an open problem with great challenges.

2.3 Planning

Planning is the decision making machine. It plans the vehicle behavior based on the output of perception and prediction, and outputs the control module to control the acceleration, deceleration, braking and other behaviors of the vehicle.

Waymo - automatic driving long tail challenge (2019)

The first consideration of planning is safety, the second is comfort, the third is to be able to interact correctly with other traffic participants, and finally to ensure that passengers arrive at the destination. How to meet these conditions and achieve good planning effect is still an open question.

Waymo - automatic driving long tail challenge (2019)

3. Machine learning at scale

Machine learning is an effective tool to solve the long tail problem of automatic driving. Using machine learning technology can realize the closed-loop cycle process from data acquisition, tagging, training and vehicle deployment, so as to realize the continuous accumulation of cases and the continuous improvement of the model.

Waymo - automatic driving long tail challenge (2019)

3.1 automated machine learning technology

Waymo uses automated machine learning technology to generate and optimize the data model for unmanned vehicles, which greatly improves the efficiency of model training.

Waymo - automatic driving long tail challenge (2019)

Waymo - automatic driving long tail challenge (2019)

Waymo - automatic driving long tail challenge (2019)

Waymo - automatic driving long tail challenge (2019)

3.2 limitations of machine learning

The machine learning model can not solve all the problems, but what we need is a safe automatic driving system. Therefore, there must be other measures to supplement the deficiency of ML.

First of all, this problem can be solved with the help of redundant complementary sensors. At the same time, the vehicle is equipped with vision, lidar and radar systems. Each system is independent and complementary to each other to ensure that the unmanned vehicle will not lose any information.

Waymo - automatic driving long tail challenge (2019)

Secondly, we can use ml and non ml hybrid system, and use expert system to make up for the shortage of ml.

Waymo - automatic driving long tail challenge (2019)

4. Large scale testing

First, waymo has a large fleet of autonomous drivers that can support large-scale tests.

Waymo - automatic driving long tail challenge (2019)

In order to test and verify these low-frequency problems, we need to build our own scenarios and conduct structured tests.

Waymo - automatic driving long tail challenge (2019)

Simulation is an important verification test method, which can construct a variety of test scenarios lightweight and safely.

Waymo - automatic driving long tail challenge (2019)

Automatic driving simulation must be able to simulate the behavior of vehicles and pedestrians. It is not enough to rely on simple rule model. We need more complex model. Waymo uses a mid-2-mid drive agent machine learning model, which receives positioning, perception and other information, and outputs more humanized motion planning.

Waymo - automatic driving long tail challenge (2019)

Chauffeurnet proposed by waymo transforms map, traffic rules, road environment and other information into image information, so as to maximize the use of more mature machine learning model, and finally output agent trajectory.

Waymo - automatic driving long tail challenge (2019)

Chauffeurnet can solve the prediction and planning problems in most simple scenarios.

Waymo - automatic driving long tail challenge (2019)

The red trail in the scene is the historical trajectory of the agent, and the green is the prediction trajectory of the next 2S.

Waymo - automatic driving long tail challenge (2019)

The scene where the main vehicle successfully passes the stationary vehicle on the side of the road

Waymo - automatic driving long tail challenge (2019)

The main vehicle decelerates after encountering a slow moving vehicle

Of course, chauffeurnet also has its limitations. For example, the following complex scenarios cannot be well handled.

Waymo - automatic driving long tail challenge (2019)

The main car ran out of the intersection directly because of the sight distance

Waymo - automatic driving long tail challenge (2019)

The vehicle did not complete the U-turn successfully

5. Long tail problem challenges that machine learning cannot cover

The biggest challenge for autopilot testing is that it is difficult to collect all corner cases. As shown in the figure below, it is the distribution of human driving behavior. It takes a very long time to accumulate some corner driving behavior cases.

Waymo - automatic driving long tail challenge (2019)

In the neural network model of automatic driving network, there may be tens of millions of parameters. If the number of corner case samples is too small, it is difficult to ensure that the network model can learn these corner scenarios.

Before the neural network model covers the long tail case, how to solve the long tail case? Expert systems are an option. Expert system with professional knowledge, through a small number of samples can obtain better results of the parameters.

For example, we plan to achieve a trajectory optimization machine learning model. After designing the expert model based on the motion control theory and a series of constraints, we can adjust the parameters to minimize the cost method by collecting the historical vehicle trajectory, so that the trajectory output of the expert system can approach the human driving trajectory as much as possible.

Waymo - automatic driving long tail challenge (2019)

Another model of trajectory optimization expert system is inverse reinforcement learning technology. Through training model parameters of historical driving trajectory, its output can approach the expected effect as much as possible.

Waymo - automatic driving long tail challenge (2019)

As shown in the figure below, the main car in red and the social vehicle in blue. The social vehicles on the left are more conservative and the social vehicles on the right are more radical. The model trained with conservative trajectory tends to be conservative, while the model trained with radical trajectory tends to be radical.

Waymo - automatic driving long tail challenge (2019)

6. Smart agent is indispensable for the scale of automatic driving

Whether expert system or neural network, they are trying to simulate the driving behavior of human, make the agent become smart, intelligent agent can assist the rapid scale of automatic driving technology.

Waymo - automatic driving long tail challenge (2019)

Waymo - automatic driving long tail challenge (2019)

Waymo - automatic driving long tail challenge (2019)