Waymo’s technology has been extensively tested on open roads, closed test sites and emulators, so that every part of the automatic driving system can have strong, reliable and safe handling capability in its ODD.
Waymo’s autopilot system consists of three independent and tightly tested subsystems:
1) OEM certified base vehicle system;
2) Internal hardware systems such as sensors and computers;
3) Automatic driving software system for making all driving decisions;
These subsystems are combined to form a complete autopilot system, and then we further test and verify it. The test and verification of each system can ensure that the automatic driving vehicle meets all the safety requirements set by us for the system.
3. Testing and Validation Methods
3.1 test verification of base vehicle system
Waymo’s current autonomous vehicle is an improved version of the 2017 Chrysler Pacifica hybrid minivan into which we integrate self driving systems. The improved 2017 Chrysler passiflica hybrid minivan sold to us by Fiat Chrysler Automobile Company (“FCA”) has been certified to the federal motor vehicle safety standard (FMVSS), which specifies the safety performance requirements of motor vehicles or motor vehicle equipment in the United States.
3.2 test and verification of sensor and hardware system
1) Building a safe and reliable supply chain system
Waymo works with suppliers and distributors to ensure the performance, safety, quality and reliability of system components. We introduce the system components provided by suppliers into the process of failure mode, impact analysis (FMEA) and risk assessment, and detect and identify the potential safety risks in the manufacturing process, independent system or other system integration when integrating with other subsystems. We monitor the performance of various sub components during the manufacturing process and conduct continuous reliability testing to ensure that they meet design expectations and safety requirements before they are integrated into our vehicles.
2）Testing At Every Level
Our multi-level safety testing approach is influenced by the engineering experience of NASA’s Mars probe, an autonomous vehicle operating millions of miles from earth.
This approach means that we need to analyze and test the lowest level components to ensure the performance and reliability of our most critical systems. For example, after our vision (camera) system is installed in the vehicle system, it has to conduct more than 100 separate tests in the laboratory.
First of all, we check the cameras, cables and connectors that make up the vision system to ensure that each part meets the design specifications. When the camera is integrated, we test again, assembling each camera into a ring, calibrating them to work together, and checking whether the angle and direction of each camera combined together can form a 360 degree panorama.
Then the whole camera system is tested. The assembled cameras are added to the automatic drive system, which is tested by our engineers to ensure that simultaneous interpreting of different sensors can work as a sensor.
Before using this new vision system on the road, we perform tasks such as seeing traffic lights under various lighting conditions, detecting pedestrians and discovering construction sites to confirm that the vision system is working. After all tests have passed, the vision system is allowed to go on the road as part of the autonomous vehicle.
3) Hardware system test
Through the technical cooperation between FCA and waymo, we have designed and integrated waymo’s autopilot sensors and hardware systems on the improved Chrysler parsifca hybrid minivan provided by FCA. To ensure that we have integrated the autopilot system into the Chrysler Pasi FICA hybrid truck, Waymo has added thousands of additional tests to the integrated minivans on private test roads, laboratories and simulators to assess the safety functions of vehicle from braking, steering, headlights and physical control of doors. Through these tests, we can ensure the safe operation of the vehicle in all scenarios, such as manual mode, automatic driving mode with test driver and fully automatic driving mode when there is no one in the car.
3.3 automatic driving software system test
Our autopilot software is also guided by our safety design concept, continuously and strictly tests all components of the software, including perception, behavior prediction and planning. Every change to the software must pass the simulation test, closed road test and public road driving test.
We identified the most challenging situations encountered by vehicles on public roads and converted them into virtual scenes so that the autopilot software could be continuously validated in the simulator.
The new version of autopilot software is first pushed to several cars, and the most experienced drivers can test it on our private test road. We can use different versions of software for different vehicles in order to test new R & D functions or specific functions.
Once we confirm that the autopilot software works as expected, we begin to deploy new software on public roads. First of all, small scale tests, self driving vehicles must prove that they can safely and consistently follow the scheduled route, and then we push software updates to the entire fleet. With the increase of automatic mileage, we continuously improve our driving style and software system based on the feedback of vehicle operation.
3.4 simulation system
Waymo’s simulator can reproduce our real-world driving behavior with each new version of the software, as well as build new virtual scenes for our software to test. Up to 25000 virtual waymo autonomous vehicles drive 8 million miles in the simulator every day to verify the improvement of old technology and test the effect of new technology, so as to help the automatic driving vehicle drive safely in the real world.
For example, at the corner of South Longmore street and South West Street in mesa, Arizona, there is a yellow flashing arrow to indicate a left turn. This type of intersection is very difficult for both human and self driving vehicles. The driver must enter a five Lane intersection and find a gap in the oncoming vehicles. An early left turn may create a gap for oncoming vehicles It is dangerous to turn left too late, which may dissatisfy the drivers behind. Simulation allows us to turn such a real-world encounter into thousands of opportunities to practice and master skills.
How Simulation Works
Step 1：Start with a Highly-Detailed Vision of the World
We build a set of virtual copies including lanes, curbs, traffic lights, intersections, etc. in the simulation, we can focus on the most challenging scenarios, such as yellow lights flashing, retrograde vehicle drivers or fast cycling cyclists, rather than just monotonous accumulation of road mileage.
Step 2: Drive, Drive, and Redrive
When the flashing yellow left turn is digitized in the virtual world, our software can debug and verify the scene thousands of times. Every time we update the software, we can test and verify it at the same intersection under different driving conditions. That’s how we teach our vehicles to move forward naturally with the yellow lights flashing and insert into oncoming traffic. In addition, we can test this new skill on every flickering yellow flashing scene we have encountered, so as to upgrade and improve our software system faster.
Step 3: Create Thousands of Variations
Next, we can explore thousands of different variation scenarios similar to flashing yellow lights through a process called “fuzzing.”. We change the speed of oncoming vehicles and the time of traffic lights to ensure that our vehicles can still find safe gaps in the traffic flow; by adding simulated pedestrian and motorcycle separation lanes, even joggers zigzagging through the streets, etc. All of these can make the scene more complex, and see how this will change the response mode of the autopilot system.
Step 4: Validate and Iterate
Our autopilot has learned how to confidently drive at a glittering yellow arrow intersection. This new technology has become part of our permanent knowledge base and is shared among all the vehicles of the entire fleet. Then we will use real-world driving test and closed road test to verify whether it is consistent with the driving behavior of the simulation scene. Simulation and real driving behavior interact and iterate to achieve the goal of automatic driving: to achieve billions of miles of safe and stable driving in the real world.
3.5 Field Tests at Our Closed-Course Facility
Waymo has set up a 91 acre private, enclosed automated driving test facility in California designed and built for our own unique test requirements. The private building, known as castle, is like a simulated city, with scenes from highways to suburban lanes to rail intersections. Our team uses these closed scenario test facilities to test and validate the new software before it is released to our fleet. In addition, we use closed automatic driving sites to build challenging or rare road scenes to help our autopilot system get the corresponding driving experience. In closed spaces, we were able to recreate specific learning and testing scenarios and conduct thousands of “structured tests” on them. To enhance the effectiveness of the simulation test, we created more than 20000 simulation scenes in the “Castle” closed field, each of which is a reproduction of the driving scene we want: an aggressive driver speeding out of the lane, or a pedestrian suddenly appearing beside a parked car, which may take hundreds of thousands of miles to meet on public roads Once. In addition, there are scenes of people jumping from canvas bags or portable toilets to the roadside, skateboarders lying on skateboards and throwing paper piles in front of sensors. This “structured testing” is essential to accelerate our technological progress and ensure the safety of our vehicles in daily and challenging driving environments.
3.6 Behavioral Competencies for Normal Driving
Fully autonomous vehicles must be able to handle all the daily driving tasks expected by human drivers within the odd. The U.S. Department of transportation recommends that level 3, level 4, and level 5 autonomous vehicles should be able to demonstrate at least 28 core capabilities. Waymo has expanded 28 core capabilities in breadth and depth, and we have tested thousands of complex scene changes to ensure that our system can safely meet the challenges of the real environment.
For each capability, waymo’s team created a variety of individual tests that were run over and over in our closed space. For example, in order to test our ability to make unprotected left turns, we designed dozens of real situations (including multi Lane oncoming vehicles, large trucks blocking our view, turning in short traffic light scenarios, etc.), and tested whether our vehicles responded appropriately.
For each scenario, we use simulators to create hundreds of variations of the same scene. Through our virtual world test, we can also create a new unprotected left turn scene to further test this skill. As our odd expands, the number of core competencies is likely to increase (for example, to drive year-round in the northern states of the United States, our systems must have the ability to drive safely in the snow), and the number of tests in each category may increase with more unique or complex scenarios.
3.7 Testing the Fully Integrated Self-Driving Vehicle
After testing the basic vehicle system, self driving system and software system respectively, it is necessary to test the integrated automatic driving vehicle. These tests include closed road collision avoidance tests, reliability and durability tests, and actual road tests by trained test drivers.
Testing on Public Roads
Waymo has a comprehensive road testing process, which has been continuously improved and improved over the past eight years. This is a key step in autopilot, enabling us to validate the skills we have developed, discover new challenge scenarios and develop new driving capabilities.
We have participated in the road driving safety training course (including the safety test of the whole road driver, and how to understand the safety of the driving system). After training, our drivers are responsible for monitoring the system during road testing and take over control of the vehicle when necessary.
We conduct tens of thousands of miles of open road testing every week to evaluate our software systems to ensure that they demonstrate the behavioral capabilities of human drivers. Real world testing provides a continuous feedback loop that allows us to continuously improve our system. This iterative approach to public road testing and validation helps us safely extend our odd and technical capabilities.
3.8 Real-World Experience
Over the past eight years, waymo has tested our autonomous vehicles in more than 20 cities in four states of the United States. More than 3.5 million miles have been traveled from Phoenix, Arizona, to the rainy city of Cochrane, Washington. As we expand to new test areas, we can collect more driving experience from different road environments, street views and driving habits. Driving in Phoenix, for example, enabled us to test sensors and software in desert conditions, including extreme temperatures and dust in the air; we learned how to deal with special types of vehicles (such as sprinklers spraying plants in the middle of the road, driving at 3 mph on a 45 mph Road); encountering horizontal traffic signals for the first time in Austin; and in Coe Crane had a scene of wet weather.
In every new city, we will encounter people who are not accustomed to seeing the automatic driving every day. This also enables us to understand what people think of autopilot cars: for example, how people want to use autopilot cars; what they think of automatic driving; and more people tell us how to develop and improve self driving technology.
Our autopilot requires reliable and safe operation under extreme cold and high temperatures. High temperature challenges all modern technologies. Mobile phone and other electronic products will overheat and shut down in the sun. However, our automatic driving system needs to operate safely under high temperature. Our cars are equipped with a special cooling system that allows them to operate at very hot temperatures (even if the engine is running at full power and the system is running at full load). Our engineers have carried out extensive tests in wind tunnels that can simulate almost any weather condition, including the hottest temperature on earth ever.
In addition to wind tunnel testing, we tested our autopilot cars in the three hottest parts of the United States: Las Vegas, Davies dam and death valley. Davis dam, on the Arizona Nevada border, has a steep desert highway that allows us to drive in the hot sun; the Las Vegas strip allows us to test our systems on hot, busy streets; Death Valley has an official record of 134 degrees Fahrenheit.
During the test, we closely monitor our system, making more than 200 different measurements per second to confirm that our internal sensor suite and calculations are working properly.
3.9 Testing Crash Avoidance Capabilities
In addition to testing core behavioral capabilities, our engineers also conduct collision avoidance tests across a variety of scenarios. Waymo has completed thousands of crash tests on our private test road, each of which reproduces a unique driving scenario and allows us to analyze the vehicle’s response, and then we use simulators to further test these scenarios and improve our overall software capabilities.
We learned about the crashes to be tested from a variety of sources, including our NHTSA fatal crash database, and using our extensive self driving vehicle experience to expand NHTSA’s 37 pre crash scenarios. We also tested other road users for potentially dangerous situations, such as sudden exit of vehicles from the lane, large vehicles crossing the target lane, motorcycles crossing traffic, and pedestrians crossing the road.
In 2015, the national highway traffic safety administration released data on the distribution of the most common pre crash scenarios. For example, four types of traffic accidents account for 84% of the total accident scenarios: rear end crashes, vehicles turning or crossing at an intersection, vehicles running off the edge of the road, and vehicles changing lanes. Therefore, to avoid or mitigate these types of accidents is an important goal of our test.
3.10 Hardware Reliability and Durability Testing
Like conventional vehicles, autonomous vehicles must operate reliably, which means that the vehicle and each individual component must operate normally under extreme environmental conditions and throughout the service life of the vehicle.
Waymo engineers use their understanding of physical failures to design unique stress tests that accelerate environmental testing of our vehicles and their components, and we compress years of actual use into days and weeks of testing. For example, we irradiate our components with ultraviolet light, bombard them with powerful water sprayers, immerse them in almost frozen water tanks, corrode them in a room full of salt spray, vibrate them with strong vibrations, and then heat and freeze them for weeks at specific temperatures and humidity. We analyze the failures in any test and make design improvements to improve the reliability of the components. We monitor the health of each sensor and the vehicle itself to identify and fix potential faults before they occur.
Google waymo 2017 automatic driving safety technology report (1)
Automatic driving high precision map – overview and analysis
Waymo – automatic driving long tail challenge (2019)
Personal blog website address: http://www.banbeichadexiaojiubei.com