Waymo just published a study showing that they can outperform an unimpaired (NIEON) driver on a set of real-world crashes. That's promising, but it's only half the story. To be a safe driver, an Autonomous Vehicle (AV) must both not only be good -- but also not be bad. Those are two different things. Let me explain...
Learning to drive includes learning how to drive well.
It also includes learning to avoid making avoidable mistakes. They're not the same thing.
A Sept. 29, 2022 blog posting by Waymo explains their next step in showing that their automated driver can perform better than people at avoiding crashes. The approach is to recreate crashes that happened in the real world and show that their automated driver would have avoided them.
For this newest work they compare their driver to not just any human, but a Non-Impaired, with Eyes always ON the conflict (NIEON) driver. They correctly point out that no human is likely to be this good (you gotta blink, right?), but that's OK. The point is to have an ideal upper bound on human performance, which is a good idea.
Setting a goal that to be acceptably safe an AV should be at least as good as a NIEON driver makes sense. It cuts out all the debate about which human driver is being used as a reference (e.g., a tired 16 year old vs. a well rested professional limo driver 50 year old will have very different driving risk exposure).
Unsurprisingly Waymo does better than a NIEON on scenarios they know they will be running, because computers can react more quickly than people when they correctly sense, detect, and model what is going on in the real world. Doing so is not trivial, to be sure. As Waymo describes, this involves not just good twitch reflexes, but also realizing when things are getting risky and slowing down to give other road users more space, etc.
That is the "be a good driver" part, and kudos to Waymo and other industry players who are making progress on this. This is the promise of AV safety in action.
But to be safe in the real world, that is not enough. You also have to not be a bad driver. Doing better at things humans mess up is half the picture. The other half is not making "stupid" mistakes that humans would likely avoid.
It is good to see Waymo showing their automated driver can do well when it correctly interprets the situation it is in. That shows it is a potentially capable driver. We still need to see it is additionally not making mistakes in novel situations that aren't part of the human driver crash dataset. If Waymo can show NIEON level safety in both the knowns and the unknowns, that will be an impressive achievement.
Missing Rare Events in Simulation:
A highly accurate simulation and system model doesn't solve the problem of what scenarios to simulate. If you don't know what edge cases to simulate, your system won't be safe.
It is common, and generally desirable, to use vehicle-level simulation rather than on-road operation is used as a proxy field testing strategy. Simulation offers a number of potential advantages over field testing of a real vehicle including lower marginal cost per mile, better scalability, and reduced risk to the public from testing. Ultimately, simulation is based upon data that generates scenarios used to exercise the system under test, commonly called the simulation workload. The validity of the simulation workload is just as relevant as the validity of the simulation models and software.
Simulation-based validation is often accomplished with a weighting of scenarios that is intentionally different than the expected operational profile. Such an approach has the virtue of being able to exercise corner cases and known rare events with less total exposure than would be required by waiting for such situations to happen by chance in real-world testing (Ding 2017). To the extent that corner cases and known rare events are intentionally induced in physical vehicle field testing or closed course testing, those amount to simulation in that the occurrence of those events is being simulated for the benefit of the test vehicle.
A more sophisticated simulation approach should use a simulation “stack” with layered levels of abstraction. High level, faster simulation can explore system-level issues while more detailed but slower simulations, bench tests, and other higher fidelity validation approaches are used for subsystems and components (Koopman & Wagner 2018).
Regardless of the mix of simulation approaches, simulation fidelity and realism of the scenarios is generally recognized as a potential threat to validity. The simulation must be validated to ensure that it produces sufficiently accurate results for aspects that matter to the safety case. This might include requiring conformance of the simulation code and model data to a safety-critical software standard.
Even with a conceptually perfect simulation, the question remains as to what events to simulate. Even if simulation were to cover enough miles to statistically assure safety, the question would remain as to whether there are gaps in the types of situations simulated. This corresponds to the representativeness issue with field testing and proven in use arguments. However, representativeness is a more pressing matter if simulation scenarios are being designed as part of a test plan rather than being based solely on statistically significant amounts of collected field data.
Another way to look at this problem is that simulation can remove the need to do field testing for rare events, but does not remove determine what rare events matter. All things being equal, simulation does not reduce the number of road miles needed for data collection to observe rare events. Rather, it permits a substantial fraction of data collection to be done with a non-autonomous vehicle. Thus, even if simulating billions of miles is feasible, there needs to be a way to ensure that the test plan and simulation workload exercise all the aspects of a vehicle that would have been exercised in field testing of the same magnitude.
As with the fly-fix-fly anti-pattern, fixing defects identified in simulation requires additional simulation input data to validate the design. Simply re-running the same simulation and fixing bugs until the simulation passes invokes the “pesticide paradox.” (Beizer 1990) This paradox holds that a system which has been debugged to the point that it passes a set of tests can’t be considered completely bug free. Rather, it is simply free of the bugs that the test suite knows how to find, leaving the system exposed to bugs that might involve only very subtle differences from the test suite.
(This is an excerpt of our SSS 2019 paper: Koopman, P., Kane, A. & Black, J., "Credible Autonomy Safety Argumentation," Safety-Critical Systems Symposium, Bristol UK, Feb. 2019. Read the full text here)
Beizer, B. (1990) Software Testing Techniques, 2nd Ed., 1990.
FiveAI has published an autonomous vehicle safety approach that includes independent verification, transparency, and data sharing. (I provided inputs to the @_FiveAI authors.)
I enjoyed presenting at AutoSens 2018 today. The audience was very engaged and asked great questions.
Here are my slides. (If you seen my other recent slide decks probably not a lot of surprises, but I remixed things to emphasize perception validation.)
Developers should create a transparent safety argument based on testing, simulation, and good engineering practices.
The race to deploy self-driving cars is in full swing, with fleets being deployed on public roads in increasing numbers. While some vehicles require driver supervision, other fleets are being deployed which don’t even have driver controls at all. The big question is: will these vehicles be safe enough?
Now that self-driving car technology is maturing, there is increasing recognition of the fact that demonstrating adequate levels of safety with on-road testing is impractical. Too many billions of miles are needed for a credible statistical safety argument, and it simply costs too much and takes too long to do that. Other safety critical application areas already have a solution to this problem in the form of internationally accepted safety standards (e.g., aircraft, trains, and even home appliances). While there is an international safety standard for conventional cars (ISO-26262), its use is not currently required by the US government.
The stakes are high. Even a single bad line of software code can — and has — caused catastrophic failures. While nobody should expect this technology to be perfect, it’s important to ensure that it follows accepted practices and is appropriately safe. Assuming we have a quantification of how much better than a human driver the technology needs to be, the technologists need to make sure that they have a way to measure and ensure that self-driving cars actually are that safe.
A starting point for understanding self-driving car safety is defining an effective role for on-road data gathering. Contrary to common discussions about the purpose of vehicle testing, on-road testing should not be primarily used to find bugs in software. (Bugs — which really should be called defects — should be found in simulation and development.) Rather, the critical need for on-road testing is to understand the driving environment and collect data to make sure that simulation will include all the typical, unusual, and just plain weird driving situations that a full-scale self-driving car fleet will experience on real-world roadways. On-road vehicle testing can be used as a graduation exercise before deploying production vehicles to make sure nothing was missed in development, but it should not be the primary strategy for finding defects in the vehicle’s autonomy.
Ensuring adequate safety requires two approaches beyond collecting on-road data. The first approach is extensive simulation across many different levels. Closed course testing is a form of simulation in that a real car’s response is measured within an environment that has artificially created scenarios designed to stress key aspects of the system. Pure software simulation can be used in addition to closed course testing to scale up to hundreds or thousands of simulated cars running 24 hours a day in a cloud computing environment. These and other types of simulation can help improve testing coverage. But, experience with other types of critical systems shows that even extensive simulation is not enough to catch all the software defects that can cause avoidable deaths.
In addition to testing and simulation, rigorous design engineering approaches must be used to ensure that the software running self-driving cars is well designed and can deal safely with the inevitable glitches, malfunctions, faults, and surprises that happen in real world systems. Following ISO 26262 and other relevant safety standards is a necessary starting point. Additional methods such as system robustness testing and using a safety net architectural approach can help develop a sufficient level of assurance for the AI-centric vehicle functions. These technical approaches should be created in parallel with the control system side of self-driving car technology. While both design and validation of self-driving cars is a maturing area, it is likely that we have the means to ensure an appropriate level of safety. However, it is crucial to ensure that developers actually use these state-of-the-art techniques rather than skimping on safety validation in the race to deploy.
As self-driving car technology continues to be developed, there are fleets of test vehicles operating on public roads. Public on-road testing clearly has the potential to put pedestrians, other vehicles, and the general public at risk if the test vehicle misbehaves. While safety drivers are charged with ensuring vehicle safety, there are challenges to make this approach effective in practice. For example, continuous safety driver vigilance turns out to be difficult to maintain if the vehicle doesn’t fail very often. And it can be difficult for even a vigilant driver to know if a vehicle is about to misbehave without having some sort of indication of the vehicle’s near term plans. In general, it is important to ensure that a test vehicle (or any partially autonomous vehicle that requires human supervision) never paints the safety driver into a corner, leaving that driver in an untenable situation from which recovery within the constraints of reasonable human performance is difficult or impossible. When the driver does intervene, it’s important that the vehicle actually responds to the driver and gracefully relinquishes relevant vehicle controls.
In the longer term, it will be important that vehicle makers create a transparent argument as to why their AV is sufficiently safe. That argument must be reviewed by credible, independent safety experts. Well-chosen safety performance metrics can give some insight into how AV technology is progressing, but tend to come with many disclaimers and limitations. Simplistic metrics such as disengagement reports are no substitute for a thorough understanding of whether an AV has been engineered to an appropriate level of safety.
A lesson learned a long time ago in other safety areas is that independent audits of some type are an absolute requirement for designing safe systems. Transparency in safety arguments does not necessarily mean design information must be made public, nor that the government must perform reviews. But some credible independent party must assess whether an AV has been designed to an appropriate level of engineering rigor to be safe. The safety reports being published by some AV developers are an encouraging first step, but more details are required for independent assessment. A good starting point is to say exactly which appropriate international safety standards are being followed for what aspects of safety, and get external assessment according to those standards.
Incorporating AI technology into life critical computer-based systems such as AVs presents unique challenges. Whether safety is ensured within the scope of existing standards or is supported by arguments in addition to those standards, a transparent safety argument should be checked by an independent assessor to ensure that vehicles really are as safe as they need to be for use on public roads. That safety argument will need to include a lot more than just on-road test results.
About the author:
Dr. Philip Koopman has been working on autonomous vehicle safety for more than 20 years. As a professor at Carnegie Mellon University he teaches safe, secure, high quality embedded system engineering. As co-founder of Edge Case Research LLC he finds ways to improve AV safety, including robustness testing, architectural safety patterns, and structured safety argument approaches. He has a blog on self-driving car safety at: https://safeautonomy.blogspot.com
Highly Autonomous Vehicle Validation: it's more than just road testing!
- Why a billion miles of testing might not be enough to ensure self-driving car safety.
- Why it's important to distinguish testing for requirements validation vs. testing for implementation validation.
- Why machine learning is the hard part of mapping autonomy validation to ISO 26262
