Using a Driving Test as a Safety Metric (Metrics Episode 4)

The part of the driver test that is problematic for self-driving cars is deciding that the system has maturity of judgement. How do you check that it is old enough to drive?

At some point, companies are done testing and they need to make a decision about whether it’s okay to let their vehicles actually be completely self-driving, driverless cars. The important change here is that there is no longer a human responsible for continuously monitoring operational safety. That changes the complexion of safety, because now there’s no human driver to rely upon to take care of anything weird that might go wrong. That means you’ll need different metrics for safety when you deploy compared to those used during road testing.

One way people talk about knowing that it’s time to deploy is to give the car a road test, just like a human gets a driver’s test. Everyone wants something quick, cheap, and painless to decide whether their self-driving car is ready to go, and a driver test has some intuitive appeal. Indeed, if a car can’t pass a driver test, then truly that’s a problem. But is passing such a test enough? 

Let’s go down the list of things that happen in a driver test.

There’s a written test. Well, surely a self-driving car needs to know the rules of the road, just like a person. But for a self-driving car it’s a little more complicated, because it’s not just the rules of the road, but also what to do about conflicting rules or incomplete rules, or how you handle justifiable rule breaking.

For example, if there’s a big truck broken down in the travel lane on a two lane road, do you wait there until the truck is towed away, several hours perhaps? Or do you go around it if there’s no traffic? Well, going over the double yellow line is clearly breaking the usual rules, but in everyday driving people do that sort of thing all the time.

Another thing you need is a vision test. Surely self-driving cars need to be able to see things on the road. For a person it’s typically just whether or not they have the right glasses on. But for a self-driving car it’s more complicated, because it isn’t just seeing that something’s in the road, but also figuring out what’s on the road and what might happen next. It isn’t just about reading road signs.

Another thing, the classic thing people have in mind is a road skills test.  Surely a self driving car needs to be able to maneuver the vehicle in traffic and account for all the things that happen. But again, it’s more complicated to know that a self-driving car is ready beyond what you see on a typical road test. Sure, a typical road test covers things like parallel parking and using turn signals. But that’s the easy stuff. Did your driver test cover spin-outs? Did it cover handling a blown out tire with an actual blown out tire? Did you have to deal with loss of brakes? Did you have another car run a red light during the driver test to see how you’d respond?  (OK, well that last one actually did happen to me by chance on my own driver test. But I digress.)

Even if you were to address all those types of things, there’s another important piece of a human driver test that people don’t think of as a test. 

That’s actually proving you’re a human. You do that by showing your birth certificate. Oh, look: I’m a 16 year old human, and while I may not be the most mature person in the world, society’s determine I’m good enough to be able to handle a driver’s license. Now, what comes with that? It isn’t just that you’re 16. It’s that being 16, or whatever the age is where you are, is a proxy for things like being able to handle ambiguity and reason about consequences well enough. It’s a proxy for knowing that a situation is unpredictable and becoming more cautious, because you’re not sure what happens next. It's a proxy for understanding consequences and personal responsibility. Humans are moderately good at knowing when they’re not sure what’s going on, but things like machine learning are notoriously bad at knowing that they’re out of their element.

It’s also a proxy for handling unexpected events. While humans aren’t perfect, they do remarkably well when completely unexpected, unstructured events happen. They usually figure out something to do that’s reasonable. 

In general, being a 16 year old human is a proxy for some level of maturity and experience at handling the unexpected. What we do is we use a driver test to say: okay, this person has basic skills, and because they’re a reasonably mature human, they can probably handle all the weird stuff that happens and they’ll figure it out as they go. 

A big problem is it’s unclear how you figure out that a self-driving car has the level of judgment maturity of at least a 16 year old human. We’re not sure how to do that.

What we have known for decades is that you can’t prove a software based system is safe via testing alone. There are just too many complex situations and too many edge cases to be handled. No set of tests can cover everything. It’s just not possible. So, a driver test alone is never going to be enough to prove that a self-driving car is safe. Sure, elements of a driving test are useful. You absolutely want the self-driving car to know the rules of the road, to be able to look down the right away and see what’s there, and to be able to maneuver in traffic. But that is a minimum and insufficient set of things to prove it’s safe enough to responsibly deploy.

The point is you need more than a test. You need good engineering rigor. You need to know the car was designed properly, just like in every other safety critical computer based system. You don’t fly airplanes around to see if they fall out of this sky, and if they don’t, you say it’s safe. In fact, you do a lot of good engineering and you make sure the system is designed to do the right thing, and the testing is just there to make sure that the engineering process did what you thought it did.

So, for self-driving cars, sure, a road test is helpful, but you’re going to need a good engineering process was executed as well. You need to know that the system is going to handle all the things that have to go right, as well as all the things that can possibly go wrong, and that goes far beyond any reasonable driver test.

Toward a framework for Highly Automated Vehicle Safety Validation

I'm presenting a paper on AV safety validation at the 2018 SAE World Congress.  Here's the unofficial version of the presentation and a preprint of the paper.

Toward a Framework for Highly Automated Vehicle Safety Validation
Philip Koopman & Michael Wagner
2018 SAE World Congress / SAE 2018-01-1071

Abstract:
Validating the safety of Highly Automated Vehicles (HAVs) is a significant autonomy challenge. HAV safety validation strategies based solely on brute force on-road testing campaigns are unlikely to be viable. While simulations and exercising edge case scenarios can help reduce validation cost, those techniques alone are unlikely to provide a sufficient level of assurance for full-scale deployment without adopting a more nuanced view of validation data collection and safety analysis. Validation approaches can be improved by using higher fidelity testing to explicitly validate the assumptions and simplifications of lower fidelity testing rather than just obtaining sampled replication of lower fidelity results. Disentangling multiple testing goals can help by separating validation processes for requirements, environmental model sufficiency, autonomy correctness, autonomy robustness, and test scenario sufficiency. For autonomy approaches with implicit designs and requirements, such as machine learning training data sets, establishing observability points in the architecture can help ensure that vehicles pass the right tests for the right reason. These principles could improve both efficiency and effectiveness for demonstrating HAV safety as part of a phased validation plan that includes both a "driver test" and lifecycle monitoring as well as explicitly managing validation uncertainty.

Paper Preprint:        http://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf

Toward a Framework for Highly Automated Vehicle Safety Validation from Philip Koopman

A Driver Test For Self-Driving Cars Isn't Enough

I recently read yet another argument that a driving road test should be enough to certify an autonomous vehicle as safe for driving. In general, the idea was that if it's good enough to put a 16 year old on the road, it should be good enough for a self-driving vehicle.  I see this idea enough that it's worth explaining why it it's a really bad one.

Even if we were to assume that a self-driving vehicle is no different than a person (which is clearly NOT true), applying the driving test is only half the driver license formula. The other half is the part about being 16 years old. If a 12 year old is proficient at operating a vehicle, we still don't issue a drivers license. In addition to technical skills and book knowledge, we as a society have imposed a maturity requirement in most states of "being 16." It is typical that you don't get an unrestricted license until you're perhaps 18. And even then you're probably not a great driver at any age until you get some experience. But, we won't even let you on the road under adult supervision at 12!

The maturity requirement is essential.  As a driver we're expected to have the maturity to recognize when something isn't right, to avoid dangerous situations, to bring the vehicle to a safe state when something has gone wrong, to avoid operating when the vehicle system (vehicle + driver) is impaired, to improvise when something weird happens, and to compensate for other drivers who are having a bad day (or are simply suffering from only being 16). Autonomous driving systems might be able to do some of that, or even most of it in the near term. (I would argue that they are especially bad at self-detecting when they don't know what's going on.) But the point is a normal driving test doesn't come close to demonstrating "maturity" if we could even define that term in a rigorous, testable way. It's not supposed to -- that's why licenses require both testing and "being 16."

To be sure, human age is not a perfect correlation to maturity. But as a society we've come to the situation in which this system is working well enough that we're not changing it except for some tweaks every few years for very young and very old drivers who have historically higher mishap rates. But the big point is if a 12 year old demonstrates they are a whiz at vehicle operation and traffic rules, they still don't get a license.  In fact, they don't even get permission to operate on a public road with adult supervision (i.e., no learners permit at 12 in any US state that I know of.)  So why does it make sense to use a human driving test analogy to give a driver license, or even a learner permit, to an autonomous vehicle that was designed in the last few months?  Where's the maturity?

Autonomy advocates argue that encapsulating skill and fleet-wide learning from diverse situations could help cut down the per-driver learning curve. And it could reduce the frequency of poor choices such as impaired driving and distracted driving. If properly implemented, this could all work well and could improve driving safety -- especially for drivers who are prone to impaired and distracted driving. But while it's plausible to argue that autonomous vehicles won't make stupid choices about driving impaired, that is not at all the same thing as saying that they will be mature drivers who can handle unexpected situations and in general display good judgment comparable to a typical, non-impaired human driver. Much of safe driving is not about technical skill, but rather amounts to driving judgment maturity. In other words, saying that autonomous vehicles won't make stupid mistakes does not automatically make them better than human drivers.  

I'd want to at least see an argument of technological maturity as a gate before even getting to a driving skills test. In other words, I want an argument that the car is the equivalent of "being 16" before we even issue the learner permit, let alone the driver license. Suggesting that a driving test is all it takes to put a vehicle on the road means: building a mind-boggling complex software system with technology we're just at the point of understanding how to validate, doing an abbreviated acceptance test of basic skills (a few minutes on the road), and then deciding it's fine to put in charge of several thousand pounds of metal, glass, and a human cargo as it hurtles down the highway. (Not to mention the innocent bystanders it puts at risk.)

This is a bad idea for any software.  We know from history that a functional acceptance test doesn't prove something is safe (at most it can prove it is unsafe if a mishap occurs during the test).  Not crashing during a driver exam is to be sure an impressive technical achievement, but on its own it's not the same as being safe!  Simple acceptance testing as the gatekeeper for autonomous vehicles is an even worse idea. For other types of software we have found that in practice that you can't understand software quality for life-critical systems without also considering the rigor of the engineering process. Think of good engineering process as a proxy for "being 16 years old." It's the same for self-driving cars. 

(BTW, "life-critical" doesn't mean perfect. It means designed with sufficient engineering rigor to be suitable for its intended criticality. See ISO 26262 or your favorite software safety standard, which is currently NOT required by the government for autonomous vehicles, but should be.)

It should be noted that some think that a few hundred million miles of testing can be a substitute for documenting engineering rigor. That’s a different discussion. What this essay is about is saying that a road test — even an hour long grueling road test — does not fulfill the operational requirement of “being 16” for issuing a driving license under our current system. I’d prefer a different, more engineering-based method of certifying self-driving vehicles for use on public roads. A road test should certainly be part of that approach. But if you really want to base everything on a driver test, please tell me how you plan to argue the part about demonstrating the vehicle, the design team, or some aspect of the development project has the equivalent 16-year-old maturity. If you can’t, you’re just waving your hands about vehicle safety.

(Originally posted on Dec. 19, 2016, with minor edits)