Safety Validation and Edge Case Testing for Autonomous Vehicles (Slides)

Here is a slide deck that expands upon the idea that the heavy tail ceiling is a problem for AV validation. It also explains ways to augment image sensor inputs to improve robustness.

Safety Validation and Edge Case Testing for Autonomous Vehicles from Philip Koopman

(If slideshare is blocked for you, try this alternate download source)

Heavy Tail Ceiling Problem for AV Testing

I enjoyed participating in the AV Benchmarking Panel hosted by Clemson ICAR last week.  Here are my slides and a preprint of my position paper on the Heavy Tail Ceiling problem for AV safety testing.

Abstract
Creating safe autonomous vehicles will require not only extensive training and testing against realistic operational scenarios, but also dealing with uncertainty. The real world can present many rare but dangerous events, suggesting that these systems will need to be robust when encountering novel, unforeseen situations. Generalizing from observed road data to hypothesize various classes of unusual situations will help. However, a heavy tail distribution of surprises from the real world could make it impossible to use a simplistic drive/fail/fix development process to achieve acceptable safety. Autonomous vehicles will need to be robust in handling novelty, and will additionally need a way to detect that they are encountering a surprise so that they can remain safe in the face of uncertainty

Paper Preprint:
http://users.ece.cmu.edu/~koopman/pubs/koopman18_heavy_tail_ceiling.pdf

Presentation:

Autonomous Vehicle Benchmarking Panel from Philip Koopman

A Reality Check on the 94 Percent Human Error Statistic for Automated Cars

Automated cars are unlikely to get rid of all the "94% human error" mishaps that are often cited as a safety rationale. But there is certainly room for improvement compared to human drivers. Let's sort out the hype from the data.

You've heard that the reason we desperately need automated cars is that 94% of crashes are due to human error, right?  And that humans make poor choices such as driving impaired, right?  Surely, then, autonomous vehicles will give us a factor of 10 or more improvement simply by not driving stupid, right?

Not so fast. That's not actually what the research data says. In the words of NTSB Chair Jennifer Homendy: It Ain't 94%!

It's important for us to set realistic expectations for this promising new technology. Probably it's more like 50%.  Let's dig deeper.

(NEW:  video explainer here  8 minutes   https://youtu.be/pYb4X5aJhgU)

(https://bit.ly/2smqwYu /CC0)

The US Department of Transportation publishes an impressive amount of data on traffic safety -- which is a good thing. And, sure enough, you can find the 94% number in DOT HS 812 115 Traffic Safety Facts, Feb. 2015.  (https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812115) which says: "The critical reason was assigned to drivers in an estimated 2,046,000 crashes that comprise 94 percent of the NMVCCS crashes at the national level. However, in none of these cases was the assignment intended to blame the driver for causing the crash" (emphasis added). There’s the 94% number.  But wait – they’re not actually blaming the driver for those crashes! We need to dig deeper here.

Before digging, it's worth noting that this isn't really 2015 data, but rather a 2015 summary of data based on a data analysis report published in 2008.  (DOT HS 811 059 National Motor Vehicle Crash Causation Survey, July 2008 https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/811059)  If you want to draw your own conclusions you should look at the original study to make sure you understand things.

Now that we can look at the primary data, first we need to see what assigning a "crash reason" to the driver really means. Page 22 of the 2008 report sheds some light on this. Basically, if something goes wrong and the driver should have (arguably) been able to avoid the crash, the mishap “reason” is assigned to the driver. That's not at all the same thing as the accident being the driver's fault to due an overt error that directly triggered the mishap. To its credit that original report makes this clear. This means that the idea of crashes or fatalities being “94% due to driver error” differs from the report's findings in a subtle but critical way.

Indeed, many crashes are caused by drunk drivers. But other crashes are caused by something bad happening that the driver doesn't manage to recover from. Still other crashes are caused by the limits of human ability to safely operate a vehicle in difficult real-world situations despite the driver not having violated any rules. We need to dig still deeper to understand what's really going on with this report.

Page 25 of the report sheds some light on this. Of the 94% of mishaps attributed to drivers, there are a number of clear driver misbehaviors listed, including distracted driving, illegal maneuvers, and sleeping. But the #1 problem is "Inadequate surveillance 20.3%." In other words, a fifth of mishaps blamed on drivers are the driver not correctly identifying an obstacle, missing some road condition, or other problem of that nature. While automated cars might have better sensor coverage than a human driver's eyes, misclassifying an object or being fooled by an unusual scenario could happen with an automated car just as it can happen to a human.  (In other words, sometimes driver blame could be assigned to an automated driver, even if part of the 94%.) This biggest bin isn’t drunk driving at all, but rather gets to a core reason of why building automated cars is so hard. Highly accurate perception is difficult, whether you're a human or a machine.

Other driver bins in the analysis include "False assumption of other's action 4.5%," "Other/unknown recognition error 2.5%," Other/unknown decision error 6.2%," and "Other/unknown driver error 7.9%".  That’s another 21% that might or might not be impaired driving, and might be a mistake that could also be made by an automated driver.

So in the end, the 94% human attribution for mishaps isn't all impaired or misbehaving drivers. Rather, many of the reasons assigned to drivers sound more like imperfect drivers. It's increasingly clear that autonomous vehicles can also be imperfect. For example, they can misclassify objects on the road. So we can't blithely claim that automated cars won't have any of failures that the study attributes to human error. Rather, at least some of these problems will likely change from being assigned to "human driver error" to instead being "robot driver error."   Humans aren't perfect. Neither are robots. Robots might be better than humans in the end, but that area is still a work in progress and we do not yet have data to prove that it will turn out in the robot driver's favor any time soon.

A more precise statement of the study's findings is that while it is indeed true that 94% of mishaps might be attributed to humans, significantly less than that number is attributable to poor human choices such as driving drunk. While I certainly appreciate that computers don't drive drunk, they just might be driving buggy. And even if they don’t drive buggy, making self driving cars just as good as an unimpaired driver is unlikely to get near the 94% number so often tossed around. Perception, modeling expected behavior of other actors, and dealing with unexpected situations are all notoriously difficult to get right, and are all cited sources of mishaps.  So this should really be no surprise. It is possible automated cars might be a lot better than people eventually, but this data doesn't support that expectation at the 94% better level.

We can get a bit more clarity by looking at another DOT report that might help set more realistic expectations. Let's take a look at 2016 Fatal Motor Vehicle Crashes: Overview (DOT HS 812 456; access via https://www.nhtsa.gov/press-releases/usdot-releases-2016-fatal-traffic-crash-data). The most relevant numbers are below (note that there is overlap, so the categories add up to more than 100%):

• Total US roadway fatalities: 37,461
• Alcohol-impaired-driving fatalities: 10,497
• Unrestrained passenger vehicle fatalities (not wearing seat belts): 10,428
• Speeding-related fatalities: 10,111
• Distraction-affected fatalities: 3,450
• Drowsy driving fatalities: 803
• Non-occupant fatalities (sum of pedestrians, cyclists, other): 7,079

I'm not going to attempt a detailed analysis, but certainly we can get the broad brush strokes from this data.  I come up with the following three conclusions:

1. Clearly impaired driving and speeding contribute to a large number of fatalities (ballpark twenty thousand, although there are overlaps in the categories).  So there is plenty of low hanging fruit to go after if we can create a automated vehicle that is as good as an unimpaired human. But it might be more like a 2x or 3x improvement than a 20x improvement. Consider the typical 100 million mile between fatality number that is quoted for human drivers. If you remove the impaired drivers, based on this data you get more like about 200 million miles between fatalities. It will take a lot to get automated cars that good. Don't get me wrong; a 2x or so improvement is potentially a lot of lives saved. But it's not zero fatalities, and it's nowhere near a 94% reduction.

2. Almost one-fifth of the fatalities are pedestrians, cyclists, and other at-risk road users. Detecting and avoiding that type of crash is notoriously difficult for autonomous vehicle technology. Occupants have all sorts of crash survivability features. Pedestrians -- not so much. Ensuring non-occupant safety has to be a high priority if we're going to deploy this technology and avoid an unintended consequence of a rise in pedestrian fatalities potentially offsetting gains in occupant safety.

3. Well over a quarter of the vehicle occupant fatalities are attributed to not wearing seat belts. Older generations have lived through the various technological attempts to enforce seat belt use. (Remember motorized seat belts?)  The main takeaway has been that some people will go to extreme lengths to avoid wearing seat belts. It's difficult to see how automated car technology alone will change that. (Yes, if there are zero crashes the need for seat belts is reduced, but we're not going to get there any time soon. It seems more likely that seat belts will play a key part in reducing fatalities when coupled with hopefully less violent and less frequent crashes.)

So where does that leave us?

There is plenty of potential for automated cars to help with safety. But the low hanging fruit is more likely cutting fatalities perhaps in half if we can achieve parity with an average well behaved human driver. The 94% number so often quoted will take a lot more than that. Over time, hopefully, automated cars can continue to improve further. ADAS features such as automatic emergency braking can likely help too. And for now, even getting rid of half the traffic deaths is well worth doing, especially if we make sure to consider pedestrian safety.

Driving safely is a complex task for man or machine. It would be a shame if a hype roller coaster ride triggers disillusionment with technology that can in the end improve safety. Let's set reasonable expectations so that automated car technology is asked to provide near term benefits that it can actually deliver.

---

Update: Mitch Turk pointed out another study from 2016 that is interesting.
http://www.pnas.org/content/pnas/113/10/2636.full.pdf

This has data from monitoring drivers who experienced crashes. The ground rules for the experiment were a little different, but it has data explaining what was going on in the vehicle before a crash.  A significant finding is that driver distraction is an issue. (Note that this data is several years later than the previous study, so that makes some sense.)

For our purposes an interesting finding is that 12.3% of crashes were NOT distracted/NOT Impaired/NOT Human Error:

Beyond that, it seems likely that some of the other categories contain scenarios that could be difficult for an AV, such as undistracted, unimpaired errors (16.5%).

---

Update 7/18/2018:  Laura Fraade-Blanar from RAND send me a nice paper from 2014 on exactly this topic:
https://web.archive.org/web/20181121213923/https://www.casact.org/pubs/forum/14fforum/CAS%20AVTF_Restated_NMVCCS.pdf

The study looked at the NMVCSS data from 2008 and asked the question of what this data means for autonomy, accounting for issues that autonomy is going to have trouble addressing. They say that "49% of accidents contain at least one limiting factor that could disable the [autonomy] technology or reduce its effectiveness."  They also point out that autonomy can create new risks not present in manually driven vehicles.

So this data suggests that eliminating perhaps half of autonomous vehicle crashes is a more realistic goal.

---

NOTES:

For an update on how many organizations are misusing this statistic, see:
  https://usa.streetsblog.org/2020/10/14/the-94-solution-we-need-to-understand-the-causes-of-crashes/

The studies referenced, as with other similar studies, don’t attempt to identify mishaps that might have been caused by computer-based system malfunction. In studies like this, potential computer-based faults and non-reproducible faults in general are commonly attributed to driver error unless there is compelling evidence to the contrary. So the human error numbers must be taken with a grain of salt. However, this doesn’t change the general nature of the argument being made here.

The data supporting the conclusion is more than 10 years old. So it would be no surprise if ADAS technology has been changing things. In any event, looking into current safety expectations should arguably require separating the effects of ADAS systems such as automatic emergency braking (AEB) from functions such as lane keeping and speed control.  ADAS can potentially improve safety even for well behaved human drivers.

Some readers will want to argue for a more aggressive safety target than 2x to 3x safer than average humans. I'm not saying that 2x or 3x is an acceptable safety target -- that's a whole different discussion. What I'm saying is that is a much more likely near term success target than the 94% number tossed around.

There will no doubt be different takes on how to interpret data from these and other reports.  That’s a vitally important discussion to have. But the point of this essay is not to argue how safe is safe enough.  Rather, the point is to have a discussion about realistic expectations based on data and engineering, not hype. So if you want to argue a different outcome than I propose, that's great. But please bring data instead of marketing claims to the discussion. Thanks.

Can Mobileye Validate ‘True Redundancy’?

I'm quoted in this article by Junko Yoshida on Mobileye's approach to AV safety.

Can Mobileye Validate ‘True Redundancy’?
Intel/Mobileye’s robocars start running in Jerusalem
Junko Yoshida
5/22/2018 02:01 PM EDT

...
Issues include how to achieve “true redundancy” in perception, how to explain objectively what “safe” really means, and how to formulate “a consistent and complete set of safety rules” agreeable to the whole AV industry, according to Phil Koopman, professor of Carnegie Mellon University.
...

Read the story here:
  https://www.eetimes.com/document.asp?doc_id=1333308

Did Uber Do Enough to Make Test AVs Safe?

I'm quoted in this article by Junko Yoshida regarding on-road AV testing safety:

Did Uber Do Enough to Make Test AVs Safe?
Junko Yoshida, Chief International Correspondent
5/26/2018 05:01 PM EDT

"The NTSB preliminary report exposes two issues. One is the immaturity of Uber's AV software stack. Another is the absence of an Uber safety strategy in creating its AV testing platform."

Read more:
  https://www.eetimes.com/author.asp?section_id=36&doc_id=1333325

AutoSens 2018 slides

I enjoyed presenting at AutoSens 2018 today.   The audience was very engaged and asked great questions.

Here are my slides. (If you seen my other recent slide decks probably not a lot of surprises, but I remixed things to emphasize perception validation.)

Approaches to Validating Autonomous Vehicle Safety from Philip Koopman

Slides from US-China Transportation Forum Presentation

On Thursday I had the honor to presenting to two Secretaries of Transportation at the 2018 U.S.-China Transportation Forum in Beijing China.  (US Secretary Chao and China Secretary Yang were in the front row -- as well as a huge room full of US and China delegates.)  It was a really interesting experience, and I truly appreciate the support and hospitality shown by the organizers and both governments.  It's not often that a hard-core techie gets face time with cabinet members!

I was the US Autonomous Vehicle technical expert in one of two technology sessions.  My topic was safe autonomous vehicle testing safety.   I gave the short version of my PA AV Summit talk.  The slides are here for anyone who is interested in seeing how I tried to boil that message down to a 10 minute slot (with simultaneous translation to Chinese).

US-China Transportation Forum Slides April 2018 from Philip Koopman

Toward a framework for Highly Automated Vehicle Safety Validation

I'm presenting a paper on AV safety validation at the 2018 SAE World Congress.  Here's the unofficial version of the presentation and a preprint of the paper.

Toward a Framework for Highly Automated Vehicle Safety Validation
Philip Koopman & Michael Wagner
2018 SAE World Congress / SAE 2018-01-1071

Abstract:
Validating the safety of Highly Automated Vehicles (HAVs) is a significant autonomy challenge. HAV safety validation strategies based solely on brute force on-road testing campaigns are unlikely to be viable. While simulations and exercising edge case scenarios can help reduce validation cost, those techniques alone are unlikely to provide a sufficient level of assurance for full-scale deployment without adopting a more nuanced view of validation data collection and safety analysis. Validation approaches can be improved by using higher fidelity testing to explicitly validate the assumptions and simplifications of lower fidelity testing rather than just obtaining sampled replication of lower fidelity results. Disentangling multiple testing goals can help by separating validation processes for requirements, environmental model sufficiency, autonomy correctness, autonomy robustness, and test scenario sufficiency. For autonomy approaches with implicit designs and requirements, such as machine learning training data sets, establishing observability points in the architecture can help ensure that vehicles pass the right tests for the right reason. These principles could improve both efficiency and effectiveness for demonstrating HAV safety as part of a phased validation plan that includes both a "driver test" and lifecycle monitoring as well as explicitly managing validation uncertainty.

Paper Preprint:        http://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf

Toward a Framework for Highly Automated Vehicle Safety Validation from Philip Koopman

Ensuring The Safety of On-Road Self-Driving Car Testing (PA AV Summit Talk Slides)

This is the slide version of my op-ed on how to make self-driving car testing safe.

The take-away is create a test vehicle with a written safety case that addresses these topics:

• Show that the safety driver is paying adequate attention
• Show that the safety driver has time to react if needed
• Show that AV disengagement/safing actually works when things go wrong

(An abbreviated version was also presented in April 2018 at the US-China Transportation Forum in Beijing China.)

Ensuring The Safety of On-Road Self-Driving Car Testing (PA AV Summit Talk Slides) from Philip Koopman

What can we learn from the UK guidelines on self-driving car testing?

The UK already has a pretty good answer for how to do self-driving car testing safely. US stakeholders could learn something from it.

You can see the document for yourself at: 

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/446316/pathway-driverless-cars.pdf

As industry and various governmental organizations decide what to do in response to the tragic Tempe Arizona pedestrian accident, it's worth looking abroad to see what others have done.  As it turns out, the UK Department for Transport issued a 14 page document in July 2015: "The Pathway to Driverless Cars: A Code of Practice for testing."   It covers test drivers, vehicle equipment, licensing, insurance, data recording, and more. So far so good, and kudos for specifically addressing the topic of test platform safety that long ago!

As I'd expect from a UK safety document, there is a lot to like.  I'm not going to try to summarize it all, but here are some comments on specific sections that are worth noting. Overall, I think that the content is useful and will be helpful to improve safety for testing Autonomous Vehicle (AV) technology on public roads.  My only criticism is that it doesn't go quite far enough in a couple places.

First, it is light on making sure that the safety process is actually performing as intended. For example, they say it's important to make sure that test drivers are not fatigued, which is good. But they don't explicitly say that you need to take operational data to make sure that the procedures intended to mitigate fatigue problems are actually resulting in alert drivers. Similarly, they say that the test drivers need time to react, but they don't require feedback to make sure that on-road vehicles are actually leaving the drivers enough time to react during operation.  (Note that this is a tricky bit to be sure you get right because distracted drivers take longer to react, so you really need to ensure that field operations are leaving sufficient reaction time margin.)

In fairness, they say "robust procedures" and for a safety person taking data to make sure the procedures are actually working should be obvious. Nonetheless I've found in practice that it's important to spell out the need for feedback to correct safety issues. If you have a high stakes environment such as the autonomy race to market, it's only natural that the testers will be under pressure to cut corners. The only way I know of to ensure that "aggressive" technology maturation doesn't cross the line to being unsafe is to have continual feedback from field operations to ensure that the assumptions and strategy underlying the safety plan are actually effective and working as intended. For example, you should detect and correct systemic problems with safety driver alertness long before you experience a pedestrian fatality.

Second, although they say it's important for the takeover mechanism to work, they don't specifically require designing it according to a suitable functional safety standard.  Again, for a safety person this should be obvious, and quite possibly it was so obvious to the authors of this document that they didn't bother mentioning it.  But again it's worth spelling out.

To be clear, it's important that any on-road testing of AV technology should be no more dangerous than the normal operation of a human-driven non-autonomous vehicle. That's the whole purpose of having a safety driver!  But getting safety drivers to be that good in practice can be a challenge. However, rather than succumb to pessimism about whether testing can actually be safe, I say let the AV developers prove that they can handle this challenge with a transparent, public safety argument.  (See also my previous posting on safe AV testing for a high level take on things.)

The UK testing safety document is well worth considering by any governmental agency or AV company contemplating how on-road testing of AV technology should be done.

---

Below are some more detailed notes. The bullets are from the source document, with some informal comments after each bullet:

• 1.3: ensure that on-road testing "is carried out with the minimum practicable risk"

This appears to be invoking the UK legal concept of ALARP ("As Low As Reasonably Practicable") and SFAIRP ("So Far As Is Reasonably Practicable.")  This is a technical concept, not an intuitive concept.  You can't simply say "this ought to be OK because I think it's OK." Rather, that you need to demonstrate via a rigorous engineering process that you've done everything reasonably practicable to reduce risk. 

• 3.4 Testing organisations should:
• ... Conduct risk analysis of any proposed tests and have appropriate risk management strategies.
• Be conscious of the effect of the use of such test vehicles on other road users and plan trials to manage the risk of adverse impacts. 

It's not OK to start driving around without having done some work to understand and mitigate risks.

• 4.16 Testing organisations should develop robust procedures to ensure that test drivers and operators are sufficiently alert to perform their role and do not suffer fatigue. This could include setting limits for the amount of time that test drivers or operators perform such a role per day and the maximum duration of any one test period. 

The test drivers have to stay alert.  Simply setting the limits isn't enough. You have to actually make sure the limits are followed, that there isn't undue pressure for drivers to skip breaks, and in the end you have to make sure that drivers are actually alert.  Solving alertness issues by firing sleepy drivers doesn't fix any systemic problem with alertness -- it just gives you fresh drivers who will have just as much trouble staying alert as the drivers you just fired.

• 4.20 Test drivers and operators should be conscious of their appearance to other road users, for example continuing to maintain gaze directions appropriate for normal driving. 

This appears to address the problem of other road users interacting with an AV. The theory seems to be that if for example the test driver makes eye contact with a pedestrian at a crosswalk, that means that even if the vehicle makes a mistake the test driver will intervene to give the pedestrian right of way. This seems like a sensible requirement, and could help the safety driver remain engaged with the driving task.

• 5.3 Organisations wishing to test automated vehicles on public roads or in other public places will need to ensure that the vehicles have successfully completed in-house testing on closed roads or test tracks.
• 5.4 Organisations should determine, as part of their risk management procedures, when sufficient in-house testing has been completed to have confidence that public road testing can proceed without creating additional risk to road users. Testing organisations should maintain an audit trail of such evidence.

You should not be doing initial development on public roads. You should be using extensive analysis and simulation to be pretty sure everything is going to work before you ever get near a public road.  On-road testing should be to check that things are OK and there are no surprises. (Moreover, surprises should be fed back to development to avoid similar surprises in the future.)  You should have written records that you're doing the right amount of validation before you ever operate on public roads. (emphasis added)

• 5.5 Vehicle sensor and control systems should be sufficiently developed to be capable of appropriately responding to all types of road user which may typically be encountered during the test in question. This includes more vulnerable road users for example disabled people, those with visual or hearing impairments, pedestrians, cyclists, motorcyclists, children and horse-riders. 

Part of your development should include making sure the system can deal with at-risk road users.  This means there should be a minimal chance that a pedestrian or other at-risk road user will be put into danger by the AV even without safety driver intervention.  (The safety driver should be handling unexpected surprises, and not be relied upon as a primary control mechanism during road testing.)

• 5.8 This data should be able to be used to determine who or what was controlling the vehicle at the time of an incident. The data should be securely stored and should be provided to the relevant authorities upon request. It is expected that testing organisations will cooperate fully with the relevant authorities in the event of an investigation

With regard to data recording, there should be no debate over whether the autonomy was in control a the time of the mishap.  (How can it possibly be that a developer says "we're not sure if the autonomy was in control at the time of the mishap?" Yet I've heard this on the news more than once.)  It's also important to be transparent about the role of autonomy at times just before any mishap.  For example, if autonomy disengages a fraction of a second before impact, it's unreasonable to just blame the human driver without a more thorough investigation.

• 5.18 Ensuring that the transition periods between manual and automated mode involve minimal risk will be an important part of the vehicle development process and one which would be expected to be developed and proven during private track testing prior to testing on public roads or other public places. 

It's really important that manual takeover by a safety driver actually works. As mentioned above, the takeover system should be designed to a suitable level of safety (e.g., according to ISO 26262).

• 5.21 ... All software and revisions have been subjected to extensive and well documented testing. This should typically start with bench testing and simulation, before moving to testing on a closed test track or private road. Only then should tests be conducted on public roads or other public places. 

Again, testing should be used to confirm that the design is right, not as an iterative drive-fix-drive approach to gradually beating the system into proper operation via brute force road testing.

These comments are based on a preliminary reading of the document. I might change my thoughts on this document over time.

A Way Forward for Safe Self-Driving Car Road Testing

The self-driving car industry's reputation has suffered a setback, but the resulting debate about whether autonomy is safe for road use is focused on solving the wrong problem.

Dashcam image from mishap released by Tempe AZ Police

The recent tragic fatality in Arizona in which a pedestrian was hit and killed by a self-driving car test vehicle has sent a shock wave through the industry. At least some companies have shut down their road testing while they see how things develop. Uber just had its Arizona road testing permission revoked, and apparently is not going to be testing in California any time soon. The longer term effects on public enthusiasm for the technology overall remain unclear.

Proponents of self-driving car testing point out that human drivers are far from perfect, and compare the one life lost to the many lives lost every day in conventional vehicle crashes. Opponents say that potentially avoidable fatalities due to immature technology are unacceptable. The debate about whether self-driving car technology can be trusted with human lives is important in the long term, but isn't actually what went wrong here.


What went wrong is that a test vehicle killed Elaine Herzberg -- not a fully autonomous vehicle. The vehicle involved wasn't supposed to have perfect autonomy technology at all. Rather, it had unproven systems still under development, and a safety driver who was supposed to ensure that failures in the technology did no harm.  Whether the autonomy failed to detect a mid-block pedestrian crossing at night isn't the real issue. The real issue is why the safety driver didn't avoid the mishap despite any potential technology flaw.


The expedient approach of blaming the safety driver (or the victim) won't make test vehicles safer. We've known for decades that putting a single human in charge of supervising a self-driving car with no way to ensure attentiveness is asking for trouble.  And we know that pedestrians don't always obey traffic rules. So to really be sure these vehicles are safe on public roads, we need to dig deeper.


Fortunately, there is a way forward that doesn't require waiting for the next death to point out any potential problems in some other company's prototype technology, and doesn't require developers to prove that their autonomy is perfect before testing. That way simply requires treating these vehicles as the test vehicles they are, and not as fully formed self-driving cars. Operators should not be required to prove their autonomy technology is perfect. But they should be required to explain why their on-road testing approach is adequately safe beyond simply saying that a safety driver is sitting in the vehicle.

A safety explanation for a self-driving test platform doesn't have to expose anything about proprietary autonomy technology, and doesn't have to be incredibly complicated. It might be as simple as a three-part safety strategy: proactively ensure the safety driver always pays attention, ensure that the safety driver has time to react if a problem occurs, and ensure that when the safety driver does react, the vehicle will obey the safety driver's commands. Note that whether the autonomy fails doesn't enter into it, other than ensuring that the autonomy gets out of the way when the safety driver needs to take over.

There is no doubt that making a credible safety explanation for a self-driving car test platform requires some effort. The fact that safety drivers can lose focus after only a few minutes must be addressed somehow. That could be as simple as using two safety drivers, as many companies do, if it can be shown that paired drivers actually achieve continuous attentiveness for a full test session. Possibly some additional monitoring technology is needed to track driver eye gaze, driver fatigue, or the like. It must also be clear that the safety driver has a way to know if the autonomy is making a mistake such as failing to detect unexpected pedestrians. For example, a heads-up display that outlines pedestrians as seen through the windshield could make it clear when one has been missed to the eyes-on-the-road safety driver. And the safety explanation has to deal with realities such as pedestrians who are not on crosswalks. Doing all this might be a challenge, but the test system doesn't have to be perfect. Arguably, the test vehicle with its safety driver(s) just has to be as good as single human driver in an ordinary vehicle -- which the self-driving car proponents remind us is far from perfect.

I live in Pittsburgh where I have seen plenty of self-driving car test vehicles, including encounters as a pedestrian. Until now I've just had to take it on faith that the test vehicles were safe.  Self-driving cars have tremendous long-term promise. But for now, I'd rather have some assurance, preferably from independent safety reviewers, that these companies are actually getting their test platform safety right.

Philip Koopman, Carnegie Mellon University.

Author info: Prof. Koopman has been helping government, commercial, and academic self-driving developers improve safety for 20 years.
Contact:  koopman@cmu.edu

This essay was originally published in EE Times:
https://www.eetimes.com/author.asp?section_id=36&doc_id=1333143

Interview on Smarter Cars podcast

I really enjoyed the discussion with  Michele Kyrouz  on the Smarter Cars podcast. Thanks so much for inviting me on as a guest!




https://anchor.fm/smarter-cars   // March 15, 2018 episode

A More Comprehensive Look at Autonomous Vehicle Testing and Validation

Developers should create a transparent safety argument based on testing, simulation, and good engineering practices.

The race to deploy self-driving cars is in full swing, with fleets being deployed on public roads in increasing numbers. While some vehicles require driver supervision, other fleets are being deployed which don’t even have driver controls at all. The big question is: will these vehicles be safe enough?

Carnegie Mellon NavLab 11 / 2001 / https://goo.gl/itksbK

Now that self-driving car technology is maturing, there is increasing recognition of the fact that demonstrating adequate levels of safety with on-road testing is impractical. Too many billions of miles are needed for a credible statistical safety argument, and it simply costs too much and takes too long to do that. Other safety critical application areas already have a solution to this problem in the form of internationally accepted safety standards (e.g., aircraft, trains, and even home appliances). While there is an international safety standard for conventional cars (ISO-26262), its use is not currently required by the US government.

The stakes are high. Even a single bad line of software code can — and has — caused catastrophic failures. While nobody should expect this technology to be perfect, it’s important to ensure that it follows accepted practices and is appropriately safe. Assuming we have a quantification of how much better than a human driver the technology needs to be, the technologists need to make sure that they have a way to measure and ensure that self-driving cars actually are that safe.

A starting point for understanding self-driving car safety is defining an effective role for on-road data gathering. Contrary to common discussions about the purpose of vehicle testing, on-road testing should not be primarily used to find bugs in software. (Bugs — which really should be called defects — should be found in simulation and development.) Rather, the critical need for on-road testing is to understand the driving environment and collect data to make sure that simulation will include all the typical, unusual, and just plain weird driving situations that a full-scale self-driving car fleet will experience on real-world roadways. On-road vehicle testing can be used as a graduation exercise before deploying production vehicles to make sure nothing was missed in development, but it should not be the primary strategy for finding defects in the vehicle’s autonomy.

Ensuring adequate safety requires two approaches beyond collecting on-road data. The first approach is extensive simulation across many different levels. Closed course testing is a form of simulation in that a real car’s response is measured within an environment that has artificially created scenarios designed to stress key aspects of the system. Pure software simulation can be used in addition to closed course testing to scale up to hundreds or thousands of simulated cars running 24 hours a day in a cloud computing environment. These and other types of simulation can help improve testing coverage. But, experience with other types of critical systems shows that even extensive simulation is not enough to catch all the software defects that can cause avoidable deaths.

In addition to testing and simulation, rigorous design engineering approaches must be used to ensure that the software running self-driving cars is well designed and can deal safely with the inevitable glitches, malfunctions, faults, and surprises that happen in real world systems. Following ISO 26262 and other relevant safety standards is a necessary starting point. Additional methods such as system robustness testing and using a safety net architectural approach can help develop a sufficient level of assurance for the AI-centric vehicle functions. These technical approaches should be created in parallel with the control system side of self-driving car technology. While both design and validation of self-driving cars is a maturing area, it is likely that we have the means to ensure an appropriate level of safety. However, it is crucial to ensure that developers actually use these state-of-the-art techniques rather than skimping on safety validation in the race to deploy.

As self-driving car technology continues to be developed, there are fleets of test vehicles operating on public roads. Public on-road testing clearly has the potential to put pedestrians, other vehicles, and the general public at risk if the test vehicle misbehaves. While safety drivers are charged with ensuring vehicle safety, there are challenges to make this approach effective in practice. For example, continuous safety driver vigilance turns out to be difficult to maintain if the vehicle doesn’t fail very often. And it can be difficult for even a vigilant driver to know if a vehicle is about to misbehave without having some sort of indication of the vehicle’s near term plans. In general, it is important to ensure that a test vehicle (or any partially autonomous vehicle that requires human supervision) never paints the safety driver into a corner, leaving that driver in an untenable situation from which recovery within the constraints of reasonable human performance is difficult or impossible. When the driver does intervene, it’s important that the vehicle actually responds to the driver and gracefully relinquishes relevant vehicle controls.

In the longer term, it will be important that vehicle makers create a transparent argument as to why their AV is sufficiently safe. That argument must be reviewed by credible, independent safety experts. Well-chosen safety performance metrics can give some insight into how AV technology is progressing, but tend to come with many disclaimers and limitations. Simplistic metrics such as disengagement reports are no substitute for a thorough understanding of whether an AV has been engineered to an appropriate level of safety.

A lesson learned a long time ago in other safety areas is that independent audits of some type are an absolute requirement for designing safe systems. Transparency in safety arguments does not necessarily mean design information must be made public, nor that the government must perform reviews. But some credible independent party must assess whether an AV has been designed to an appropriate level of engineering rigor to be safe. The safety reports being published by some AV developers are an encouraging first step, but more details are required for independent assessment. A good starting point is to say exactly which appropriate international safety standards are being followed for what aspects of safety, and get external assessment according to those standards.

Incorporating AI technology into life critical computer-based systems such as AVs presents unique challenges. Whether safety is ensured within the scope of existing standards or is supported by arguments in addition to those standards, a transparent safety argument should be checked by an independent assessor to ensure that vehicles really are as safe as they need to be for use on public roads. That safety argument will need to include a lot more than just on-road test results.

About the author:
Dr. Philip Koopman has been working on autonomous vehicle safety for more than 20 years. As a professor at Carnegie Mellon University he teaches safe, secure, high quality embedded system engineering. As co-founder of Edge Case Research LLC he finds ways to improve AV safety, including robustness testing, architectural safety patterns, and structured safety argument approaches. He has a blog on self-driving car safety at: https://safeautonomy.blogspot.com