Safe Autonomy

I recently read yet another argument that a driving road test should be enough to certify an autonomous vehicle as safe for driving. In general, the idea was that if it's good enough to put a 16 year old on the road, it should be good enough for a self-driving vehicle.  I see this idea enough that it's worth explaining why it it's a really bad one. Even if we were to assume that a self-driving vehicle is no different than a person (which is clearly NOT true), applying the driving test is only half the driver license formula. The other half is the part about being 16 years old . If a 12 year old is proficient at operating a vehicle, we still don't issue a drivers license. In addition to technical skills and book knowledge, we as a society have imposed a maturity requirement in most states of "being 16." It is typical that you don't get an unrestricted license until you're perhaps 18. And even then you're probably not a great driver at any a

Welcome! This blog primarily covers Autonomous Vehicle safety, often known as self-driving car safety. I'm splitting this blog off from my Better Embedded Software blog to reflect my increased emphasis on Autonomous Vehicle (AV) safety. In my professor gig, these days my main research is on AV stress testing (the ASTAA and RIOT projects at CMU NREC ). I'm also very active in the startup company that I co-founded with Mike Wagner: Edge Case Research LLC . Over the past couple years we have emphasized both stress testing and creating safety cases for AVs. Comments are moderated. I read them all.  Comments that ask a question typically get approved, although I can't offer specific advise on your particular system this way.  Additional thoughts and responsible contrary views are also typically approved. While I appreciate the "nice blog" type posts, I typically don't approve them for posting.  (So many of them are comment spam!)  If you really want t

Here are the slides from my TechAD talk on self driving car safety: Highly Autonomous Vehicle Validation from Philip Koopman Highly Autonomous Vehicle Validation: it's more than just road testing! - Why a billion miles of testing might not be enough to ensure self-driving car safety. - Why it's important to distinguish testing for requirements validation vs. testing for implementation validation. - Why machine learning is the hard part of mapping autonomy validation to ISO 26262 ( Originally posted on Nov 11, 2017 )

Challenges in Autonomous Vehicle Testing and Validation   from  Philip Koopman Challenges in Autonomous Vehicle Validation Keynote Presentation Abstract Philip Koopman Carnegie Mellon University; Edge Case Research LLC ECE Dept. HH A-308, 5000 Forbes Ave., Pittsburgh, PA, USA koopman@cmu.edu Developers of autonomous systems face distinct challenges in conforming to established methods of validating safety. It is well known that testing alone is insufficient to assure safety, because testing long enough to establish ultra-dependability is generally impractical. That’s why software safety standards emphasize high quality development processes. Testing then validates process execution rather than directly validating dependability. Two significant challenges arise in applying traditional safety processes to autonomous vehicles. First, simply gathering a complete set of system requirements is difficult because of the sheer number of combinations of possible scenarios and

Autonomous Vehicle Safety: An Interdisciplinary Challenge By Phil Koopman & Mike Wagner Abstract: Ensuring the safety of fully autonomous vehicles requires a multi-disciplinary approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory approaches. Significant open technical challenges include validating inductive learning in the face of novel environmental inputs and achieving the very high levels of dependability required for full-scale fleet deployment. However, the biggest challenge may be in creating an end-to-end design and deployment process that integrates the safety concerns of a myriad of technical specialties into a unified approach. Read the preprint version here for free ( link / .pdf ) Official IEEE version (subscription required): http://iee

[While DoT has since published revised draft policies, the original draft policy and my response are still relevant to provide an overall picture about critical needs for self-driving vehicle safety.] I've prepared a draft response to DoT/NHTSA on their proposed policy for highly automated vehicle safety. EE Times article that summarizes my response The topics I cover are: 1. Requiring a safety argument that deals with the challenges of validating machine learning 2. Requiring transparent independence in safety assessment 3. Triggering safety reassessment based on safety integrity, rather than “significant” functionality 4. Requiring assessment of changes that can compromise the triggering of fall-back strategies 5. Characterizing what “reasonable” might mean regarding anticipation of exceptional scenarios 6. Assuring the integrity of data that is likely to be used for crash investigations 7. Diagnostics that encompass non-collision failures of components and en