Showing posts from May, 2022

Cruise robotaxi struggles with real-world emergency vehicle situation

A Cruise robotaxi failed to yield effectively to a fire truck, delaying it. Sub-headline: Garbage truck driver saves the day when Cruise autonomous vehicle proves itself to not be autonomous. This referenced article explains the incident in detail, which involves a garbage truck blocking one lane and the Cruise vehicle pulling over into a position that did not leave enough room for the fire truck to pass. But it also argues that things like this should be excused because it is in the cause of developing life saving technology. I have to disagree. Real harm done now to real people should not be balanced against theoretical harm potentially saved in the future. Especially when there is no reason (other than business incentives) to be doing the harm today, and the deployment continues once it is obvious that near-term harm is likely. I would say that if the car can't drive in the city like a human driver, it should have a human driver to take over when the car can't. Whatever rem

Tesla emergency door releases -- what a mess!

The Tesla manual door releases -- and lack thereof in some cases -- present unreasonable risk. What in the world were they thinking? Really bad human interface design. Cool design shouldn't come at expense of life critical peril. This article this week sums up the latest, but this has been going on for a long time. Tesla fans seem to be saying that it is the driver's responsibility to know where the manual release latch is to escape in case of fire. Anyone who doesn't is (and has in past fires) been ridiculed on-line for not knowing where the manual release is hidden. Even if they died due to not successfully operating the control, or having to kick the window out, somehow they are the idiots and it is their fault, not Tesla's. (If someone you love has died or been injured in this way you have my sympathy, and it is the trolls who are idiots, not your loved one.) On-line articles saying "here's how to operate the door release so you don't die in a Tesla fi

A gentle introduction to autonomous vehicle safety cases

I recently ran into this readable article about AV safety cases by Thomas & Vandenberg from 2019. While things have changed a bit, it still is a reasonable introduction for anyone asking "what exactly would an AV safety case look like." A real industry-strength safety case is going to be complicated in many ways. In particular, there are many different approaches for breaking down G1 which will significantly affect things. On the other hand all the pieces will need to be there somewhere, so choosing this high level breakdown is more of an architectural choice (for the safety case, not necessarily the system). We do not yet have a consensus on an optimal strategy for building such safety cases, but this is not a bad starting place from safety folks who were previously at Uber ATG. Thomas & Vandenberg, Harnessing Uncertainty in Autonomous Vehicle Safety, Journal of System Safety, Vol. 55, No. 2 (2019) (Uber ATG also published a  muc

SEAMS Keynote talk: Safety Performance Indicators and Continuous Improvement Feedback

Abstract: Successful autonomous ground vehicles will require a continuous improvement strategy after deployment. Feedback from road testing and deployed operation will be required to ensure enduring safety in the face of newly discovered rare events. Additionally, the operational environment will change over time, requiring the system design to adapt to new conditions. The need for ensuring life critical safety is likely to limit the amount of real time adaptation that can be relied upon. Beyond runtime responses, lifecycle safety approaches will need to incorporate significant field engineering feedback based on safety performance indicator monitoring. A continuous monitoring and improvement approach will require a fundamental shift in the safety world-view for automotive applications. Previously, a useful fiction was maintained that vehicles were safe for their entire lifecycle when deployed, and any safety defect was an unwelcome surprise. This approach too often provoked denial and

ICSE keynote: Autonomous Vehicles and Software Safety Engineering

Abstract: Safety assurance remains a significant hurdle for widespread deployment of autonomous vehicle technology. The emphasis for decades has been on getting the technology to work well enough on everyday situations. However, achieving safety for these life-critical systems requires more. While safety encompasses correct operation for the mundane, it also requires special attention to mitigating the risk presented by rare but high consequence potential loss events. In this talk I'll cover some history of autonomous vehicle development and safety at the Carnegie Mellon National Robotics Engineering Center that led over the years to the development of the ANSI/UL 4600 standard for autonomous vehicle safety. I'll also touch upon activities specific to safety engineering, why a heavy tail distribution of rare events makes ensuring safety so difficult, why brute force road testing won't ensure safety, and the emergence of safety assurance cases as the approach of