Blame should not be a factor in AV incident reports

A proposal: Any mention of blame in an autonomous vehicle incident report should immediately discredit the reporting entity's claims of caring about safety and tar them with the brush of "safety theater"

Reading mishap reports from California and NHTSA data it is obvious that so many AV companies are trying as hard as they can to blame anyone and anything other than themselves for crashes. That's not how you get safety -- that's how you do damage control. Sure, in the near term people might buy the damage control. And for many who just see a mention it sticks permanently (think "pedestrian jumped out of the shadows" narrative for the Uber ATG fatality -- the opposite of what actually happened). But that only gets short term publicity benefit at the expense of degrading long term safety. If companies spin and distort safety narratives for each crash, they do not deserve trust for safety.

If a crash report sounds like it was written by lawyers defending the company instead of engineers seeking to improve safety, people can tell. It's not a good look for safety, and indicates the company cares more about image management than safety reality. Any ambiguities and omissions should be interpreted to be the most negative look for the company, especially if it assigns blame (because if something were in the company's favor, they would have said so). Blame insinuation can be very subtle. For example, "AV was stopped at time of crash." OK, so that is factually correct. But why did it stop? Was that because it was the smart defensive driving thing to do, or just a way to make it look like the other guy was at fault? Another example "the other guy was speeding" -- well maybe, but how exactly did that cause the AV to misjudge the other car's speed and subsequently crash? If human drivers crashed every time another driver went faster than the speed limit we would not have enough body repair shops to keep up with demand.

A better approach is that any AV incident report should contain the facts without attempting to assign blame. Vehicle X did A, B, C. Vehicle Y/Pedestrian/bicyclist did D, E, F. AV was in operating mode Z.

For bonus credit mention the things the AV development team could do to avoid the next similar crash. Release video. Release a concise, complete mishap summary report for any injury event or significant damage crash (if you didn't make one internally you don't care about safety, do you?). If the other guy was doing something naughty compare that to the prevalence of such behavior and say why the AV did not take that into account. Make it easy to understand what happened. Explain how the AV will do better next time. Build trust.

In the end blame does not change the number of crashes. If the net harm done by an AV is higher than for human drivers that means the AV is less safe than a human driver. Even if every single crash might be blamed on other road users, how does that make the net harm done disappear? Blame doesn't enter into it.

PA Autonomous Vehicle Testing Legislation Still Needs Work

PA HB 2398 would legalize autonomous vehicles (AVs) without human drivers in Pennsylvania. Having passed the PA House, it is pending in the PA Senate Transportation Committee. While the bill has improved, my 25 years of experience working on AV safety at Carnegie Mellon University leave me with significant remaining concerns:

1. A municipal preemption clause would prevent Pittsburgh from restricting the testing of immature self-driving vehicle technology in active school zones and other high risk locations.
2. A loophole regarding vehicles “approved for noncommercial use” apparently exempts most AVs from certification when using conventional vehicle retrofits, potentially rendering the bill toothless.
3. Test drivers are not required to conform to an established industry standard for testing safety as done elsewhere in the US. Argo AI is the sole company conforming to the relevant safety standard: SAE J3018.
4. The recent National Highway Traffic Safety Administration release of AV crash data makes it clear that crashes will happen. The bill should require AV testers to attest that their technology is acceptably safe. Perfect safety might be unrealistic, but AV companies should at least promise on the record that their testing will be no more dangerous than human driven vehicles.

The current bill leaves Commonwealth constituents unnecessarily vulnerable on public roads. In exchange for placing road users at risk from this work-in-progress technology, we at least deserve to have these issues fixed before the bill is made into law.

Philip Koopman, Ph.D.
Squirrel Hill neighborhood, Pittsburgh PA

----

Pittsburgh Post-Gazette article was published here: https://www.post-gazette.com/news/transportation/2022/06/22/self-driving-vehicles-expanded-testing-safety-concerns-emergency-drivers-pennsylvania-house/stories/202206220121

Update 6/30/22: the PA Senate Transportation Committee updated SB 965 to largely include language from the PA House bill.  It does seem to have addressed two of my points to a degree:

• I no longer see the "approved for noncommercial use" loophole, which is good.
• A safety management plan must be filed, which partially addresses the issue of promising that operations and testing will be no more dangerous than human driven vehicles (but does not require that level of safety). 

See printer number 1839 here:  https://www.legis.state.pa.us/cfdocs/billInfo/billInfo.cfm?sYear=2021&sInd=0&body=S&type=B&bn=0965

PA House HAV bill progress & issues

This past week the PA House Transportation Committee significantly revised and then passed a bill granting sweeping authority to operate Highly Automated Vehicles (HAVs) in the Commonwealth of Pennsylvania. That includes light vehicles, heavy trucks, and platoons of heavy trucks. This bill has evolved over time and seems a better candidate for a final law than the older, much more problematic Senate bill.

It has some good points compared to what we've seen in other states, such as an insurance minimum of $1M, and placing PennDOT in regulatory control instead of Public Safety. By way of contrast, in other states the State Police are in charge of regulating (they have no real ability to do so, and realize this, but the HAV industry pushed to have it this way), and insurance minimums are as low as $25K or $50K. So we're doing better than some other states. 

The PA bill establishes an advisory committee, but it is unclear whether it will have much power, and its current mandate is to report benefits of HAVs without being tasked to report on any public safety concerns (or benefits).

However, a great number of issues identified in earlier versions have not been addressed. A very significant concern is a municipal preemption clause. For example, cities are prevented from curtailing testing of experimental, immature HAVs in school zones, even with no safety driver in the vehicle. 

There are a number of other serious concerns unaddressed by this bill especially in the area of safety, but also with regard to compensation, transparency, inclusion, and non-discrimination: see Five Principles for Regulation of Highly Automated Vehicles.

A particular problematic issue boils down to who goes to jail if an HAV has a software defect that results in driving behavior that would, for a human driver, result in criminal penalties. This bill is at least clear about the "certificate holder" being on the hook, whereas other states are silent on this topic. However, it is unclear if a certificate holder who might have no understanding of HAV software and no ability to influence HAV operational safety is the right person to be sending to jail for reckless driving by an HAV that results in deaths. (Yes, this is a difficult problem. But the HAV industry has had years and years to address concerns such of this. Apparently their plan is to deflect blame away from the tech companies and onto whoever ends up holding the bag as a certificate holder.)

The manner of how HAV bills are being pushed through the legislature is also extremely disappointing. The Senate rammed through a bill that was not disclosed until the last minute with no public hearing. To its credit the House did have a public hearing on its initial bill. However, this very significant modification was kept secret until the Transportation Committee meeting voted it through along party lines. The industry certainly knows what is in the bills and amendments well in advance, because we have had public events in which they were thanked for helping author them. If they really believed that public safety was #1 and stakeholder engagement mattered, the industry would not be resorting to releasing legislation in the dead of night to ram it through votes.

I fully expect this will be pushed through both House and Senate in the most industry-friendly way that can be managed. The PA Governor has already promised to sign HAV legislation. We're going to be stuck with regulations that disproportionately favor the industry so that they can attempt to reap the IPO and SPAC compensation rewards of chasing a trillion dollar market while exporting risks of public road testing to other road users. (Some companies are doing better than others on safety, but the industry as a whole as, for example, represented by AVIA is quite clearly all about the $$$ and not really about public safety.)

It is sad to see legislators seduced by the "jobs and economic opportunity" mantra of the HAV industry while most companies are merely paying lip service to safety. But I guess this is how it will be until we have a sufficiently high number of crashes and other adverse newsworthy events to put on public pressure to do better.

Note: there is one clause that is a potentially HUGE issue. Page 29 lines 16-18 appear to exempt any vehicle that is not strictly commercial (in practice anything except heavy trucks) from the requirement for a PennDOT certificate. It is unclear whether this is an intentional loophole or just a drafting mistake. Either way it should be fixed. 

Cruise robotaxi struggles with real-world emergency vehicle situation

A Cruise robotaxi failed to yield effectively to a fire truck, delaying it.

Sub-headline: Garbage truck driver saves the day when Cruise autonomous vehicle proves itself to not be autonomous.

This referenced article explains the incident in detail, which involves a garbage truck blocking one lane and the Cruise vehicle pulling over into a position that did not leave enough room for the fire truck to pass. But it also argues that things like this should be excused because it is in the cause of developing life saving technology. I have to disagree. Real harm done now to real people should not be balanced against theoretical harm potentially saved in the future. Especially when there is no reason (other than business incentives) to be doing the harm today, and the deployment continues once it is obvious that near-term harm is likely.

I would say that if the car can't drive in the city like a human driver, it should have a human driver to take over when the car can't. Whatever remote system Cruise has is clearly inadequate, because we've seen three problems recently (the article mentions them; an important one is driving with headlights off at night and Cruise's reaction to that incident). The article attributes the root cause of this incident to Cruise not having worked through all interactions with emergency vehicles, which is a reasonable analysis as far as it goes. But why are they operating in a major city with half-baked emergency vehicle interaction?

This time no major harm was done because the garbage truck driver was able to move that vehicle instead. (Realize that garbage trucks stopped in-lane with no driver in the vehicle is business as usual for them, as a human driver would know.) The fire did in fact cause property damage and injuries, so things could have been a lot worse due to a longer delay if not for a quick-acting truck driver. (Who by the way has my admiration for acting quickly and decisively.) What if the garbage truck had been a disabled vehicle or a the truck had been in the middle of an operation it could not be moved during? Then the fire truck would have been stuck. The article says the situation was complex, but driving in the real world is complex. I've personally been in situations where I needed to do something unconventional to let an emergency vehicle pass. A competent human driver understands the situation and acts. Yep, it's complex. If you can't handle complex, don't get on the road without a human backup driver.

The safety driver should not be removed until the vehicle can 100% conform to safety relevant traffic laws and practical resolution of related situations such as this. "Testing" without a human safety driver when the vehicle isn't safe is not testing -- it's just plain irresponsible. There is no technical reason preventing Cruise from keeping a safety driver in their vehicles while they continue testing. Doing so wouldn't delay the technology development in the slightest -- if what they care about is safety. If the safety driver literally has to do nothing, you've still done your testing and your multi-billion dollar company is out a few bucks for safety driver wages. If safety is truly #1, why would you choose to cut costs and remove that safety driver if you know your system isn't at 100% safe yet? Removing the safety driver is pure theater playing to public opinion and, one assumes, investors.

Cruise says that they apply a Safety Management System (SMS) "rigorously across the company." A main point of an SMS is to recognize operational hazards and alter operations in response to discovered hazards. In this case, it is clear that interaction with emergency vehicles requires more sophistication and presents a public safety hazard as currently implemented. Safety drivers should go back into the vehicles until they fix all such issues (not just this one particular interaction scenario) and their vehicle can really drive safely. Unless they simply decide that letting fire trucks on the way to a burning building pass is low priority for them.

Cruise is lucky the delayed fire truck arrival was not attributable to a death -- this time. This incident happened at 4 AM, which shows even in a nearly empty city you need to have a very sophisticated driver to avoid safety issues. At the very least they should halt no-human-driver operations until they can attest that they can handle every possible emergency vehicle interaction without causing more delay to the emergency vehicle than a proficient human driver, including situations in which a human driver would normally get creative to allow emergency vehicle progress. City officials wrote in a filing to the California Public Utilities Commission: "This incident slowed SFFD response to a fire that resulted in property damage and personal injuries,” and were concerned that frequent in-lane stops by Cruise vehicles could have a "negative impact" on fire department response times. Every safety related incident needs to be addressed. It is a golden opportunity to improve before you get unlucky. Cruise says they have a "rigorous" SMS, but I'm not seeing it. Will Cruise learn? Or will they keep rolling dice without safety drivers? Cruise shouldn't wait for something worse to happen before getting the message that they need to do better if they want to operate without a safety driver.

Tesla emergency door releases -- what a mess!

The Tesla manual door releases -- and lack thereof in some cases -- present unreasonable risk. What in the world were they thinking? Really bad human interface design. Cool design shouldn't come at expense of life critical peril. This article this week sums up the latest, but this has been going on for a long time.

Tesla fans seem to be saying that it is the driver's responsibility to know where the manual release latch is to escape in case of fire. Anyone who doesn't is (and has in past fires) been ridiculed on-line for not knowing where the manual release is hidden. Even if they died due to not successfully operating the control, or having to kick the window out, somehow they are the idiots and it is their fault, not Tesla's. (If someone you love has died or been injured in this way you have my sympathy, and it is the trolls who are idiots, not your loved one.)

On-line articles saying "here's how to operate the door release so you don't die in a Tesla fire" tell you there is a problem. This design is unreasonably risky for real world use. A "bet you didn't know -- so here is how to not die" article in social media means there is unacceptable risk. Example:  "Tesla Model Y fire incident: remember, there's a manual door release, here's how to use it in an emergency."

Front doors you have to lift up a not particularly obvious lever in front of the window switches that is easy to miss if you don't know it is there. Maybe if you have used it a few times -- but if you never realized it is there or you have rented/borrowed the car, good luck with that.  I'd probably have trouble finding it even if I weren't suffocating from smoke from a battery fire. (Have you ever had to consult the owner manual to find your hood release? Imagine doing that to find out how to open the door when your car is literally on fire -- oh, but if it is an electronic manual and you've lost power, you can't do that on the center console, can you?)

And if you're a passenger and driver is unconscious you will have issues. Etc. Do you read all the safety instructions in the driver manual when you catch a quick ride as a passenger with a friend? Does your friend brief you on escape safety features so you can exit before a 5 minute ride? Thought not.

But wait, there's more:

• Model S rear door: "fold back the edge of the carpet" to find a pull cable
• Model X falcon wing doors: "carefully remove the speaker grille from the door and pull the mechanical release cable..."
• Model 3 rear door -- NOT EQUIPPED WITH MANUAL RELEASE (from manual: "Only the front doors are equipped with a manual door release")

So I guess the passengers in the back are kind of expendable. For many that will be the kids.

This is stunningly bad human interface design. It is entirely unreasonable to expect an ordinary car owner to know where a hidden/non-obvious emergency control is and activate it when they are trapped inside a burning car. Let alone passengers. Apparently without mandatory training and mandatory periodic refresher training.

Anyone who thinks it is reasonable to expect someone not trained in military/aviation/etc. to get this right probably has not served or been through that type of training. I have been through tons of training. Emergency drills that might give some nightmares (sealed inside a tank with broken pipes and told to plug the flooding is extra-special). And a few times the real thing. Not always with perfect execution, because there is compelling data showing humans suck at performing complicated, non-reflex-trained tasks under stress (and thus, more practice, more drills). After all that, I wouldn't want to risk my life on this hot mess of an egress system. 

Education and shaming won't prevent the next death from this unreasonable risk. 

I can't imagine why NHTSA wouldn't want to do a recall on this.

(To the extent this is true of other brands that is equally problematic. I don't have info on them.)

EDIT: a Linkedin commenter pointed me to this story about a Corvette fatality related to a similar issue. From what I can tell repair parts for Corvettes indicated a clearly marked egress pull that is on the floorboards. So not ideal, and possibly difficult to see if you are already in the seat. Worth reconsidering. But not literally hidden (or missing) as in Teslas, and certainly not in vehicles being sold as family cars.  Perhaps now that Tesla has pushed the enveloped past any reasonable limits it's time for standards on egress actuator visibility and accessibility.

This has been a known issue at least since a 2019 crash, summarized here: https://www.autoblog.com/2019/02/28/tesla-fiery-crash-closer-look-door-locks/     That fatality also had to do with door handles not popping up after a crash, so a rescuer was unable to open doors from the outside. It's time to pay attention before more people get trapped inside burning cars.

A gentle introduction to autonomous vehicle safety cases

I recently ran into this readable article about AV safety cases by Thomas & Vandenberg from 2019. While things have changed a bit, it still is a reasonable introduction for anyone asking "what exactly would an AV safety case look like."

A real industry-strength safety case is going to be complicated in many ways. In particular, there are many different approaches for breaking down G1 which will significantly affect things. On the other hand all the pieces will need to be there somewhere, so choosing this high level breakdown is more of an architectural choice (for the safety case, not necessarily the system). We do not yet have a consensus on an optimal strategy for building such safety cases, but this is not a bad starting place from safety folks who were previously at Uber ATG.

Thomas & Vandenberg, Harnessing Uncertainty in Autonomous Vehicle Safety, Journal of System Safety, Vol. 55, No. 2 (2019)

https://doi.org/10.56094/jss.v55i2.46

(Uber ATG also published a much more complex safety case. However, I recommend this overview paper rather than that more complex safety case to get insight if you are just getting started.)

SEAMS Keynote talk: Safety Performance Indicators and Continuous Improvement Feedback

Abstract: Successful autonomous ground vehicles will require a continuous improvement strategy after deployment. Feedback from road testing and deployed operation will be required to ensure enduring safety in the face of newly discovered rare events. Additionally, the operational environment will change over time, requiring the system design to adapt to new conditions. The need for ensuring life critical safety is likely to limit the amount of real time adaptation that can be relied upon. Beyond runtime responses, lifecycle safety approaches will need to incorporate significant field engineering feedback based on safety performance indicator monitoring.

A continuous monitoring and improvement approach will require a fundamental shift in the safety world-view for automotive applications. Previously, a useful fiction was maintained that vehicles were safe for their entire lifecycle when deployed, and any safety defect was an unwelcome surprise. This approach too often provoked denial and minimization of the risk presented by evidence of operational safety issues so as to avoid expensive recalls and blame. In the future, the industry will need to embrace a model in which issues are proactively detected and corrected in a way that avoids most loss events, and that uses field incident data as a primary driver of improvement. Responding to automatically generated field incident reports to avoid later losses should be a daily practice in the normal course of business rather than evidence of an engineering mistake for which blame is assigned. This type of engineering feedback approach should complement any on-board runtime adaptation and fault mitigation.

• Talk video: https://youtu.be/mRXotHN0Z6I
• Slides: https://users.ece.cmu.edu/~koopman/lectures/L127-2022-04-SEAMS-Feedback-SPIs.pdf
• Archive.org downloadable mirror: http://archive.org/details/l127-safety-performance-indicators-and-continuous-improvement-feedback_202205

ICSE keynote: Autonomous Vehicles and Software Safety Engineering

Abstract: Safety assurance remains a significant hurdle for widespread deployment of autonomous vehicle technology. The emphasis for decades has been on getting the technology to work well enough on everyday situations. However, achieving safety for these life-critical systems requires more. While safety encompasses correct operation for the mundane, it also requires special attention to mitigating the risk presented by rare but high consequence potential loss events. In this talk I'll cover some history of autonomous vehicle development and safety at the Carnegie Mellon National Robotics Engineering Center that led over the years to the development of the ANSI/UL 4600 standard for autonomous vehicle safety. I'll also touch upon activities specific to safety engineering, why a heavy tail distribution of rare events makes ensuring safety so difficult, why brute force road testing won't ensure safety, and the emergence of safety assurance cases as the approach of choice for autonomous vehicle safety.

• YouTube Link:   https://youtu.be/oE_2rBxNrfc
• Downloadable/alternate video source: https://archive.org/details/koopman-2022-icse-av-sw-safety
• Slides: https://users.ece.cmu.edu/~koopman/lectures/L128-2022-ICSE_AI_SW_Safety.pdf

OTA updates won't save buggy autonomous vehicle software

There is a feeling that it's OK for software to ship with questionable quality if you have the ability to send out updates quickly. You might be able to get away for this with human-driven vehicles, but for autonomous vehicles (no human driver responsible for safety) this strategy might collapse.

Right now, companies are all pushing hard to do quick-turn Over The Air (OTA) software updates, with Tesla being the poster child of both shipping dodgy software and pushing out quick updates (not all of which actually solve the problem as intended). There is a moral hazard that comes with the ability to do quick OTAs in that you might not spend much time on quality since you know you can just send another update if the first one doesn't turn out as you hoped.

"There's definitely the mindset that you can fix fast so you can take a higher risk," Florian Rohde, a former Tesla validation manager   (https://www.reuters.com/article/tesla-recalls-idTRNIKBN2KN171)

For now companies across an increasing number of industries have been getting away with shipping lower quality software, and the ability to do internet-based updates has let them get by with such a strategy. The practice is so prevalent that the typical trouble-shooting for any software after "is the power turned on" has become "have you downloaded the latest updates."

But the reason this approach works (after a fashion) is that there is a human user or vehicle operator present to recognize something is wrong, work around the problem, and participate in the trouble shooting. In a fully automated vehicle, that human isn't going to be there to save the day.

What happens when there is no human to counter-act the defective and potentially fatally dangerous defective software behavior? The biggest weakness of any automation is typically that it is not "smart" enough to know when something is going wrong that is not supposed to happen. People are pretty good at this, which is why even for very serious software defects in cars we often see huge numbers of complaints compared to few actual instances of harm -- because human drivers have compensated for the bad software behavior.

Here's a concrete example of a surprising software defect pulled from my extensive list of problematic automotive software defects: NHTSA Recall 14V-204:

• Due to software calibration error vehicle may be in and display "drive" but engage "reverse" for 1.5 seconds.

If a human driver notices the vehicle going the wrong direction they'll stop accelerating pretty quickly. They might hit something at slow speed during the reaction time, but they'll realize something is wrong without having explicit instructions for that particular failure scenario. In contract, a computer-based system that has been taught the car always moves in the direction of the transmission display might not even realize something is wrong and accelerate into collisions.

Obviously a computer can be programmed to deal with such a situation if it has been thought of at design time. But the whole point here is that this is something that isn't supposed to happen -- so why would you waste time programming a computer to handle an "impossible" event? Safety engineering deals with hazard analysis to mitigate low risk things, but even that often overlooks "impossible" events until after they've occurred. Sure, you can send an OTA update after the crash -- but that doesn't bring crash victims back to life.

In practice the justification that it is OK to ship out less-than-perfect automotive software has been that human drivers can compensate for problems. (In the ISO 26262 functional safety standard one takes credit for "controllability" in reducing the risk of a potential defect.) When there is no human driver, that's a problem, and shipping defective software is more likely to result in harm to a vehicle occupant or other road user before it can be noticed there is a problem for OTA to correct.

Right now, a significant challenge to OTA updates is the moral hazard that software will be a bit more dangerous than it should be due to pushing the boundaries of human driver ability to compensate for defects. With fully automated vehicles there will be a huge cliff of ability, and even small OTA update defects could result in large numbers of crashes across a fleet before there is time to correct the problem. (If you push a bad updated to millions of cars, you can have a lot of crashes even in a single day for a defect that affects a common driving situation.)

The industry is going all-in on fast and loose OTAs to be more "agile" and iterate software changes more quickly without worrying as much about quality. But I think they're heading right into a proverbial brick wall that will be hit when the human drivers are taken out of the loop. Getting software quality right will become more important than ever for fully autonomous vehicles.

 

Maturity Levels for Autonomous Vehicle Safety

I've been on a personal journey to understand what safety really means for autonomous vehicles. As part of this I repeatedly find myself in conversations in which participants have wildly different notions of what it means to be "safe."  Here is an attempt to put some structure around the discussion:

An inspiration for this idea is Maslow's famous hierarchy of needs. The idea is that organizations developing autonomous vehicles have to take care of the lower levels before they might be able to afford at higher levels. For example, if your vehicle crashes every 100 meters because it struggles to detect obstacles in ideal conditions, worrying about nuances of lifecycle support won't get you your next funding round.

To succeed as a viable at-scale company, you need to address all the levels in the AV maturity hierarchy. But in reality companies will likely climb the levels like rungs in a ladder. To draw the parallel to Maslow's needs hierarchy, if a company is starving for cash to run its operations, they're going to care more about getting the next demo or funding milestone compared to lifecycle safety considerations. That will only change when venture funding bakes higher levels of this safety maturity hierarchy into their milestones. 

• Basic driving functionality: the vehicle works in a defined environment without hitting any objects or other road users on the defined scale of a funding milestone demo. When people say that their vehicle is safe because it has high crash safety ratings, that aligns with this bin. (I personally prefer my safety to happen without the part where the vehicle crashes.)
• Defensive driving: vehicle has expert driving skills, actively avoiding driving situations that present increased risk. This is analogous to sending a human driver to defensive driving school. At some point the automated driver becomes expert in terms of being able to drive in failure-free situations.
• Systematic hazard analysis: engineering effort has been spent analyzing and mitigating risks not just from driving functions, but also potential technical malfunctions, forced exits from the intended operational design domain, etc.  (For example, HARA from ISO 26262.) Common hazards that aren't easy or inexpensive to mitigate might well be pushed onto the driver (e.g., incomplete redundancy to handle component failure, or required driver intervention to mitigate risks).
• Functional safety: analysis and redundancy have been added, and a principled approach (e.g., based on safety integrity levels) has been taken to ensure risks from technical faults in the system have been mitigated (e.g., ISO 26262 conformance).
• Safety of the Intended Function (SOTIF): ensuring "unknowns" have been addressed, dealing with environmental influences (e.g., not all radar pings will be returned), closing gaps in requirements, and accounting for aspects of machine learning. (e.g., ISO 21448 conformance).
• System level safety: accounting for things beyond just the driving task, including lifecycle considerations. Ensuring that hazard analysis and mitigation extends to process aspects, and a safety case has been used to ensure acceptable safety. (e.g., ANSI/UL 4600)
  Cybersecurity needs to be addressed to achieve system safety, but should not wait to get started until reaching this level.
• Just Safety Culture: operating and continuously improving the organization and execution of other levels of the hierarchy according to Just Culture principles rather than blame.
  Specific common anti-patterns for Just Culture relevant to autonomous vehicles are:
• Using humans as Moral Crumple Zones (blaming humans so as to evade addressing technical or process failures)
• Using propaganda and regulatory lobbying to deflect liability away from companies, reducing pressure to achieve acceptable safety.

As with the Maslow hierarchy the levels are not exclusive. Rather all levels need to operate concurrently, with the highest concurrently active level indicating progress toward safety maturity.

You might see this differently, see some things I've missed, etc. Comments welcome!

Cruise Stopped by Police for Headlights Off -- Why Is This a Big Deal?

In April 2022 San Francisco police pulled over an uncrewed Cruise autonomous test vehicle for not having its headlights on. Much fun was had on social media about the perplexed officer having to chase the car a few meters after it repositioned during the traffic stop. Cruise said it was somehow intentional behavior. They also said their vehicle "did not have its headlights on because of a human error" (Source: https://www.theguardian.com/technology/2022/apr/11/self-driving-car-police-pull-over-san-francisco) 

The traffic behavior indicates that Cruise needs to do better making it easier for local police to do traffic stops -- but that's not the main event. The real issue here is the headlights being off.

Cruise said in a public statement: "we have fixed the issue that led to this." Forgive me if I'm not reassured. A purportedly completely autonomous car (the ones that are supposed to be nearly perfect compared to those oh-so-flawed human drivers) that always drives at night didn't know to turn its headlights on when driving in the dark. Seriously? Remember that headlights on at night is not just for the AV itself (which might not need them in good city lighting with a lidar etc.) but also to be visible to other road users. So headlights off at night is a major safety problem. (Headlights on in day is also a good idea -- at least white forward-facing running lights, which this vehicle also did not seem to have on.)

Cruise says these vehicles are autonomous -- no human driver required. And they only test at night, so needing headlights can hardly be a surprise. But they got it wrong. How can that be? 

This can't just be hand-waved away. Something really fundamental seems broken, and it is just a question of what it is.

The entire AV industry, including Cruise, has a history of extreme secrecy and in particular lack of transparency with safety. So we can only speculate. Here are the possibilities I can think of. None of them inspire confidence in safety, and they tend to get worse as we go down the list.

1. The autonomy software had a defect that didn't turn headlights on at night. Perhaps, but seems unlikely. That would (one assumes) affect the entire fleet. And there should be software checks to make sure the headlights turn on as a safety check built into the software. If true, this never should have slipped through quality control let alone a rigorous safety critical software design process, and indicates significant concerns with overall software safety.
2. The headlight switch is supposed to be on at all times. Many (most?) vehicles have "smart" lights. You can turn them off if you want, but in practice you just turn it to "on" and leave it there for months or years and the headlights just do the right thing, switching from daytime running lights to full on automatically, and turning off when the vehicle does. If you're in urban San Francisco high beams are unlikely to be relevant. So the autonomy doesn't mess with the lights at all. Except -- why does the software not check to see that the vehicle condition is safe in terms of headlights? Seems like a design oversight. How did a hazard like this get missed in hazard analysis? If this is the situation, this really calls into question whether the hazard analysis was done properly for the software. Even if this was fixed, what else did hazard analysis miss?
• 2a) A passenger in the vehicle turned the headlights off as a prank. If this is possible, even more important for this to be called out for software monitoring in the hazard analysis. But the check for headlights off obviously isn't there now. 
3. The software is completely ignorant of headlight state, and there is a maintenance tech who is supposed to turn the lights "on" as part of the check-out process each day to prepare the vehicle to run. This manual headlight-on check didn't get done. This is a huge issue with the Safety Management System, because if they missed that check what other checks did they miss? There are plenty of things to check on these vehicles for safety (e.g., sensor calibration, sensor cleaning/cleaner systems, correct software configuration). If they forget to turn on the headlight switch, what else did they forget? While this might be a "within normal tolerance" deviation, given lack of safety transparency Cruise doesn't get the benefit of the doubt. A broken SMS puts all road users at risk. This is a big deal unless proven otherwise. Firing, training, or having a stern talk with the human who made the "error" won't stop it from happening again. Blaming the driver/pilot/human never really does.
4. There is no SMS. That's basically how Uber ATG ended up killing a pedestrian in Tempe Arizona. If this is the case, it is even scarier.

Again, we don't really know the situation because Cruise is trying the "nothing to see here .. move along" response to this incident. But none of these scenarios is comforting.  If I had to guess my money would be on #3, simply because #4 would be too irresponsible to have to contemplate. But really, we have no way to know what's really going on. And it might be another alternative I have not considered.

Cruise should take this as a wakeup call to get their safety house in order before they have a big event. Blaming safety critical failures on "human error" is generally indicative of a poor safety culture. They have a chance here to turn the ship around -- before there is harm to a road user.

Is there a scenario I missed that is less of a concern? Maybe Cruise will give us a substantive explanation. If they do I'll be happy to post it here as a follow-up.

-----------------------------------------------------------------------------------

Kyle Vogt at Cruise sent me a response on LinkedIn on 4/16/2022:

Kyle Vogt
CEO at Cruise

I’d like to respectfully disagree with your characterization of this event. Allow me to provide some context.

We apply SMS rigorously across the company, which as you probably know includes estimating the safety risk for known hazards and having a process to continuously surface new ones.

We then apply resources accordingly to attempt to reduce the overall safety risk of our service as effectively as possible. This is a continuous process and we will continue even though we’ve passed the point where we can operate without backup drivers.

Rarely would this direct us to focus on things that are extremely unlikely to result in injury, such as lack of headlights on a single vehicle for a short duration of time. We have a process in place to ensure they are functioning properly, but the safety impact of that process failing is low.

Our use of SMS directs the majority of our resources towards higher risk areas like pedestrian and cyclist interactions. These are far more complex and of higher potential severity.

I firmly believe that’s this is the right approach, even if it means our vehicles occasionally do something that seems obviously wrong to a human.

-----------------------------------------------------------------------------------

My response on 4/17/2022:

@Kyle Vogt: Thank you for responding to my post. I am glad to hear that you have a Safety Management System in place now. This is excellent news.
Hopefully your SMS folks are telling you that the headlight-off incident should be treated as a serious safety process issue rather than being sloughed off as a "no big deal." A healthy safety culture would acknowledge this is a process failure that needs to be corrected rather than making excuses based on "safety impact of that process failing is low." Process failures in safety critical system operations are a problem -- period. An attitude that less critical processes don't matter is easily toxic for safety culture. Safety culture is how you do business, and if disregarding procedures that are seen as less than the most critical is how you do business, eventually that will catch up with you resulting in loss events. More simply: discounting a low severity process failure is at odds with your statement that you "apply SMS rigorously across the company."
I would say that headlights off has a substantive potential severity for other road users because they are signaling/warning devices to help people know to get out of the way if your vehicle should malfunction. For example, with headlights off you cannot take controllability credit for the other road user jumping out of your way if you fail to detect them, because they won't see headlights as an indicator of your approach at night. Moreover, having headlights on is the law.
Regardless of your statement that you are focusing on higher severity issues, a process failure for something as simple and obvious as making sure headlights are on at night is very disconcerting, and raises concerns over your safety process quality in general. A more transparent analysis of how that process failed and whether other higher severity process failures are occurring that have not been made public would be the right thing to do here to restore public trust. If this is the only time the headlights were off and your other pre-mission procedures have a high compliance rate, then OK, stuff happens. However -- if pre-mission procedures are hit or miss and you don't even keep track of procedural errors that don't result in a scary pedestrian near-miss, that is quite another. If you don't have the procedural metrics to show you are operating safely, then probably you aren't operating safely. Which is the case?
Your GM-issued VSSA from 2018 seems outdated and has no mention of an SMS. So it is difficult for the public to know what your plan is for safety. More transparency and communication regarding your safety process is essential to help build public trust.

-----------------------------------------------------------------------------------

ANSI/UL 4600 Version 2 (2022)

Version 2 of ANSI/UL 4600 has just been issued. This standard provides guidance on how to ensure that autonomous vehicles safety cases are created and maintained to ensure acceptable safety for deployment.

Since version 1 of the standard was issued in April 2020, the Standards Technical Panel members (the voting committee) and stakeholders have been involved in suggesting clarifications, upgrades, and other improvements as part of the standard's continuous improvement process. 

Version 1 of the standard included chapters on: terminology, safety cases, risk assessment, interaction with humans, autonomy functions, software/system engineering processes, dependability, data/networking, verification/validation/test, tool qualification/COTS/legacy components, lifecycle concerns, maintenance, metrics, and assessment. The standard is designed to work with other safety standards such as ISO 26262 and ISO 21448 to make sure that all the bases are covered for system-level safety on autonomous road vehicles.

Version 2 has some substantive changes:

• The assessment terminology and role of independent assessment have been changed. This is compatible with the previous approach except that the independent assessor has a more substantive role.
• Self-assessment: the development team creates and vets its own safety case.
• Independent assessment: an independent organization examines both the form and the technical substance of the safety case to ensure it is acceptable.  (This independent organization might be within the same company at an arms-length relationship, or could be an external assessor at the company's option.)
• Significant clean-up of the discussion of safety case terminology and structure. This is overall a significant improvement of the ideas that were already there in version 1, but a lot of work went into this area.
• The terminology section has been substantially rewritten to clean up wording and improve alignment with other standards without substantively changing the intended meaning of terms being used.

Version 2 maintains the same structure as Version 1 with some minor changes to top level requirements. Significant attention has been paid to ensure a smooth transition between versions. While much of the document has been updated, the updates beyond the substantive changes tend to be relatively minor in scope and are more in the nature of clarification and adding helpful detail. The changes significantly improve the standard, but the vast majority do not fundamentally change its nature or general requirements.

Overall there were seven different task groups whose members spent many hours contributing to improving the standard, and there are important changes in each area to improve the standard.  Those task groups drafted proposals to modify content regarding: safety cases, faults/hazard/risks, assessment, sensor requirements, object tracking, safe egress, and terminology. 

Everyone who participated deserves a big "Thank You!"  Also, a special thanks to Deborah Prince and Heather Sakellariou at Underwriters Laboratories for coordinating all the activities and keeping everything on track, especially with the challenges presented by the whole process being done electronically via e-mail and on-line meetings. It has truly been a pleasure to work with a group that is so dedicated to this collaborative effort to ensure the safety of autonomous vehicle technology.

The official kickoff of UL 4600 version 3 is planned for April, with the biggest goal being extending the standard to cover special considerations that apply to autonomous heavy trucks. Anyone interested in contributing to that discussion can request stakeholder status (no participation fee required).

You can find a pointer to a copy of the newly released version 2 including FREE (with free digital account) digital access to a full copy of the standard, a list of committee members, and more here: https://users.ece.cmu.edu/~koopman/ul4600/index.html 

The official news release from the standards organization is here: https://ul.org/standards-and-engagement/presenting-standard-safety-evaluation-autonomous-vehicles/second-edition 

Maturation path for safety & security practices

Brief informal notes from a wrap-up quick position statement talk I did at a workshop today.

Both safety and security have a lot in common in terms of how they are maturing over time. Without getting into a religious debate about the difference between them, I note that their trajectory seems to include the following steps, especially for autonomous systems. I'd argue that each step is in a sense more mature than the previous step.

1. Get the system to work. Safety/security can come later.
2. Get the system to work almost all the time. Conflate this with safety/security even though you're still really just getting it to work in the common cases (safety for a vehicle is "doesn't hit stuff" while security is "doesn't get taken down by the usual continuous stream of automated attacks")
3. Brute force problem fixes:   fly/crash/fix/fly (air) and drive/crash/fix/drive (ground)
4. Create a set of best practices in the nature of a building code ("build your system this way")
• Create a useful fiction that you have completely characterized the requirements and operational environment and that your building code will always work.
• Any failure is an embarrassing piece of bad news that violates the fiction of complete understanding.
5. As system matures, complain about false alarm safety/security shutdowns
• It might feel like this means your system has problems, but in fact you're a lot more safe and secure than systems that operate oblivious to their vulnerabilities
6. Start permitting breaking the building code standard rules by arguing that exceptions still result in equivalent safety/security
7. Evolve to full-up deductive assurance cases to argue safety/security beyond building codes
• Still maintain the fiction of complete knowledge of requirements and environment
8. Start operating in more open environments and admit you didn't really understand requirements, nor environment
• Spend a lot of time chasing down problems that reveal defects in your safety case (safety case does not match environmental assumptions, or might not even match deployed system)
9. Switch to an inductive safety case approach:
• Account for risk from epistemic uncertainty (unknown unknowns)
• Instrument system for failure precursors (e.g., safety performance indicators tied to safety case claims)
• Treat incidents as an opportunity to fix problems before there is a loss event.