Tesla emergency door releases -- what a mess!

The Tesla manual door releases -- and lack thereof in some cases -- present unreasonable risk. What in the world were they thinking? Really bad human interface design. Cool design shouldn't come at expense of life critical peril. This article this week sums up the latest, but this has been going on for a long time.

Tesla fans seem to be saying that it is the driver's responsibility to know where the manual release latch is to escape in case of fire. Anyone who doesn't is (and has in past fires) been ridiculed on-line for not knowing where the manual release is hidden. Even if they died due to not successfully operating the control, or having to kick the window out, somehow they are the idiots and it is their fault, not Tesla's. (If someone you love has died or been injured in this way you have my sympathy, and it is the trolls who are idiots, not your loved one.)

On-line articles saying "here's how to operate the door release so you don't die in a Tesla fire" tell you there is a problem. This design is unreasonably risky for real world use. A "bet you didn't know -- so here is how to not die" article in social media means there is unacceptable risk. Example:  "Tesla Model Y fire incident: remember, there's a manual door release, here's how to use it in an emergency."

Front doors you have to lift up a not particularly obvious lever in front of the window switches that is easy to miss if you don't know it is there. Maybe if you have used it a few times -- but if you never realized it is there or you have rented/borrowed the car, good luck with that.  I'd probably have trouble finding it even if I weren't suffocating from smoke from a battery fire. (Have you ever had to consult the owner manual to find your hood release? Imagine doing that to find out how to open the door when your car is literally on fire -- oh, but if it is an electronic manual and you've lost power, you can't do that on the center console, can you?)

And if you're a passenger and driver is unconscious you will have issues. Etc. Do you read all the safety instructions in the driver manual when you catch a quick ride as a passenger with a friend? Does your friend brief you on escape safety features so you can exit before a 5 minute ride? Thought not.

But wait, there's more:

• Model S rear door: "fold back the edge of the carpet" to find a pull cable
• Model X falcon wing doors: "carefully remove the speaker grille from the door and pull the mechanical release cable..."
• Model 3 rear door -- NOT EQUIPPED WITH MANUAL RELEASE (from manual: "Only the front doors are equipped with a manual door release")

So I guess the passengers in the back are kind of expendable. For many that will be the kids.

This is stunningly bad human interface design. It is entirely unreasonable to expect an ordinary car owner to know where a hidden/non-obvious emergency control is and activate it when they are trapped inside a burning car. Let alone passengers. Apparently without mandatory training and mandatory periodic refresher training.

Anyone who thinks it is reasonable to expect someone not trained in military/aviation/etc. to get this right probably has not served or been through that type of training. I have been through tons of training. Emergency drills that might give some nightmares (sealed inside a tank with broken pipes and told to plug the flooding is extra-special). And a few times the real thing. Not always with perfect execution, because there is compelling data showing humans suck at performing complicated, non-reflex-trained tasks under stress (and thus, more practice, more drills). After all that, I wouldn't want to risk my life on this hot mess of an egress system. 

Education and shaming won't prevent the next death from this unreasonable risk. 

I can't imagine why NHTSA wouldn't want to do a recall on this.

(To the extent this is true of other brands that is equally problematic. I don't have info on them.)

EDIT: a Linkedin commenter pointed me to this story about a Corvette fatality related to a similar issue. From what I can tell repair parts for Corvettes indicated a clearly marked egress pull that is on the floorboards. So not ideal, and possibly difficult to see if you are already in the seat. Worth reconsidering. But not literally hidden (or missing) as in Teslas, and certainly not in vehicles being sold as family cars.  Perhaps now that Tesla has pushed the enveloped past any reasonable limits it's time for standards on egress actuator visibility and accessibility.

This has been a known issue at least since a 2019 crash, summarized here: https://www.autoblog.com/2019/02/28/tesla-fiery-crash-closer-look-door-locks/     That fatality also had to do with door handles not popping up after a crash, so a rescuer was unable to open doors from the outside. It's time to pay attention before more people get trapped inside burning cars.

A gentle introduction to autonomous vehicle safety cases

I recently ran into this readable article about AV safety cases by Thomas & Vandenberg from 2019. While things have changed a bit, it still is a reasonable introduction for anyone asking "what exactly would an AV safety case look like."

A real industry-strength safety case is going to be complicated in many ways. In particular, there are many different approaches for breaking down G1 which will significantly affect things. On the other hand all the pieces will need to be there somewhere, so choosing this high level breakdown is more of an architectural choice (for the safety case, not necessarily the system). We do not yet have a consensus on an optimal strategy for building such safety cases, but this is not a bad starting place from safety folks who were previously at Uber ATG.

Thomas & Vandenberg, Harnessing Uncertainty in Autonomous Vehicle Safety, Journal of System Safety, Vol. 55, No. 2 (2019)

https://doi.org/10.56094/jss.v55i2.46

(Uber ATG also published a much more complex safety case. However, I recommend this overview paper rather than that more complex safety case to get insight if you are just getting started.)

SEAMS Keynote talk: Safety Performance Indicators and Continuous Improvement Feedback

Abstract: Successful autonomous ground vehicles will require a continuous improvement strategy after deployment. Feedback from road testing and deployed operation will be required to ensure enduring safety in the face of newly discovered rare events. Additionally, the operational environment will change over time, requiring the system design to adapt to new conditions. The need for ensuring life critical safety is likely to limit the amount of real time adaptation that can be relied upon. Beyond runtime responses, lifecycle safety approaches will need to incorporate significant field engineering feedback based on safety performance indicator monitoring.

A continuous monitoring and improvement approach will require a fundamental shift in the safety world-view for automotive applications. Previously, a useful fiction was maintained that vehicles were safe for their entire lifecycle when deployed, and any safety defect was an unwelcome surprise. This approach too often provoked denial and minimization of the risk presented by evidence of operational safety issues so as to avoid expensive recalls and blame. In the future, the industry will need to embrace a model in which issues are proactively detected and corrected in a way that avoids most loss events, and that uses field incident data as a primary driver of improvement. Responding to automatically generated field incident reports to avoid later losses should be a daily practice in the normal course of business rather than evidence of an engineering mistake for which blame is assigned. This type of engineering feedback approach should complement any on-board runtime adaptation and fault mitigation.

• Talk video: https://youtu.be/mRXotHN0Z6I
• Slides: https://users.ece.cmu.edu/~koopman/lectures/L127-2022-04-SEAMS-Feedback-SPIs.pdf
• Archive.org downloadable mirror: http://archive.org/details/l127-safety-performance-indicators-and-continuous-improvement-feedback_202205

ICSE keynote: Autonomous Vehicles and Software Safety Engineering

Abstract: Safety assurance remains a significant hurdle for widespread deployment of autonomous vehicle technology. The emphasis for decades has been on getting the technology to work well enough on everyday situations. However, achieving safety for these life-critical systems requires more. While safety encompasses correct operation for the mundane, it also requires special attention to mitigating the risk presented by rare but high consequence potential loss events. In this talk I'll cover some history of autonomous vehicle development and safety at the Carnegie Mellon National Robotics Engineering Center that led over the years to the development of the ANSI/UL 4600 standard for autonomous vehicle safety. I'll also touch upon activities specific to safety engineering, why a heavy tail distribution of rare events makes ensuring safety so difficult, why brute force road testing won't ensure safety, and the emergence of safety assurance cases as the approach of choice for autonomous vehicle safety.

• YouTube Link:   https://youtu.be/oE_2rBxNrfc
• Downloadable/alternate video source: https://archive.org/details/koopman-2022-icse-av-sw-safety
• Slides: https://users.ece.cmu.edu/~koopman/lectures/L128-2022-ICSE_AI_SW_Safety.pdf

OTA updates won't save buggy autonomous vehicle software

There is a feeling that it's OK for software to ship with questionable quality if you have the ability to send out updates quickly. You might be able to get away for this with human-driven vehicles, but for autonomous vehicles (no human driver responsible for safety) this strategy might collapse.

Right now, companies are all pushing hard to do quick-turn Over The Air (OTA) software updates, with Tesla being the poster child of both shipping dodgy software and pushing out quick updates (not all of which actually solve the problem as intended). There is a moral hazard that comes with the ability to do quick OTAs in that you might not spend much time on quality since you know you can just send another update if the first one doesn't turn out as you hoped.

"There's definitely the mindset that you can fix fast so you can take a higher risk," Florian Rohde, a former Tesla validation manager   (https://www.reuters.com/article/tesla-recalls-idTRNIKBN2KN171)

For now companies across an increasing number of industries have been getting away with shipping lower quality software, and the ability to do internet-based updates has let them get by with such a strategy. The practice is so prevalent that the typical trouble-shooting for any software after "is the power turned on" has become "have you downloaded the latest updates."

But the reason this approach works (after a fashion) is that there is a human user or vehicle operator present to recognize something is wrong, work around the problem, and participate in the trouble shooting. In a fully automated vehicle, that human isn't going to be there to save the day.

What happens when there is no human to counter-act the defective and potentially fatally dangerous defective software behavior? The biggest weakness of any automation is typically that it is not "smart" enough to know when something is going wrong that is not supposed to happen. People are pretty good at this, which is why even for very serious software defects in cars we often see huge numbers of complaints compared to few actual instances of harm -- because human drivers have compensated for the bad software behavior.

Here's a concrete example of a surprising software defect pulled from my extensive list of problematic automotive software defects: NHTSA Recall 14V-204:

• Due to software calibration error vehicle may be in and display "drive" but engage "reverse" for 1.5 seconds.

If a human driver notices the vehicle going the wrong direction they'll stop accelerating pretty quickly. They might hit something at slow speed during the reaction time, but they'll realize something is wrong without having explicit instructions for that particular failure scenario. In contract, a computer-based system that has been taught the car always moves in the direction of the transmission display might not even realize something is wrong and accelerate into collisions.

Obviously a computer can be programmed to deal with such a situation if it has been thought of at design time. But the whole point here is that this is something that isn't supposed to happen -- so why would you waste time programming a computer to handle an "impossible" event? Safety engineering deals with hazard analysis to mitigate low risk things, but even that often overlooks "impossible" events until after they've occurred. Sure, you can send an OTA update after the crash -- but that doesn't bring crash victims back to life.

In practice the justification that it is OK to ship out less-than-perfect automotive software has been that human drivers can compensate for problems. (In the ISO 26262 functional safety standard one takes credit for "controllability" in reducing the risk of a potential defect.) When there is no human driver, that's a problem, and shipping defective software is more likely to result in harm to a vehicle occupant or other road user before it can be noticed there is a problem for OTA to correct.

Right now, a significant challenge to OTA updates is the moral hazard that software will be a bit more dangerous than it should be due to pushing the boundaries of human driver ability to compensate for defects. With fully automated vehicles there will be a huge cliff of ability, and even small OTA update defects could result in large numbers of crashes across a fleet before there is time to correct the problem. (If you push a bad updated to millions of cars, you can have a lot of crashes even in a single day for a defect that affects a common driving situation.)

The industry is going all-in on fast and loose OTAs to be more "agile" and iterate software changes more quickly without worrying as much about quality. But I think they're heading right into a proverbial brick wall that will be hit when the human drivers are taken out of the loop. Getting software quality right will become more important than ever for fully autonomous vehicles.

 

Maturity Levels for Autonomous Vehicle Safety

I've been on a personal journey to understand what safety really means for autonomous vehicles. As part of this I repeatedly find myself in conversations in which participants have wildly different notions of what it means to be "safe."  Here is an attempt to put some structure around the discussion:

An inspiration for this idea is Maslow's famous hierarchy of needs. The idea is that organizations developing autonomous vehicles have to take care of the lower levels before they might be able to afford at higher levels. For example, if your vehicle crashes every 100 meters because it struggles to detect obstacles in ideal conditions, worrying about nuances of lifecycle support won't get you your next funding round.

To succeed as a viable at-scale company, you need to address all the levels in the AV maturity hierarchy. But in reality companies will likely climb the levels like rungs in a ladder. To draw the parallel to Maslow's needs hierarchy, if a company is starving for cash to run its operations, they're going to care more about getting the next demo or funding milestone compared to lifecycle safety considerations. That will only change when venture funding bakes higher levels of this safety maturity hierarchy into their milestones. 

• Basic driving functionality: the vehicle works in a defined environment without hitting any objects or other road users on the defined scale of a funding milestone demo. When people say that their vehicle is safe because it has high crash safety ratings, that aligns with this bin. (I personally prefer my safety to happen without the part where the vehicle crashes.)
• Defensive driving: vehicle has expert driving skills, actively avoiding driving situations that present increased risk. This is analogous to sending a human driver to defensive driving school. At some point the automated driver becomes expert in terms of being able to drive in failure-free situations.
• Systematic hazard analysis: engineering effort has been spent analyzing and mitigating risks not just from driving functions, but also potential technical malfunctions, forced exits from the intended operational design domain, etc.  (For example, HARA from ISO 26262.) Common hazards that aren't easy or inexpensive to mitigate might well be pushed onto the driver (e.g., incomplete redundancy to handle component failure, or required driver intervention to mitigate risks).
• Functional safety: analysis and redundancy have been added, and a principled approach (e.g., based on safety integrity levels) has been taken to ensure risks from technical faults in the system have been mitigated (e.g., ISO 26262 conformance).
• Safety of the Intended Function (SOTIF): ensuring "unknowns" have been addressed, dealing with environmental influences (e.g., not all radar pings will be returned), closing gaps in requirements, and accounting for aspects of machine learning. (e.g., ISO 21448 conformance).
• System level safety: accounting for things beyond just the driving task, including lifecycle considerations. Ensuring that hazard analysis and mitigation extends to process aspects, and a safety case has been used to ensure acceptable safety. (e.g., ANSI/UL 4600)
  Cybersecurity needs to be addressed to achieve system safety, but should not wait to get started until reaching this level.
• Just Safety Culture: operating and continuously improving the organization and execution of other levels of the hierarchy according to Just Culture principles rather than blame.
  Specific common anti-patterns for Just Culture relevant to autonomous vehicles are:
• Using humans as Moral Crumple Zones (blaming humans so as to evade addressing technical or process failures)
• Using propaganda and regulatory lobbying to deflect liability away from companies, reducing pressure to achieve acceptable safety.

As with the Maslow hierarchy the levels are not exclusive. Rather all levels need to operate concurrently, with the highest concurrently active level indicating progress toward safety maturity.

You might see this differently, see some things I've missed, etc. Comments welcome!

Cruise Stopped by Police for Headlights Off -- Why Is This a Big Deal?

In April 2022 San Francisco police pulled over an uncrewed Cruise autonomous test vehicle for not having its headlights on. Much fun was had on social media about the perplexed officer having to chase the car a few meters after it repositioned during the traffic stop. Cruise said it was somehow intentional behavior. They also said their vehicle "did not have its headlights on because of a human error" (Source: https://www.theguardian.com/technology/2022/apr/11/self-driving-car-police-pull-over-san-francisco) 

The traffic behavior indicates that Cruise needs to do better making it easier for local police to do traffic stops -- but that's not the main event. The real issue here is the headlights being off.

Cruise said in a public statement: "we have fixed the issue that led to this." Forgive me if I'm not reassured. A purportedly completely autonomous car (the ones that are supposed to be nearly perfect compared to those oh-so-flawed human drivers) that always drives at night didn't know to turn its headlights on when driving in the dark. Seriously? Remember that headlights on at night is not just for the AV itself (which might not need them in good city lighting with a lidar etc.) but also to be visible to other road users. So headlights off at night is a major safety problem. (Headlights on in day is also a good idea -- at least white forward-facing running lights, which this vehicle also did not seem to have on.)

Cruise says these vehicles are autonomous -- no human driver required. And they only test at night, so needing headlights can hardly be a surprise. But they got it wrong. How can that be? 

This can't just be hand-waved away. Something really fundamental seems broken, and it is just a question of what it is.

The entire AV industry, including Cruise, has a history of extreme secrecy and in particular lack of transparency with safety. So we can only speculate. Here are the possibilities I can think of. None of them inspire confidence in safety, and they tend to get worse as we go down the list.

1. The autonomy software had a defect that didn't turn headlights on at night. Perhaps, but seems unlikely. That would (one assumes) affect the entire fleet. And there should be software checks to make sure the headlights turn on as a safety check built into the software. If true, this never should have slipped through quality control let alone a rigorous safety critical software design process, and indicates significant concerns with overall software safety.
2. The headlight switch is supposed to be on at all times. Many (most?) vehicles have "smart" lights. You can turn them off if you want, but in practice you just turn it to "on" and leave it there for months or years and the headlights just do the right thing, switching from daytime running lights to full on automatically, and turning off when the vehicle does. If you're in urban San Francisco high beams are unlikely to be relevant. So the autonomy doesn't mess with the lights at all. Except -- why does the software not check to see that the vehicle condition is safe in terms of headlights? Seems like a design oversight. How did a hazard like this get missed in hazard analysis? If this is the situation, this really calls into question whether the hazard analysis was done properly for the software. Even if this was fixed, what else did hazard analysis miss?
• 2a) A passenger in the vehicle turned the headlights off as a prank. If this is possible, even more important for this to be called out for software monitoring in the hazard analysis. But the check for headlights off obviously isn't there now. 
3. The software is completely ignorant of headlight state, and there is a maintenance tech who is supposed to turn the lights "on" as part of the check-out process each day to prepare the vehicle to run. This manual headlight-on check didn't get done. This is a huge issue with the Safety Management System, because if they missed that check what other checks did they miss? There are plenty of things to check on these vehicles for safety (e.g., sensor calibration, sensor cleaning/cleaner systems, correct software configuration). If they forget to turn on the headlight switch, what else did they forget? While this might be a "within normal tolerance" deviation, given lack of safety transparency Cruise doesn't get the benefit of the doubt. A broken SMS puts all road users at risk. This is a big deal unless proven otherwise. Firing, training, or having a stern talk with the human who made the "error" won't stop it from happening again. Blaming the driver/pilot/human never really does.
4. There is no SMS. That's basically how Uber ATG ended up killing a pedestrian in Tempe Arizona. If this is the case, it is even scarier.

Again, we don't really know the situation because Cruise is trying the "nothing to see here .. move along" response to this incident. But none of these scenarios is comforting.  If I had to guess my money would be on #3, simply because #4 would be too irresponsible to have to contemplate. But really, we have no way to know what's really going on. And it might be another alternative I have not considered.

Cruise should take this as a wakeup call to get their safety house in order before they have a big event. Blaming safety critical failures on "human error" is generally indicative of a poor safety culture. They have a chance here to turn the ship around -- before there is harm to a road user.

Is there a scenario I missed that is less of a concern? Maybe Cruise will give us a substantive explanation. If they do I'll be happy to post it here as a follow-up.

-----------------------------------------------------------------------------------

Kyle Vogt at Cruise sent me a response on LinkedIn on 4/16/2022:

Kyle Vogt
CEO at Cruise

I’d like to respectfully disagree with your characterization of this event. Allow me to provide some context.

We apply SMS rigorously across the company, which as you probably know includes estimating the safety risk for known hazards and having a process to continuously surface new ones.

We then apply resources accordingly to attempt to reduce the overall safety risk of our service as effectively as possible. This is a continuous process and we will continue even though we’ve passed the point where we can operate without backup drivers.

Rarely would this direct us to focus on things that are extremely unlikely to result in injury, such as lack of headlights on a single vehicle for a short duration of time. We have a process in place to ensure they are functioning properly, but the safety impact of that process failing is low.

Our use of SMS directs the majority of our resources towards higher risk areas like pedestrian and cyclist interactions. These are far more complex and of higher potential severity.

I firmly believe that’s this is the right approach, even if it means our vehicles occasionally do something that seems obviously wrong to a human.

-----------------------------------------------------------------------------------

My response on 4/17/2022:

@Kyle Vogt: Thank you for responding to my post. I am glad to hear that you have a Safety Management System in place now. This is excellent news.
Hopefully your SMS folks are telling you that the headlight-off incident should be treated as a serious safety process issue rather than being sloughed off as a "no big deal." A healthy safety culture would acknowledge this is a process failure that needs to be corrected rather than making excuses based on "safety impact of that process failing is low." Process failures in safety critical system operations are a problem -- period. An attitude that less critical processes don't matter is easily toxic for safety culture. Safety culture is how you do business, and if disregarding procedures that are seen as less than the most critical is how you do business, eventually that will catch up with you resulting in loss events. More simply: discounting a low severity process failure is at odds with your statement that you "apply SMS rigorously across the company."
I would say that headlights off has a substantive potential severity for other road users because they are signaling/warning devices to help people know to get out of the way if your vehicle should malfunction. For example, with headlights off you cannot take controllability credit for the other road user jumping out of your way if you fail to detect them, because they won't see headlights as an indicator of your approach at night. Moreover, having headlights on is the law.
Regardless of your statement that you are focusing on higher severity issues, a process failure for something as simple and obvious as making sure headlights are on at night is very disconcerting, and raises concerns over your safety process quality in general. A more transparent analysis of how that process failed and whether other higher severity process failures are occurring that have not been made public would be the right thing to do here to restore public trust. If this is the only time the headlights were off and your other pre-mission procedures have a high compliance rate, then OK, stuff happens. However -- if pre-mission procedures are hit or miss and you don't even keep track of procedural errors that don't result in a scary pedestrian near-miss, that is quite another. If you don't have the procedural metrics to show you are operating safely, then probably you aren't operating safely. Which is the case?
Your GM-issued VSSA from 2018 seems outdated and has no mention of an SMS. So it is difficult for the public to know what your plan is for safety. More transparency and communication regarding your safety process is essential to help build public trust.

-----------------------------------------------------------------------------------

ANSI/UL 4600 Version 2 (2022)

Version 2 of ANSI/UL 4600 has just been issued. This standard provides guidance on how to ensure that autonomous vehicles safety cases are created and maintained to ensure acceptable safety for deployment.

Since version 1 of the standard was issued in April 2020, the Standards Technical Panel members (the voting committee) and stakeholders have been involved in suggesting clarifications, upgrades, and other improvements as part of the standard's continuous improvement process. 

Version 1 of the standard included chapters on: terminology, safety cases, risk assessment, interaction with humans, autonomy functions, software/system engineering processes, dependability, data/networking, verification/validation/test, tool qualification/COTS/legacy components, lifecycle concerns, maintenance, metrics, and assessment. The standard is designed to work with other safety standards such as ISO 26262 and ISO 21448 to make sure that all the bases are covered for system-level safety on autonomous road vehicles.

Version 2 has some substantive changes:

• The assessment terminology and role of independent assessment have been changed. This is compatible with the previous approach except that the independent assessor has a more substantive role.
• Self-assessment: the development team creates and vets its own safety case.
• Independent assessment: an independent organization examines both the form and the technical substance of the safety case to ensure it is acceptable.  (This independent organization might be within the same company at an arms-length relationship, or could be an external assessor at the company's option.)
• Significant clean-up of the discussion of safety case terminology and structure. This is overall a significant improvement of the ideas that were already there in version 1, but a lot of work went into this area.
• The terminology section has been substantially rewritten to clean up wording and improve alignment with other standards without substantively changing the intended meaning of terms being used.

Version 2 maintains the same structure as Version 1 with some minor changes to top level requirements. Significant attention has been paid to ensure a smooth transition between versions. While much of the document has been updated, the updates beyond the substantive changes tend to be relatively minor in scope and are more in the nature of clarification and adding helpful detail. The changes significantly improve the standard, but the vast majority do not fundamentally change its nature or general requirements.

Overall there were seven different task groups whose members spent many hours contributing to improving the standard, and there are important changes in each area to improve the standard.  Those task groups drafted proposals to modify content regarding: safety cases, faults/hazard/risks, assessment, sensor requirements, object tracking, safe egress, and terminology. 

Everyone who participated deserves a big "Thank You!"  Also, a special thanks to Deborah Prince and Heather Sakellariou at Underwriters Laboratories for coordinating all the activities and keeping everything on track, especially with the challenges presented by the whole process being done electronically via e-mail and on-line meetings. It has truly been a pleasure to work with a group that is so dedicated to this collaborative effort to ensure the safety of autonomous vehicle technology.

The official kickoff of UL 4600 version 3 is planned for April, with the biggest goal being extending the standard to cover special considerations that apply to autonomous heavy trucks. Anyone interested in contributing to that discussion can request stakeholder status (no participation fee required).

You can find a pointer to a copy of the newly released version 2 including FREE (with free digital account) digital access to a full copy of the standard, a list of committee members, and more here: https://users.ece.cmu.edu/~koopman/ul4600/index.html 

The official news release from the standards organization is here: https://ul.org/standards-and-engagement/presenting-standard-safety-evaluation-autonomous-vehicles/second-edition 

Maturation path for safety & security practices

Brief informal notes from a wrap-up quick position statement talk I did at a workshop today.

Both safety and security have a lot in common in terms of how they are maturing over time. Without getting into a religious debate about the difference between them, I note that their trajectory seems to include the following steps, especially for autonomous systems. I'd argue that each step is in a sense more mature than the previous step.

1. Get the system to work. Safety/security can come later.
2. Get the system to work almost all the time. Conflate this with safety/security even though you're still really just getting it to work in the common cases (safety for a vehicle is "doesn't hit stuff" while security is "doesn't get taken down by the usual continuous stream of automated attacks")
3. Brute force problem fixes:   fly/crash/fix/fly (air) and drive/crash/fix/drive (ground)
4. Create a set of best practices in the nature of a building code ("build your system this way")
• Create a useful fiction that you have completely characterized the requirements and operational environment and that your building code will always work.
• Any failure is an embarrassing piece of bad news that violates the fiction of complete understanding.
5. As system matures, complain about false alarm safety/security shutdowns
• It might feel like this means your system has problems, but in fact you're a lot more safe and secure than systems that operate oblivious to their vulnerabilities
6. Start permitting breaking the building code standard rules by arguing that exceptions still result in equivalent safety/security
7. Evolve to full-up deductive assurance cases to argue safety/security beyond building codes
• Still maintain the fiction of complete knowledge of requirements and environment
8. Start operating in more open environments and admit you didn't really understand requirements, nor environment
• Spend a lot of time chasing down problems that reveal defects in your safety case (safety case does not match environmental assumptions, or might not even match deployed system)
9. Switch to an inductive safety case approach:
• Account for risk from epistemic uncertainty (unknown unknowns)
• Instrument system for failure precursors (e.g., safety performance indicators tied to safety case claims)
• Treat incidents as an opportunity to fix problems before there is a loss event.

Law Commission report on Automated Vehicles

Comprehensive effort by Law Commissions (England, Wales, Scotland) on Automated Vehicle regulations. A lot of work, and some valuable insights.

Web site with all documents:   https://www.lawcom.gov.uk/project/automated-vehicles/

Summary of findings: https://s3-eu-west-2.amazonaws.com/lawcom-prod-storage-11jsxou24uy7q/uploads/2022/01/AV-Summary-25-01-22-2.pdf

Full report: https://s3-eu-west-2.amazonaws.com/lawcom-prod-storage-11jsxou24uy7q/uploads/2022/01/AV-Summary-25-01-22-2.pdf

Excerpt from the summary:

"Key recommendations

Throughout this project, we have strived to keep safety at the forefront of our proposals, while also retaining the flexibility required to accommodate future development.

Our recommendations cover initial approval and authorisation of self-driving vehicles, ongoing monitoring of their performance while they are on the road, misleading marketing, and both criminal and civil liability. They include:

• Writing the test for self-driving into law, with a bright line distinguishing it from driver support features, a transparent process for setting a safety standard, and new offences to prevent misleading marketing.
• A two-stage approval and authorisation process building on current international and domestic technical vehicle approval schemes and adding a new second stage to authorise vehicles for use as self-driving on GB roads.
• A new in-use safety assurance scheme to provide regulatory oversight of automated vehicles throughout their lifetimes to ensure they continue to be safe and comply with road rules.
• New legal roles for users, manufacturers and service operators, with removal of criminal responsibility for the person in the passenger seat.
• Holding manufacturers and service operators criminally responsible for misrepresentation or non-disclosure of safety-relevant information."

AV Regulation Bill Hearings: case study materials for regulations, democracy and how the sausage gets made

Kansas has been holding hearings about setting Autonomous Vehicle regulations. The situation provides an interesting, publicly accessible view into what's happening across the US. It also provides an exercise in transparency compared to the process used at about the same time in Pennsylvania.

Note: see update at end for pointers to more comprehensive treatment created after this blog post was written.

A simplified summary of the Kansas situation:

• Walmart is using its lobbying weight to expand its "middle mile" automated delivery trials from Arkansas to also include Kansas. The bill is being considered at their request. But they confess the technology is beyond them.
• Gatik is the Walmart partner doing the heavy lifting here. They tell a plausible story about care and diligence, phased incremental approach, etc., although without mentioning conformance to industry standards. However, regardless of the story they tell, the bill would permit other companies to also operate in the state, so what is in the bill matters beyond Gatik's statements.
• The proposed bill is short (2 pages) and is aimed at narrow permission to do things that look like depot-to-store un-crewed logistics runs on Kansas public roads.
• The bill omits a number of important things pointed out by various parties, so the question is how short can it be while still hitting the important points. It's clear it will be revised with various parties hammering out a revision, and we'll hear more about it later. But for now, the discussion itself is illuminating.

The testimony covers a lot of ground in terms of real issues that need to be resolved. It's worth listening to all three sessions to get a feel for what people care about.  

Kansas AV regulation case study materials:

• The bill itself: http://kslegislature.com/li/b2021_22/measures/sb379/
• Written testimony link
• First day testimony (proponents -- 51 minutes):  https://youtu.be/VH-zSQDp3sk
  
• 00:50: Briefing on bill
• 03:35: Walmart
• 14:50: Gatik government relations presentation
• 24:20: Gatik technical presentation
• 38:20: Q&A
• Note: video corruption at approximately 48:30-49:10 is in original stream
• Second day testimony (neutral/opponents -- 50 minutes): https://youtu.be/zdSwKPDicZQ
  (Times are from start of hearing. Add 1 minute for YouTube stream time)
• 00:20: Briefing on bill
• 01:45: Kansas DOT
• 08:40: Kansas Police organizations
• 19:50: Teamsters/organized labor
• 26:35: League of Kansas Municipalities
• 30:05: Ford Motor Company
• 33:35: Michael DeKort 
• 40:55: Alliance for Automotive Innovation
• 47:15: General Q&A
• Third day testimony (opponents -- 32 minutes): https://youtu.be/jHrkH0oSB5A
  (Times are from start of hearing. Add 3 minutes for YouTube stream time)
• 00:40: Autonomous Vehicle Industry Association
• 07:10: AUVSI
• 10:30: Technet
• 12:45: Trial Lawyers / American Association of Justice
• 17:40: Trial Lawyers/ Hutton & Hutton  (voice only in original stream)
• 24:40: Carnegie Mellon University
• 30:15: General Q&A
• A new bill was introduced in March 2022: SB 546
• Hearing video from March 10, 2022: https://youtu.be/YuAQnZs_Dj8
• 2nd hearing video from March 14, 2022: https://youtu.be/ozOtg4gCajg
• Kansas house hearing on SB 546, March 29, 2022
• Be sure to listen to the exchange at about 2:02:00 about whether time pressure should be a reason to move the bill forward still in need of work.
• Finally a Kansas bill was signed by the governor: KS SB 313

Suggested activities for educators regarding SB 379:

• Warm up by reading a news article on the bill, such as: https://www.repairerdrivennews.com/2022/02/14/kansas-bill-would-bring-in-av-tech-for-middle-mile-goods-transportation/
• All students should watch the proponent first day testimony and read the bill.
• Assign each student to watch one speaker for the second/third day videos and summarize the position of that person in two to five points.
• Re-enact the hearing with each student just giving the three points to boil it down to essentials.
• Who was the speaker advocating for (general public, special interest group, government function, etc.)?
• What do you think that person's testimony contributed to the conversation about the bill within the committee that was different than other speakers?
• Did you find that person credible and effective at advocating for their position? Why?
• Assign five groups within the class, and assign each group one of the five regulatory topic areas listed in this blog post.
• Each group comments on strengths and weaknesses of the bill according to the five topic areas.
• Which of the elements within the assigned group of regulatory issues is addressed by the Walmart/Gatik testimony?
• Pick just one weakness of the bill in your group's assigned area and propose a sentence to add to the bill to fix that one point.
• Discussion:
1. Do you think the bill's scope is just right, too narrow, too broad, or should not be passed at all?
2. Do you think Gatik's deployment will be acceptably safe? Why?
3. Do you think deploying this technology on Kansas roads provides a reasonable tradeoff between economic benefit and jobs to the region vs. risk to other road users and potential issues?
4. How strong a regulatory role should Kansas DOT be given?
5. If you could only change one thing about the bill, what would you change?
6. Look for your speaker/topic in the follow-up bill SB 546. Did your speaker's position on your topic change? Why do you think that happened?

Pennsylvania AV regulation case study materials:

• Compare and contrast the Kansas process with the process used for Pennsylvania SB 965 on Autonomous vehicles a month earlier. For that bill no public hearings were held by the PA Senate Transportation Committee before sending the bill to the full PA Senate:
• Press conference introducing PA SB 965: https://youtu.be/ewULUNDBZO4
• PA SB 965 bill and a proposed amendment: https://www.legis.state.pa.us/cfdocs/billInfo/bill_history.cfm?syear=2021&body=S&type=B&bn=965
• PA Senate Transportation Committee Voting meeting video: https://transportation.pasenategop.com/transportation-012622/
• Pittsburgh Post-Gazette story on concerns: https://www.post-gazette.com/news/transportation/2022/01/26/Pennsylvania-Senate-Transportation-Committee-self-driving-vehicles-emergency-driver-Senate-Bill-965/stories/202201260121
• Podcast on issues raised by PA SB 965: https://ojoyoshidareport.com/podcast-can-av-bill-do-right-by-people/
• In an interesting twist, the PA Senate adopted much of the House language and did a complete revision of PA SB 965 here and passed it to go to the PA Senate (Printer's No. 1839):  https://www.legis.state.pa.us/cfdocs/billInfo/billInfo.cfm?sYear=2021&sInd=0&body=S&type=B&bn=0965
• Compare and contrast the process used by the Pennsylvania House for a companion bill 
• PA HB 2398 Bill drafts: https://legiscan.com/PA/bill/HB2398/2021
• PA HB 2398 Bill public hearings 3/17/2022:  https://s3.us-east-2.amazonaws.com/pagopvideo/179878804.mp4 
• Bill passed by PA House on 6/15/2022: https://legiscan.com/PA/text/HB2398/id/2596376/Pennsylvania-2021-HB2398-Amended.pdf
• A version of the House Bill was passed as Act 130 of 2022: https://www.legis.state.pa.us/cfdocs/legis/li/uconsCheck.cfm?yr=2022&sessInd=0&act=130
• Governor letter acknowledging safety was not ensured by bill: https://www.governor.pa.gov/wp-content/uploads/2022/11/20221103-2398.pdf

Other bills from the 2021-2022 legislative season:

• Oklahoma Senate Bill 1541, which makes the automated driving system a licensed driver in control of the vehicle rather than a nature person:  http://oklegislature.gov/cf_pdf/2021-22%20INT/sb/SB1541%20int.pdf
• This bill is active at a time in which Canoo is setting up shop in Tulsa OK, but from the hearing it sounds more likely that this is a play by Walmart/Gatik to expand operations to stretch from Texas through Oklahoma and into Kansas.
• The OK senate passed the bill without apparent amendment http://www.oklegislature.gov/BillInfo.aspx?Bill=SB1541&Session=2200
• Washington SB 5828 https://app.leg.wa.gov/billsummary?BillNumber=5828&Year=2021
• In general, unwinds recommendations of AV work group that ended up as current law. (Worth reading the analysis document posted there.)
• Washington HB 1731 https://app.leg.wa.gov/billsummary?billnumber=1731&year=2021
  Washington HB 2100 https://app.leg.wa.gov/billsummary?BillNumber=2100&Year=2021
• HB 1731 Enhances requirements for reporting and accountability for AV testing. (Worth reading the analysis document posted there.)
• HB 2100 reduces reporting requirements and increases secrecy of data -- supported by AV industry.
• Hearing video (HB1731 & HB2100): https://tvw.org/video/house-transportation-committee-2022021030/?eventID=2022021030
• New York S8468 https://www.nysenate.gov/legislation/bills/2021/S8468
• Note that municipal preemption would over-ride NYC DOT requirement for testers to conform to SAE J3018 for testing safety.
• West Virginia HB 4787 (passed as a law
• http://www.wvlegislature.gov/Bill_Status/bills_history.cfm?INPUT=4787&year=2022&sessiontype=RS
• Michigan law: https://www.legislature.mi.gov/(S(pkadam2ahg4uxjem1px2kvk1))/mileg.aspx?page=GetObject&objectname=2021-SB-0706
• Washington SB 5594 AV bill 2023:
• Bill: https://app.leg.wa.gov/billsummary?BillNumber=5594&Year=2023&Initiative=False
• WA Senate Transportation Hearing Video: https://tvw.org/video/senate-transportation-2023021157/?eventID=2023021157 (8:20-15:15 and 27:20-1:00:00)

Background materials:

• Autonomous Vehicle Regulation and Trust: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214
• Highly Automated Vehicle Testing & Discrimination Against the Poor: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4016783
• Video: Trust and Governance for Autonomous Vehicle Deployment 2022 https://youtu.be/hZQyFc9ETCE

Updates added ad hoc: incomplete and not in the time capsule

• US House E&C committee, 7/26/2023:
• Blog post: https://safeautonomy.blogspot.com/2023/07/av-safety-claims-and-more-on-my.html
• Washington state, 1/30/2024:
• SB 5594: AVIA bill
• SB5872: requires human test driver
• Testimony: https://app.leg.wa.gov/committeeschedules/#/Senate/438/01-30-2024/01-30-2024/Schedule///Bill/
• South Dakota 2024:
• House Bill 1095: AVIA bill: https://sdlegislature.gov/Session/Bill/24662

Download time capsule for off-line educational use:   https://archive.org/details/2022-02-av-regulation-case-study

Notes:

• My Kansas testimony is at the very end of the last day of the first set of senate hearings. It needs to be clearly stated that my testimony is in the specific context of this bill and the situation on the ground there. If it were a broader situation I'd have a lot more to say per the five topic list link.
• You will notice some people, especially in earlier hearings, wearing masks due to the Covid19 pandemic. (This is obvious at the time I create this page, but in a few years it might not be as obvious what is going on with this.)

The Oklahoma bill has a definition of the driver being the computer, which is both problematic and illustrative of the outstanding state law challenges. I'm documenting it here for reference:

Final OK bill (enrolled): http://webserver1.lsb.state.ok.us/cf_pdf/2021-22%20ENR/SB/SB1541%20ENR.PDF

Quoted starting at bottom of page 7:

1.  The automated driving system is considered the driver or operator, for the purpose of assessing compliance with applicable traffic or motor vehicle laws, and shall be deemed to satisfy electronically all physical acts required by a driver or operator of the vehicle; and

2.  The automated driving system is considered to be licensed to operate the vehicle. 

Florida also says that the computer is the driver:  http://flsenate.gov/Laws/Statutes/2021/316.85 

"(3)(a) For purposes of this chapter, unless the context otherwise requires, the automated driving system, when engaged, shall be deemed to be the operator of an autonomous vehicle, regardless of whether a person is physically present in the vehicle while the vehicle is operating with the automated driving system engaged."

Tesla Rolling Stops -- 10 reasons this is a Big Deal

The Tesla rolling stop discussion continues, with thoughtful people sometimes not sure which way to go on this. Here is a breakdown of the issues...

This is a big deal and should be treated as such:

1. It's illegal. Ignoring the law until you get caught is untrustworthy behavior (in this case apparently part of a pattern). Tesla could have and should have worked this out with NHTSA in advance. Then lobbied for regulatory change if needed.
2. Any rule-bending behavior needs to adjust to local conditions and context to be safe-ish, which wasn't done here. Local customs are relevant. Not doing it in school zones, near playgrounds. Etc.
3. These vehicles are not yet competent drivers. They regularly try to hit vehicles and other things they detect (show as detected on screen -- see YouTube). Any statement that they will proceed only when conditions are clear ignores this severe issue that they clearly are not competent judges of when intersections are clear. (Safety driver being responsible from stopping running over a pedestrian is a ridiculous argument to build in dangerous behavior, but I know it will be made.)
4. Even if you think rolling stops are fine (I disagree), the full stop is still necessary -- for the safety driver to have time to assess safety before going.
5. Top end of 5.6 mph is jogging speed -- more like a yield sign -- it's not a 0.5 mph crawl. If the vehicle needs to creep out into the intersection for sight lines the procedure I remember from my driver training is very clear: first stop completely, then creep as necessary.
6. They are removing agency from the human driver in deciding when to break the law. People selecting a "rolling stop" mode had no idea what they were really signing up for (true behaviors were only disclosed in recall)-- but were going to bear the full burden of any liability for hitting a bicyclist or child at an intersection. That is really wrong. Deciding to break the law on behalf of a driver in an opaque way without involving driver approval every time it is done is inherently a problem.
7. Arguing you'll be safer because you don't make bad human driver choices and then emulating human drivers who bend rules is not a particularly convincing approach to building trust.
8. This differs in kind from crossing a centerline to go around a disabled vehicle. Shaving a few seconds off a trip does not justify breaking traffic laws. There is no pressing no overriding utility of mission completion requirement to break that rule.
9. Saying "people can break the law" is whataboutism. Huge difference between setting a law-breaking policy that sticks basically forever unless changed vs. instance-by-instance human choices to direct a machine to violate speed limit (or whatever) with that human driver being connected to the context the decision is being made in--and consequences. If they wanted to have the ability to do rolling stops they could have just had the driver tap the accelerator as the car slows down and used that as a basis for training for when regulations were relaxed to account for this behavior. (Not condoning breaking traffic laws -- but pointing out that there is an obvious alternative that engaged the safety driver in the process that was not used.)
10. NHTSA has better things to do with its limited resources than play Whac-a-mole with Tesla. Tesla can flout rules faster than NHTSA has resources to enforce them. This appears to be just one incident in a larger campaign of that sort.

Sure, revisiting regulations to codify "do something reasonable" makes sense. I've been pointing this out as a challenge for years. But move fast and break things while unilaterally deciding what you can get away with is not the right approach to safety.

Reference AP article here:  https://www.npr.org/2022/02/01/1077274384/tesla-recalls-autos-over-software-that-allows-them-to-roll-through-stop-signs

NHTSA Part 573 Safety Recall Report: https://static.nhtsa.gov/odi/rcl/2022/RCLRPT-22V037-4462.PDF

Five Principles for Regulation of Highly Automated Vehicles

February 1, 2022

FIVE PRINCIPLES FOR REGULATION OF HIGHLY AUTOMATED VEHICLES

Philip Koopman∗ & William H. Widen**

Providing economic opportunities and jobs is important, but these benefits alone are not an equitable and fair exchange for the use of public highways as a testing ground, and the exposure of the public to an increased risk of harm from highly automated vehicle (HAV) accidents during development. The principles below summarize concrete actions that HAV legislation should include to provide an appropriate balance.

Typeset Acrobat version here: https://archive.org/details/av-regulation-principles-koopman-widen-2022-02-01

1. Safety

Operational Safety: commit to a testing and deployment standard for automated driving system (ADS) performance of “substantially better than the average unimpaired human driver” (AUHD) rather than the vague “sufficiently safe” criteria currently in use by HAV companies.

Metrics: state the metric(s) used to make the performance comparison between an ADS and the AUHD.

Industry Standards: commitment to follow published professional industry standards appropriate for the type of HAV operation (i.e., testing, trials or deployment) including: SAE J3018, UL 4600, ISO 21448 and ISO 26262, as informed by AVSC00001201911 AVSC Best Practice for safety operator selection, training, and oversight procedures for automated vehicles under test and AVSC0007202107 AVSC Information Report for Adapting a Safety Management System (SMS) for Automated Driving System (ADS) SAE Level 4 and 5 Testing and Evaluation.

Regulatory time-out: allow all regulators (federal, state, and local) to temporarily enjoin HAV operations (including testing) as a response to discrete events which raise safety concerns and to revoke testing permits for significant adverse events as well as for patterns of unsafe HAV operations or operations that violate law.

2. Responsibility for Loss (Compensation)

Duty of Care: Any company or person operating an AV should, by statue, owe the public a non-delegable duty of care for safe operation.

Insurance Levels: Use a required insurance amount not less than the US DOT’s statistical value of a human life ($11.6 million per person in 2020, adjusted annually). Do not allow self-insurance by HAV companies which are not cash flow positive. Establish clear single-risk limits and treatment of multiple injuries/fatalities in a single event, with separate treatment for special cases, such as truck platooning which might be expected to cause greater harm on a per incident basis.

Owner/Occupant Liability: Clarify the liability of an owner/occupant of an HAV for loss caused when an ADS is properly engaged. May an injured party sue the owner/occupant in the absence of negligence and collect against the owner’s insurance policy or are claims limited to ADS designers, manufacturers and upfitters for defects? Explain interaction with negligence claims for maintenance, upgrades, and installation failures.

Single Collection Point for Plaintiffs: Allow a plaintiff to collect from a single responsible defendant in full, with joint and several liability and a right of contribution from other responsible parties (regardless of whether the liability is based in negligence or product defect).

Insurance Policy with Plaintiffs in Position of a Named Insured: Allow plaintiffs to make a claim against HAV company insurance policies as if they were a named insured to facilitate prompt payment of medical bills and other amounts so plaintiffs do not face financial pressure to settle for less than full compensation.

Eliminate Barriers to Collection for Full Damages: As applicable, lift caps on damage collection by potential plaintiffs with lower cost insurance policies, such as limited tort option policies in Pennsylvania.

3. Transparency

Periodic reporting: require timely publication of HAV safety performance, including how actual performance of an HAV compares with promised performance relative to the AUHD.

Crash Data: require timely publication of all crash data and police reports for any incident involving a HAV.

Publicize testing, trials, and deployment plans: a requirement to issue publicly a testing plan prior to commencement of testing, trials or deployment which explain how the HAV company addresses safety, responsibility for loss, transparency, inclusion, and non-discrimination. Allow local regulators to review, comment upon, and approve the testing plan.

Identification of times, manner and locations for testing and trials: requirement to post information about the times and locations for testing and trials so the public understands areas of increased risk.

Don’t Promulgate Myths: advocates for an HAV law, rule or regulation, should not use marketing and outreach materials containing untruthful or misleading statements or material omissions, such as the myth that 94% of serious crashes are caused by human error.

Disclose a Harm Now, Benefits Later Justification for Deployment: If an HAV company plans to deploy HAVs at scale when the technology does not perform substantially better than an AUHD, or its level of performance cannot be determined with reasonable certainty, disclose this fact in the testing plan so the public can evaluate a policy which exposes the public to harms on the promise of future improvements to HAV technology.

4. Inclusion

IEEE 7000: require HAV companies to follow the IEEE 7000 Standard Model Process for Addressing Ethical Concerns procedures to identify all interested parties affected by the testing, trials, and deployment of HAV technology (including emergency service responders, hazardous material transporters, and those in a work zone). Address the concerns of all stakeholders in public plans and with respect to testing, trials, and deployments.

Adaptation to Local Conditions: give municipalities the power to limit HAV operations based on time, manner, and location to account for local conditions (such as preventing truck platooning in certain neighborhoods or driverless trials in school zones; account for local special events; account for recent incidents which present safety concerns); remove blanket pre-emption by state law to allow municipalities to exercise this power.

Review Global Approaches to HAV Regulation: as part of approving any bill, review the recommended approaches to HAV regulation taken in other jurisdictions, such as the joint report of the UK law commissions on automated vehicles, and the EU’s ethics guidelines for the development of trustworthy artificial intelligence.

5. Non-Discrimination:

No Concentrated Operations in Areas of Concern: Municipal review of both public testing and trial plans and the time, manner and locations for testing and trials to ensure that at-risk communities, such as low-income neighborhoods, do not experience a disproportionate increased risk of loss from HAV operations (without mitigating the identified risk) as opposed to other communities not deemed to be at special risk.

Mitigation Strategies: If a need arises to concentrate certain types of testing and trials in an at-risk community, use other means to mitigate the adverse impact of the concentration—such as conducting testing only with an on-board safety driver, use of two safety drivers; or elimination of uncrewed trials.

Special Review of Related Laws: review applicable existing state and local laws to identify situations in which HAV operations (including crashes) might adversely impact low-income and other at-risk persons, such as limited damage collection for low-cost policies, and the difficulty and expense of pursuing a product defect claim given legal complexity and number of possible defendants.

Justice40: situate and harmonize non-discrimination efforts within the broader framework of social justice and equity values, including those contained in the Federal Government’s Justice40 initiative.