OTA updates won't save buggy autonomous vehicle software

There is a feeling that it's OK for software to ship with questionable quality if you have the ability to send out updates quickly. You might be able to get away for this with human-driven vehicles, but for autonomous vehicles (no human driver responsible for safety) this strategy might collapse.

Right now, companies are all pushing hard to do quick-turn Over The Air (OTA) software updates, with Tesla being the poster child of both shipping dodgy software and pushing out quick updates (not all of which actually solve the problem as intended). There is a moral hazard that comes with the ability to do quick OTAs in that you might not spend much time on quality since you know you can just send another update if the first one doesn't turn out as you hoped.

"There's definitely the mindset that you can fix fast so you can take a higher risk," Florian Rohde, a former Tesla validation manager   (https://www.reuters.com/article/tesla-recalls-idTRNIKBN2KN171)

For now companies across an increasing number of industries have been getting away with shipping lower quality software, and the ability to do internet-based updates has let them get by with such a strategy. The practice is so prevalent that the typical trouble-shooting for any software after "is the power turned on" has become "have you downloaded the latest updates."

But the reason this approach works (after a fashion) is that there is a human user or vehicle operator present to recognize something is wrong, work around the problem, and participate in the trouble shooting. In a fully automated vehicle, that human isn't going to be there to save the day.

What happens when there is no human to counter-act the defective and potentially fatally dangerous defective software behavior? The biggest weakness of any automation is typically that it is not "smart" enough to know when something is going wrong that is not supposed to happen. People are pretty good at this, which is why even for very serious software defects in cars we often see huge numbers of complaints compared to few actual instances of harm -- because human drivers have compensated for the bad software behavior.

Here's a concrete example of a surprising software defect pulled from my extensive list of problematic automotive software defects: NHTSA Recall 14V-204:

• Due to software calibration error vehicle may be in and display "drive" but engage "reverse" for 1.5 seconds.

If a human driver notices the vehicle going the wrong direction they'll stop accelerating pretty quickly. They might hit something at slow speed during the reaction time, but they'll realize something is wrong without having explicit instructions for that particular failure scenario. In contract, a computer-based system that has been taught the car always moves in the direction of the transmission display might not even realize something is wrong and accelerate into collisions.

Obviously a computer can be programmed to deal with such a situation if it has been thought of at design time. But the whole point here is that this is something that isn't supposed to happen -- so why would you waste time programming a computer to handle an "impossible" event? Safety engineering deals with hazard analysis to mitigate low risk things, but even that often overlooks "impossible" events until after they've occurred. Sure, you can send an OTA update after the crash -- but that doesn't bring crash victims back to life.

In practice the justification that it is OK to ship out less-than-perfect automotive software has been that human drivers can compensate for problems. (In the ISO 26262 functional safety standard one takes credit for "controllability" in reducing the risk of a potential defect.) When there is no human driver, that's a problem, and shipping defective software is more likely to result in harm to a vehicle occupant or other road user before it can be noticed there is a problem for OTA to correct.

Right now, a significant challenge to OTA updates is the moral hazard that software will be a bit more dangerous than it should be due to pushing the boundaries of human driver ability to compensate for defects. With fully automated vehicles there will be a huge cliff of ability, and even small OTA update defects could result in large numbers of crashes across a fleet before there is time to correct the problem. (If you push a bad updated to millions of cars, you can have a lot of crashes even in a single day for a defect that affects a common driving situation.)

The industry is going all-in on fast and loose OTAs to be more "agile" and iterate software changes more quickly without worrying as much about quality. But I think they're heading right into a proverbial brick wall that will be hit when the human drivers are taken out of the loop. Getting software quality right will become more important than ever for fully autonomous vehicles.

 

Maturity Levels for Autonomous Vehicle Safety

I've been on a personal journey to understand what safety really means for autonomous vehicles. As part of this I repeatedly find myself in conversations in which participants have wildly different notions of what it means to be "safe."  Here is an attempt to put some structure around the discussion:

An inspiration for this idea is Maslow's famous hierarchy of needs. The idea is that organizations developing autonomous vehicles have to take care of the lower levels before they might be able to afford at higher levels. For example, if your vehicle crashes every 100 meters because it struggles to detect obstacles in ideal conditions, worrying about nuances of lifecycle support won't get you your next funding round.

To succeed as a viable at-scale company, you need to address all the levels in the AV maturity hierarchy. But in reality companies will likely climb the levels like rungs in a ladder. To draw the parallel to Maslow's needs hierarchy, if a company is starving for cash to run its operations, they're going to care more about getting the next demo or funding milestone compared to lifecycle safety considerations. That will only change when venture funding bakes higher levels of this safety maturity hierarchy into their milestones. 

• Basic driving functionality: the vehicle works in a defined environment without hitting any objects or other road users on the defined scale of a funding milestone demo. When people say that their vehicle is safe because it has high crash safety ratings, that aligns with this bin. (I personally prefer my safety to happen without the part where the vehicle crashes.)
• Defensive driving: vehicle has expert driving skills, actively avoiding driving situations that present increased risk. This is analogous to sending a human driver to defensive driving school. At some point the automated driver becomes expert in terms of being able to drive in failure-free situations.
• Systematic hazard analysis: engineering effort has been spent analyzing and mitigating risks not just from driving functions, but also potential technical malfunctions, forced exits from the intended operational design domain, etc.  (For example, HARA from ISO 26262.) Common hazards that aren't easy or inexpensive to mitigate might well be pushed onto the driver (e.g., incomplete redundancy to handle component failure, or required driver intervention to mitigate risks).
• Functional safety: analysis and redundancy have been added, and a principled approach (e.g., based on safety integrity levels) has been taken to ensure risks from technical faults in the system have been mitigated (e.g., ISO 26262 conformance).
• Safety of the Intended Function (SOTIF): ensuring "unknowns" have been addressed, dealing with environmental influences (e.g., not all radar pings will be returned), closing gaps in requirements, and accounting for aspects of machine learning. (e.g., ISO 21448 conformance).
• System level safety: accounting for things beyond just the driving task, including lifecycle considerations. Ensuring that hazard analysis and mitigation extends to process aspects, and a safety case has been used to ensure acceptable safety. (e.g., ANSI/UL 4600)
  Cybersecurity needs to be addressed to achieve system safety, but should not wait to get started until reaching this level.
• Just Safety Culture: operating and continuously improving the organization and execution of other levels of the hierarchy according to Just Culture principles rather than blame.
  Specific common anti-patterns for Just Culture relevant to autonomous vehicles are:
• Using humans as Moral Crumple Zones (blaming humans so as to evade addressing technical or process failures)
• Using propaganda and regulatory lobbying to deflect liability away from companies, reducing pressure to achieve acceptable safety.

As with the Maslow hierarchy the levels are not exclusive. Rather all levels need to operate concurrently, with the highest concurrently active level indicating progress toward safety maturity.

You might see this differently, see some things I've missed, etc. Comments welcome!

Cruise Stopped by Police for Headlights Off -- Why Is This a Big Deal?

In April 2022 San Francisco police pulled over an uncrewed Cruise autonomous test vehicle for not having its headlights on. Much fun was had on social media about the perplexed officer having to chase the car a few meters after it repositioned during the traffic stop. Cruise said it was somehow intentional behavior. They also said their vehicle "did not have its headlights on because of a human error" (Source: https://www.theguardian.com/technology/2022/apr/11/self-driving-car-police-pull-over-san-francisco) 

The traffic behavior indicates that Cruise needs to do better making it easier for local police to do traffic stops -- but that's not the main event. The real issue here is the headlights being off.

Cruise said in a public statement: "we have fixed the issue that led to this." Forgive me if I'm not reassured. A purportedly completely autonomous car (the ones that are supposed to be nearly perfect compared to those oh-so-flawed human drivers) that always drives at night didn't know to turn its headlights on when driving in the dark. Seriously? Remember that headlights on at night is not just for the AV itself (which might not need them in good city lighting with a lidar etc.) but also to be visible to other road users. So headlights off at night is a major safety problem. (Headlights on in day is also a good idea -- at least white forward-facing running lights, which this vehicle also did not seem to have on.)

Cruise says these vehicles are autonomous -- no human driver required. And they only test at night, so needing headlights can hardly be a surprise. But they got it wrong. How can that be? 

This can't just be hand-waved away. Something really fundamental seems broken, and it is just a question of what it is.

The entire AV industry, including Cruise, has a history of extreme secrecy and in particular lack of transparency with safety. So we can only speculate. Here are the possibilities I can think of. None of them inspire confidence in safety, and they tend to get worse as we go down the list.

1. The autonomy software had a defect that didn't turn headlights on at night. Perhaps, but seems unlikely. That would (one assumes) affect the entire fleet. And there should be software checks to make sure the headlights turn on as a safety check built into the software. If true, this never should have slipped through quality control let alone a rigorous safety critical software design process, and indicates significant concerns with overall software safety.
2. The headlight switch is supposed to be on at all times. Many (most?) vehicles have "smart" lights. You can turn them off if you want, but in practice you just turn it to "on" and leave it there for months or years and the headlights just do the right thing, switching from daytime running lights to full on automatically, and turning off when the vehicle does. If you're in urban San Francisco high beams are unlikely to be relevant. So the autonomy doesn't mess with the lights at all. Except -- why does the software not check to see that the vehicle condition is safe in terms of headlights? Seems like a design oversight. How did a hazard like this get missed in hazard analysis? If this is the situation, this really calls into question whether the hazard analysis was done properly for the software. Even if this was fixed, what else did hazard analysis miss?
• 2a) A passenger in the vehicle turned the headlights off as a prank. If this is possible, even more important for this to be called out for software monitoring in the hazard analysis. But the check for headlights off obviously isn't there now. 
3. The software is completely ignorant of headlight state, and there is a maintenance tech who is supposed to turn the lights "on" as part of the check-out process each day to prepare the vehicle to run. This manual headlight-on check didn't get done. This is a huge issue with the Safety Management System, because if they missed that check what other checks did they miss? There are plenty of things to check on these vehicles for safety (e.g., sensor calibration, sensor cleaning/cleaner systems, correct software configuration). If they forget to turn on the headlight switch, what else did they forget? While this might be a "within normal tolerance" deviation, given lack of safety transparency Cruise doesn't get the benefit of the doubt. A broken SMS puts all road users at risk. This is a big deal unless proven otherwise. Firing, training, or having a stern talk with the human who made the "error" won't stop it from happening again. Blaming the driver/pilot/human never really does.
4. There is no SMS. That's basically how Uber ATG ended up killing a pedestrian in Tempe Arizona. If this is the case, it is even scarier.

Again, we don't really know the situation because Cruise is trying the "nothing to see here .. move along" response to this incident. But none of these scenarios is comforting.  If I had to guess my money would be on #3, simply because #4 would be too irresponsible to have to contemplate. But really, we have no way to know what's really going on. And it might be another alternative I have not considered.

Cruise should take this as a wakeup call to get their safety house in order before they have a big event. Blaming safety critical failures on "human error" is generally indicative of a poor safety culture. They have a chance here to turn the ship around -- before there is harm to a road user.

Is there a scenario I missed that is less of a concern? Maybe Cruise will give us a substantive explanation. If they do I'll be happy to post it here as a follow-up.

-----------------------------------------------------------------------------------

Kyle Vogt at Cruise sent me a response on LinkedIn on 4/16/2022:

Kyle Vogt
CEO at Cruise

I’d like to respectfully disagree with your characterization of this event. Allow me to provide some context.

We apply SMS rigorously across the company, which as you probably know includes estimating the safety risk for known hazards and having a process to continuously surface new ones.

We then apply resources accordingly to attempt to reduce the overall safety risk of our service as effectively as possible. This is a continuous process and we will continue even though we’ve passed the point where we can operate without backup drivers.

Rarely would this direct us to focus on things that are extremely unlikely to result in injury, such as lack of headlights on a single vehicle for a short duration of time. We have a process in place to ensure they are functioning properly, but the safety impact of that process failing is low.

Our use of SMS directs the majority of our resources towards higher risk areas like pedestrian and cyclist interactions. These are far more complex and of higher potential severity.

I firmly believe that’s this is the right approach, even if it means our vehicles occasionally do something that seems obviously wrong to a human.

-----------------------------------------------------------------------------------

My response on 4/17/2022:

@Kyle Vogt: Thank you for responding to my post. I am glad to hear that you have a Safety Management System in place now. This is excellent news.
Hopefully your SMS folks are telling you that the headlight-off incident should be treated as a serious safety process issue rather than being sloughed off as a "no big deal." A healthy safety culture would acknowledge this is a process failure that needs to be corrected rather than making excuses based on "safety impact of that process failing is low." Process failures in safety critical system operations are a problem -- period. An attitude that less critical processes don't matter is easily toxic for safety culture. Safety culture is how you do business, and if disregarding procedures that are seen as less than the most critical is how you do business, eventually that will catch up with you resulting in loss events. More simply: discounting a low severity process failure is at odds with your statement that you "apply SMS rigorously across the company."
I would say that headlights off has a substantive potential severity for other road users because they are signaling/warning devices to help people know to get out of the way if your vehicle should malfunction. For example, with headlights off you cannot take controllability credit for the other road user jumping out of your way if you fail to detect them, because they won't see headlights as an indicator of your approach at night. Moreover, having headlights on is the law.
Regardless of your statement that you are focusing on higher severity issues, a process failure for something as simple and obvious as making sure headlights are on at night is very disconcerting, and raises concerns over your safety process quality in general. A more transparent analysis of how that process failed and whether other higher severity process failures are occurring that have not been made public would be the right thing to do here to restore public trust. If this is the only time the headlights were off and your other pre-mission procedures have a high compliance rate, then OK, stuff happens. However -- if pre-mission procedures are hit or miss and you don't even keep track of procedural errors that don't result in a scary pedestrian near-miss, that is quite another. If you don't have the procedural metrics to show you are operating safely, then probably you aren't operating safely. Which is the case?
Your GM-issued VSSA from 2018 seems outdated and has no mention of an SMS. So it is difficult for the public to know what your plan is for safety. More transparency and communication regarding your safety process is essential to help build public trust.

-----------------------------------------------------------------------------------

ANSI/UL 4600 Version 2 (2022)

Version 2 of ANSI/UL 4600 has just been issued. This standard provides guidance on how to ensure that autonomous vehicles safety cases are created and maintained to ensure acceptable safety for deployment.

Since version 1 of the standard was issued in April 2020, the Standards Technical Panel members (the voting committee) and stakeholders have been involved in suggesting clarifications, upgrades, and other improvements as part of the standard's continuous improvement process. 

Version 1 of the standard included chapters on: terminology, safety cases, risk assessment, interaction with humans, autonomy functions, software/system engineering processes, dependability, data/networking, verification/validation/test, tool qualification/COTS/legacy components, lifecycle concerns, maintenance, metrics, and assessment. The standard is designed to work with other safety standards such as ISO 26262 and ISO 21448 to make sure that all the bases are covered for system-level safety on autonomous road vehicles.

Version 2 has some substantive changes:

• The assessment terminology and role of independent assessment have been changed. This is compatible with the previous approach except that the independent assessor has a more substantive role.
• Self-assessment: the development team creates and vets its own safety case.
• Independent assessment: an independent organization examines both the form and the technical substance of the safety case to ensure it is acceptable.  (This independent organization might be within the same company at an arms-length relationship, or could be an external assessor at the company's option.)
• Significant clean-up of the discussion of safety case terminology and structure. This is overall a significant improvement of the ideas that were already there in version 1, but a lot of work went into this area.
• The terminology section has been substantially rewritten to clean up wording and improve alignment with other standards without substantively changing the intended meaning of terms being used.

Version 2 maintains the same structure as Version 1 with some minor changes to top level requirements. Significant attention has been paid to ensure a smooth transition between versions. While much of the document has been updated, the updates beyond the substantive changes tend to be relatively minor in scope and are more in the nature of clarification and adding helpful detail. The changes significantly improve the standard, but the vast majority do not fundamentally change its nature or general requirements.

Overall there were seven different task groups whose members spent many hours contributing to improving the standard, and there are important changes in each area to improve the standard.  Those task groups drafted proposals to modify content regarding: safety cases, faults/hazard/risks, assessment, sensor requirements, object tracking, safe egress, and terminology. 

Everyone who participated deserves a big "Thank You!"  Also, a special thanks to Deborah Prince and Heather Sakellariou at Underwriters Laboratories for coordinating all the activities and keeping everything on track, especially with the challenges presented by the whole process being done electronically via e-mail and on-line meetings. It has truly been a pleasure to work with a group that is so dedicated to this collaborative effort to ensure the safety of autonomous vehicle technology.

The official kickoff of UL 4600 version 3 is planned for April, with the biggest goal being extending the standard to cover special considerations that apply to autonomous heavy trucks. Anyone interested in contributing to that discussion can request stakeholder status (no participation fee required).

You can find a pointer to a copy of the newly released version 2 including FREE (with free digital account) digital access to a full copy of the standard, a list of committee members, and more here: https://users.ece.cmu.edu/~koopman/ul4600/index.html 

The official news release from the standards organization is here: https://ul.org/standards-and-engagement/presenting-standard-safety-evaluation-autonomous-vehicles/second-edition 

Maturation path for safety & security practices

Brief informal notes from a wrap-up quick position statement talk I did at a workshop today.

Both safety and security have a lot in common in terms of how they are maturing over time. Without getting into a religious debate about the difference between them, I note that their trajectory seems to include the following steps, especially for autonomous systems. I'd argue that each step is in a sense more mature than the previous step.

1. Get the system to work. Safety/security can come later.
2. Get the system to work almost all the time. Conflate this with safety/security even though you're still really just getting it to work in the common cases (safety for a vehicle is "doesn't hit stuff" while security is "doesn't get taken down by the usual continuous stream of automated attacks")
3. Brute force problem fixes:   fly/crash/fix/fly (air) and drive/crash/fix/drive (ground)
4. Create a set of best practices in the nature of a building code ("build your system this way")
• Create a useful fiction that you have completely characterized the requirements and operational environment and that your building code will always work.
• Any failure is an embarrassing piece of bad news that violates the fiction of complete understanding.
5. As system matures, complain about false alarm safety/security shutdowns
• It might feel like this means your system has problems, but in fact you're a lot more safe and secure than systems that operate oblivious to their vulnerabilities
6. Start permitting breaking the building code standard rules by arguing that exceptions still result in equivalent safety/security
7. Evolve to full-up deductive assurance cases to argue safety/security beyond building codes
• Still maintain the fiction of complete knowledge of requirements and environment
8. Start operating in more open environments and admit you didn't really understand requirements, nor environment
• Spend a lot of time chasing down problems that reveal defects in your safety case (safety case does not match environmental assumptions, or might not even match deployed system)
9. Switch to an inductive safety case approach:
• Account for risk from epistemic uncertainty (unknown unknowns)
• Instrument system for failure precursors (e.g., safety performance indicators tied to safety case claims)
• Treat incidents as an opportunity to fix problems before there is a loss event.

Law Commission report on Automated Vehicles

Comprehensive effort by Law Commissions (England, Wales, Scotland) on Automated Vehicle regulations. A lot of work, and some valuable insights.

Web site with all documents:   https://www.lawcom.gov.uk/project/automated-vehicles/

Summary of findings: https://s3-eu-west-2.amazonaws.com/lawcom-prod-storage-11jsxou24uy7q/uploads/2022/01/AV-Summary-25-01-22-2.pdf

Full report: https://s3-eu-west-2.amazonaws.com/lawcom-prod-storage-11jsxou24uy7q/uploads/2022/01/AV-Summary-25-01-22-2.pdf

Excerpt from the summary:

"Key recommendations

Throughout this project, we have strived to keep safety at the forefront of our proposals, while also retaining the flexibility required to accommodate future development.

Our recommendations cover initial approval and authorisation of self-driving vehicles, ongoing monitoring of their performance while they are on the road, misleading marketing, and both criminal and civil liability. They include:

• Writing the test for self-driving into law, with a bright line distinguishing it from driver support features, a transparent process for setting a safety standard, and new offences to prevent misleading marketing.
• A two-stage approval and authorisation process building on current international and domestic technical vehicle approval schemes and adding a new second stage to authorise vehicles for use as self-driving on GB roads.
• A new in-use safety assurance scheme to provide regulatory oversight of automated vehicles throughout their lifetimes to ensure they continue to be safe and comply with road rules.
• New legal roles for users, manufacturers and service operators, with removal of criminal responsibility for the person in the passenger seat.
• Holding manufacturers and service operators criminally responsible for misrepresentation or non-disclosure of safety-relevant information."

AV Regulation Bill Hearings: case study materials for regulations, democracy and how the sausage gets made

Kansas has been holding hearings about setting Autonomous Vehicle regulations. The situation provides an interesting, publicly accessible view into what's happening across the US. It also provides an exercise in transparency compared to the process used at about the same time in Pennsylvania.

Note: see update at end for pointers to more comprehensive treatment created after this blog post was written.

A simplified summary of the Kansas situation:

• Walmart is using its lobbying weight to expand its "middle mile" automated delivery trials from Arkansas to also include Kansas. The bill is being considered at their request. But they confess the technology is beyond them.
• Gatik is the Walmart partner doing the heavy lifting here. They tell a plausible story about care and diligence, phased incremental approach, etc., although without mentioning conformance to industry standards. However, regardless of the story they tell, the bill would permit other companies to also operate in the state, so what is in the bill matters beyond Gatik's statements.
• The proposed bill is short (2 pages) and is aimed at narrow permission to do things that look like depot-to-store un-crewed logistics runs on Kansas public roads.
• The bill omits a number of important things pointed out by various parties, so the question is how short can it be while still hitting the important points. It's clear it will be revised with various parties hammering out a revision, and we'll hear more about it later. But for now, the discussion itself is illuminating.

The testimony covers a lot of ground in terms of real issues that need to be resolved. It's worth listening to all three sessions to get a feel for what people care about.  

Kansas AV regulation case study materials:

• The bill itself: http://kslegislature.com/li/b2021_22/measures/sb379/
• Written testimony link
• First day testimony (proponents -- 51 minutes):  https://youtu.be/VH-zSQDp3sk
  
• 00:50: Briefing on bill
• 03:35: Walmart
• 14:50: Gatik government relations presentation
• 24:20: Gatik technical presentation
• 38:20: Q&A
• Note: video corruption at approximately 48:30-49:10 is in original stream
• Second day testimony (neutral/opponents -- 50 minutes): https://youtu.be/zdSwKPDicZQ
  (Times are from start of hearing. Add 1 minute for YouTube stream time)
• 00:20: Briefing on bill
• 01:45: Kansas DOT
• 08:40: Kansas Police organizations
• 19:50: Teamsters/organized labor
• 26:35: League of Kansas Municipalities
• 30:05: Ford Motor Company
• 33:35: Michael DeKort 
• 40:55: Alliance for Automotive Innovation
• 47:15: General Q&A
• Third day testimony (opponents -- 32 minutes): https://youtu.be/jHrkH0oSB5A
  (Times are from start of hearing. Add 3 minutes for YouTube stream time)
• 00:40: Autonomous Vehicle Industry Association
• 07:10: AUVSI
• 10:30: Technet
• 12:45: Trial Lawyers / American Association of Justice
• 17:40: Trial Lawyers/ Hutton & Hutton  (voice only in original stream)
• 24:40: Carnegie Mellon University
• 30:15: General Q&A
• A new bill was introduced in March 2022: SB 546
• Hearing video from March 10, 2022: https://youtu.be/YuAQnZs_Dj8
• 2nd hearing video from March 14, 2022: https://youtu.be/ozOtg4gCajg
• Kansas house hearing on SB 546, March 29, 2022
• Be sure to listen to the exchange at about 2:02:00 about whether time pressure should be a reason to move the bill forward still in need of work.
• Finally a Kansas bill was signed by the governor: KS SB 313

Suggested activities for educators regarding SB 379:

• Warm up by reading a news article on the bill, such as: https://www.repairerdrivennews.com/2022/02/14/kansas-bill-would-bring-in-av-tech-for-middle-mile-goods-transportation/
• All students should watch the proponent first day testimony and read the bill.
• Assign each student to watch one speaker for the second/third day videos and summarize the position of that person in two to five points.
• Re-enact the hearing with each student just giving the three points to boil it down to essentials.
• Who was the speaker advocating for (general public, special interest group, government function, etc.)?
• What do you think that person's testimony contributed to the conversation about the bill within the committee that was different than other speakers?
• Did you find that person credible and effective at advocating for their position? Why?
• Assign five groups within the class, and assign each group one of the five regulatory topic areas listed in this blog post.
• Each group comments on strengths and weaknesses of the bill according to the five topic areas.
• Which of the elements within the assigned group of regulatory issues is addressed by the Walmart/Gatik testimony?
• Pick just one weakness of the bill in your group's assigned area and propose a sentence to add to the bill to fix that one point.
• Discussion:
1. Do you think the bill's scope is just right, too narrow, too broad, or should not be passed at all?
2. Do you think Gatik's deployment will be acceptably safe? Why?
3. Do you think deploying this technology on Kansas roads provides a reasonable tradeoff between economic benefit and jobs to the region vs. risk to other road users and potential issues?
4. How strong a regulatory role should Kansas DOT be given?
5. If you could only change one thing about the bill, what would you change?
6. Look for your speaker/topic in the follow-up bill SB 546. Did your speaker's position on your topic change? Why do you think that happened?

Pennsylvania AV regulation case study materials:

• Compare and contrast the Kansas process with the process used for Pennsylvania SB 965 on Autonomous vehicles a month earlier. For that bill no public hearings were held by the PA Senate Transportation Committee before sending the bill to the full PA Senate:
• Press conference introducing PA SB 965: https://youtu.be/ewULUNDBZO4
• PA SB 965 bill and a proposed amendment: https://www.legis.state.pa.us/cfdocs/billInfo/bill_history.cfm?syear=2021&body=S&type=B&bn=965
• PA Senate Transportation Committee Voting meeting video: https://transportation.pasenategop.com/transportation-012622/
• Pittsburgh Post-Gazette story on concerns: https://www.post-gazette.com/news/transportation/2022/01/26/Pennsylvania-Senate-Transportation-Committee-self-driving-vehicles-emergency-driver-Senate-Bill-965/stories/202201260121
• Podcast on issues raised by PA SB 965: https://ojoyoshidareport.com/podcast-can-av-bill-do-right-by-people/
• In an interesting twist, the PA Senate adopted much of the House language and did a complete revision of PA SB 965 here and passed it to go to the PA Senate (Printer's No. 1839):  https://www.legis.state.pa.us/cfdocs/billInfo/billInfo.cfm?sYear=2021&sInd=0&body=S&type=B&bn=0965
• Compare and contrast the process used by the Pennsylvania House for a companion bill 
• PA HB 2398 Bill drafts: https://legiscan.com/PA/bill/HB2398/2021
• PA HB 2398 Bill public hearings 3/17/2022:  https://s3.us-east-2.amazonaws.com/pagopvideo/179878804.mp4 
• Bill passed by PA House on 6/15/2022: https://legiscan.com/PA/text/HB2398/id/2596376/Pennsylvania-2021-HB2398-Amended.pdf
• A version of the House Bill was passed as Act 130 of 2022: https://www.legis.state.pa.us/cfdocs/legis/li/uconsCheck.cfm?yr=2022&sessInd=0&act=130
• Governor letter acknowledging safety was not ensured by bill: https://www.governor.pa.gov/wp-content/uploads/2022/11/20221103-2398.pdf

Other bills from the 2021-2022 legislative season:

• Oklahoma Senate Bill 1541, which makes the automated driving system a licensed driver in control of the vehicle rather than a nature person:  http://oklegislature.gov/cf_pdf/2021-22%20INT/sb/SB1541%20int.pdf
• This bill is active at a time in which Canoo is setting up shop in Tulsa OK, but from the hearing it sounds more likely that this is a play by Walmart/Gatik to expand operations to stretch from Texas through Oklahoma and into Kansas.
• The OK senate passed the bill without apparent amendment http://www.oklegislature.gov/BillInfo.aspx?Bill=SB1541&Session=2200
• Washington SB 5828 https://app.leg.wa.gov/billsummary?BillNumber=5828&Year=2021
• In general, unwinds recommendations of AV work group that ended up as current law. (Worth reading the analysis document posted there.)
• Washington HB 1731 https://app.leg.wa.gov/billsummary?billnumber=1731&year=2021
  Washington HB 2100 https://app.leg.wa.gov/billsummary?BillNumber=2100&Year=2021
• HB 1731 Enhances requirements for reporting and accountability for AV testing. (Worth reading the analysis document posted there.)
• HB 2100 reduces reporting requirements and increases secrecy of data -- supported by AV industry.
• Hearing video (HB1731 & HB2100): https://tvw.org/video/house-transportation-committee-2022021030/?eventID=2022021030
• New York S8468 https://www.nysenate.gov/legislation/bills/2021/S8468
• Note that municipal preemption would over-ride NYC DOT requirement for testers to conform to SAE J3018 for testing safety.
• West Virginia HB 4787 (passed as a law
• http://www.wvlegislature.gov/Bill_Status/bills_history.cfm?INPUT=4787&year=2022&sessiontype=RS
• Michigan law: https://www.legislature.mi.gov/(S(pkadam2ahg4uxjem1px2kvk1))/mileg.aspx?page=GetObject&objectname=2021-SB-0706
• Washington SB 5594 AV bill 2023:
• Bill: https://app.leg.wa.gov/billsummary?BillNumber=5594&Year=2023&Initiative=False
• WA Senate Transportation Hearing Video: https://tvw.org/video/senate-transportation-2023021157/?eventID=2023021157 (8:20-15:15 and 27:20-1:00:00)

Background materials:

• Autonomous Vehicle Regulation and Trust: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214
• Highly Automated Vehicle Testing & Discrimination Against the Poor: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4016783
• Video: Trust and Governance for Autonomous Vehicle Deployment 2022 https://youtu.be/hZQyFc9ETCE

Updates added ad hoc: incomplete and not in the time capsule

• US House E&C committee, 7/26/2023:
• Blog post: https://safeautonomy.blogspot.com/2023/07/av-safety-claims-and-more-on-my.html
• Washington state, 1/30/2024:
• SB 5594: AVIA bill
• SB5872: requires human test driver
• Testimony: https://app.leg.wa.gov/committeeschedules/#/Senate/438/01-30-2024/01-30-2024/Schedule///Bill/
• South Dakota 2024:
• House Bill 1095: AVIA bill: https://sdlegislature.gov/Session/Bill/24662

Download time capsule for off-line educational use:   https://archive.org/details/2022-02-av-regulation-case-study

Notes:

• My Kansas testimony is at the very end of the last day of the first set of senate hearings. It needs to be clearly stated that my testimony is in the specific context of this bill and the situation on the ground there. If it were a broader situation I'd have a lot more to say per the five topic list link.
• You will notice some people, especially in earlier hearings, wearing masks due to the Covid19 pandemic. (This is obvious at the time I create this page, but in a few years it might not be as obvious what is going on with this.)

The Oklahoma bill has a definition of the driver being the computer, which is both problematic and illustrative of the outstanding state law challenges. I'm documenting it here for reference:

Final OK bill (enrolled): http://webserver1.lsb.state.ok.us/cf_pdf/2021-22%20ENR/SB/SB1541%20ENR.PDF

Quoted starting at bottom of page 7:

1.  The automated driving system is considered the driver or operator, for the purpose of assessing compliance with applicable traffic or motor vehicle laws, and shall be deemed to satisfy electronically all physical acts required by a driver or operator of the vehicle; and

2.  The automated driving system is considered to be licensed to operate the vehicle. 

Florida also says that the computer is the driver:  http://flsenate.gov/Laws/Statutes/2021/316.85 

"(3)(a) For purposes of this chapter, unless the context otherwise requires, the automated driving system, when engaged, shall be deemed to be the operator of an autonomous vehicle, regardless of whether a person is physically present in the vehicle while the vehicle is operating with the automated driving system engaged."

Tesla Rolling Stops -- 10 reasons this is a Big Deal

The Tesla rolling stop discussion continues, with thoughtful people sometimes not sure which way to go on this. Here is a breakdown of the issues...

This is a big deal and should be treated as such:

1. It's illegal. Ignoring the law until you get caught is untrustworthy behavior (in this case apparently part of a pattern). Tesla could have and should have worked this out with NHTSA in advance. Then lobbied for regulatory change if needed.
2. Any rule-bending behavior needs to adjust to local conditions and context to be safe-ish, which wasn't done here. Local customs are relevant. Not doing it in school zones, near playgrounds. Etc.
3. These vehicles are not yet competent drivers. They regularly try to hit vehicles and other things they detect (show as detected on screen -- see YouTube). Any statement that they will proceed only when conditions are clear ignores this severe issue that they clearly are not competent judges of when intersections are clear. (Safety driver being responsible from stopping running over a pedestrian is a ridiculous argument to build in dangerous behavior, but I know it will be made.)
4. Even if you think rolling stops are fine (I disagree), the full stop is still necessary -- for the safety driver to have time to assess safety before going.
5. Top end of 5.6 mph is jogging speed -- more like a yield sign -- it's not a 0.5 mph crawl. If the vehicle needs to creep out into the intersection for sight lines the procedure I remember from my driver training is very clear: first stop completely, then creep as necessary.
6. They are removing agency from the human driver in deciding when to break the law. People selecting a "rolling stop" mode had no idea what they were really signing up for (true behaviors were only disclosed in recall)-- but were going to bear the full burden of any liability for hitting a bicyclist or child at an intersection. That is really wrong. Deciding to break the law on behalf of a driver in an opaque way without involving driver approval every time it is done is inherently a problem.
7. Arguing you'll be safer because you don't make bad human driver choices and then emulating human drivers who bend rules is not a particularly convincing approach to building trust.
8. This differs in kind from crossing a centerline to go around a disabled vehicle. Shaving a few seconds off a trip does not justify breaking traffic laws. There is no pressing no overriding utility of mission completion requirement to break that rule.
9. Saying "people can break the law" is whataboutism. Huge difference between setting a law-breaking policy that sticks basically forever unless changed vs. instance-by-instance human choices to direct a machine to violate speed limit (or whatever) with that human driver being connected to the context the decision is being made in--and consequences. If they wanted to have the ability to do rolling stops they could have just had the driver tap the accelerator as the car slows down and used that as a basis for training for when regulations were relaxed to account for this behavior. (Not condoning breaking traffic laws -- but pointing out that there is an obvious alternative that engaged the safety driver in the process that was not used.)
10. NHTSA has better things to do with its limited resources than play Whac-a-mole with Tesla. Tesla can flout rules faster than NHTSA has resources to enforce them. This appears to be just one incident in a larger campaign of that sort.

Sure, revisiting regulations to codify "do something reasonable" makes sense. I've been pointing this out as a challenge for years. But move fast and break things while unilaterally deciding what you can get away with is not the right approach to safety.

Reference AP article here:  https://www.npr.org/2022/02/01/1077274384/tesla-recalls-autos-over-software-that-allows-them-to-roll-through-stop-signs

NHTSA Part 573 Safety Recall Report: https://static.nhtsa.gov/odi/rcl/2022/RCLRPT-22V037-4462.PDF

Five Principles for Regulation of Highly Automated Vehicles

February 1, 2022

FIVE PRINCIPLES FOR REGULATION OF HIGHLY AUTOMATED VEHICLES

Philip Koopman∗ & William H. Widen**

Providing economic opportunities and jobs is important, but these benefits alone are not an equitable and fair exchange for the use of public highways as a testing ground, and the exposure of the public to an increased risk of harm from highly automated vehicle (HAV) accidents during development. The principles below summarize concrete actions that HAV legislation should include to provide an appropriate balance.

Typeset Acrobat version here: https://archive.org/details/av-regulation-principles-koopman-widen-2022-02-01

1. Safety

Operational Safety: commit to a testing and deployment standard for automated driving system (ADS) performance of “substantially better than the average unimpaired human driver” (AUHD) rather than the vague “sufficiently safe” criteria currently in use by HAV companies.

Metrics: state the metric(s) used to make the performance comparison between an ADS and the AUHD.

Industry Standards: commitment to follow published professional industry standards appropriate for the type of HAV operation (i.e., testing, trials or deployment) including: SAE J3018, UL 4600, ISO 21448 and ISO 26262, as informed by AVSC00001201911 AVSC Best Practice for safety operator selection, training, and oversight procedures for automated vehicles under test and AVSC0007202107 AVSC Information Report for Adapting a Safety Management System (SMS) for Automated Driving System (ADS) SAE Level 4 and 5 Testing and Evaluation.

Regulatory time-out: allow all regulators (federal, state, and local) to temporarily enjoin HAV operations (including testing) as a response to discrete events which raise safety concerns and to revoke testing permits for significant adverse events as well as for patterns of unsafe HAV operations or operations that violate law.

2. Responsibility for Loss (Compensation)

Duty of Care: Any company or person operating an AV should, by statue, owe the public a non-delegable duty of care for safe operation.

Insurance Levels: Use a required insurance amount not less than the US DOT’s statistical value of a human life ($11.6 million per person in 2020, adjusted annually). Do not allow self-insurance by HAV companies which are not cash flow positive. Establish clear single-risk limits and treatment of multiple injuries/fatalities in a single event, with separate treatment for special cases, such as truck platooning which might be expected to cause greater harm on a per incident basis.

Owner/Occupant Liability: Clarify the liability of an owner/occupant of an HAV for loss caused when an ADS is properly engaged. May an injured party sue the owner/occupant in the absence of negligence and collect against the owner’s insurance policy or are claims limited to ADS designers, manufacturers and upfitters for defects? Explain interaction with negligence claims for maintenance, upgrades, and installation failures.

Single Collection Point for Plaintiffs: Allow a plaintiff to collect from a single responsible defendant in full, with joint and several liability and a right of contribution from other responsible parties (regardless of whether the liability is based in negligence or product defect).

Insurance Policy with Plaintiffs in Position of a Named Insured: Allow plaintiffs to make a claim against HAV company insurance policies as if they were a named insured to facilitate prompt payment of medical bills and other amounts so plaintiffs do not face financial pressure to settle for less than full compensation.

Eliminate Barriers to Collection for Full Damages: As applicable, lift caps on damage collection by potential plaintiffs with lower cost insurance policies, such as limited tort option policies in Pennsylvania.

3. Transparency

Periodic reporting: require timely publication of HAV safety performance, including how actual performance of an HAV compares with promised performance relative to the AUHD.

Crash Data: require timely publication of all crash data and police reports for any incident involving a HAV.

Publicize testing, trials, and deployment plans: a requirement to issue publicly a testing plan prior to commencement of testing, trials or deployment which explain how the HAV company addresses safety, responsibility for loss, transparency, inclusion, and non-discrimination. Allow local regulators to review, comment upon, and approve the testing plan.

Identification of times, manner and locations for testing and trials: requirement to post information about the times and locations for testing and trials so the public understands areas of increased risk.

Don’t Promulgate Myths: advocates for an HAV law, rule or regulation, should not use marketing and outreach materials containing untruthful or misleading statements or material omissions, such as the myth that 94% of serious crashes are caused by human error.

Disclose a Harm Now, Benefits Later Justification for Deployment: If an HAV company plans to deploy HAVs at scale when the technology does not perform substantially better than an AUHD, or its level of performance cannot be determined with reasonable certainty, disclose this fact in the testing plan so the public can evaluate a policy which exposes the public to harms on the promise of future improvements to HAV technology.

4. Inclusion

IEEE 7000: require HAV companies to follow the IEEE 7000 Standard Model Process for Addressing Ethical Concerns procedures to identify all interested parties affected by the testing, trials, and deployment of HAV technology (including emergency service responders, hazardous material transporters, and those in a work zone). Address the concerns of all stakeholders in public plans and with respect to testing, trials, and deployments.

Adaptation to Local Conditions: give municipalities the power to limit HAV operations based on time, manner, and location to account for local conditions (such as preventing truck platooning in certain neighborhoods or driverless trials in school zones; account for local special events; account for recent incidents which present safety concerns); remove blanket pre-emption by state law to allow municipalities to exercise this power.

Review Global Approaches to HAV Regulation: as part of approving any bill, review the recommended approaches to HAV regulation taken in other jurisdictions, such as the joint report of the UK law commissions on automated vehicles, and the EU’s ethics guidelines for the development of trustworthy artificial intelligence.

5. Non-Discrimination:

No Concentrated Operations in Areas of Concern: Municipal review of both public testing and trial plans and the time, manner and locations for testing and trials to ensure that at-risk communities, such as low-income neighborhoods, do not experience a disproportionate increased risk of loss from HAV operations (without mitigating the identified risk) as opposed to other communities not deemed to be at special risk.

Mitigation Strategies: If a need arises to concentrate certain types of testing and trials in an at-risk community, use other means to mitigate the adverse impact of the concentration—such as conducting testing only with an on-board safety driver, use of two safety drivers; or elimination of uncrewed trials.

Special Review of Related Laws: review applicable existing state and local laws to identify situations in which HAV operations (including crashes) might adversely impact low-income and other at-risk persons, such as limited damage collection for low-cost policies, and the difficulty and expense of pursuing a product defect claim given legal complexity and number of possible defendants.

Justice40: situate and harmonize non-discrimination efforts within the broader framework of social justice and equity values, including those contained in the Federal Government’s Justice40 initiative.

Simplified Proposal for Vehicle Automation Modes

 Vehicle Automation Modes emphasize the responsibilities of a self-driving vehicle user

By: Dr. Philip Koopman, Carnegie Mellon University

Now that the AV industry has backed away from SAE Levels, especially the highly problematic Level 3, it's time for a fresh look at operating modes of vehicle automation technology.

Vehicle Automation Modes

If you follow self-driving car technology it’s likely you’ve encountered the SAE Levels of automation. The SAE Levels range from 0 to 5, with higher numbers indicating driving automation technology with more control authority (but not a linear progression, and not necessarily higher levels of safety). Unfortunately, in public discussions there is significant confusion and misuse (even abuse) of that terminology. In large part that is because the SAE Levels are primarily based on an engineering view rather than the perspective of a person driving the car.

We need a different categorization approach. One that emphasizes how drivers and organizations will deploy these vehicles rather than the underlying technology. Such an approach needs to emphasize the practical aspects of the driver’s role in vehicle operation.

If you doubt that another set of terminology is needed, consider the common informal use of the term “Level 2+,” which is undefined by the underlying SAE J3016 standard that sets the SAE Levels. Consider also the fact that different companies mean significantly different things when they say “Level 3.” In some cases Level 3 follows SAE J3016, meaning that the driver is responsible for monitoring vehicle operation and being ready to jump in — even without any notice at all — to take over if something goes wrong. In other cases vehicles described as Level 3 are expected to safely bring themselves to a stop even if the driver does not notice a problem, which is more like a “Level 3+” concept (also undefined by SAE J3016).

Even more importantly, the SAE Levels say nothing about all the safety relevant tasks that a human driver does beyond actual driving. For example, someone has to make sure that the kids are buckled into their car seats. To actually deploy such vehicles, we need to cover the whole picture, in which driving is critical but only a piece of the safety puzzle.

With the recent apparent removal of support for the SAE J3016 level system by the Autonomous Vehicle Industry Association, the time is ripe for revisiting how we talk about the different operational modes for vehicle automation. 

We start with the premise that for practical purposes all new vehicles will have some sort of active safety system such as Automated Emergency Braking (AEB) and so skip a category specifically for vehicles with no driver assistance. (One could use a "No Assistance" mode if desired, but it adds unnecessary clutter for most purposes.) We also include a distinct category for testing to help close the SAE Level 2 Loophole which let companies test immature technology without regulatory oversight simply by (improperly) claiming the presence of a safety driver makes an autonomous driving feature testbed SAE Level 2. There is no mapping to the SAE Levels, because that would import baggage that could compromise safety.

The Four Operational Modes

In creating a driver-centric description of capabilities, the most important thing is not the details of the technology, but rather what role and responsibility the driver is assigned in overall vehicle operation. We propose four categories of vehicle operation: 

• Driver Assistance
• Supervised Automation
• Autonomous Operation
• Vehicle Testing.

Driver Assistance:

Driver Assistance: A licensed human driver drives, and the vehicle assists.

• Human Role: Licensed driver performs driving task
• Vehicle Role: Active Safety, Driver Support, Driving Convenience

The technology’s job is to help the driver do better by improving the vehicle’s ability to execute the driver’s commands and try to mitigate potential harm from some types of impending crashes. Convenience features might also be provided, excluding sustained automated steering.

Capabilities included as driver assistance might include anti-lock brakes, stability control, cruise control, adaptive cruise control, and automatic emergency braking. The driver always remains in the steering loop, exerting at least some form of sustained control over lane keeping and turns to ensure active engagement and situational awareness. 

Momentary intervention by active safety and driver support functions in the steering function such as a steering wheel bump at lane boundaries is considered driver support rather than steering automation. Active safety might momentarily intervene in steering in response to a specific situation but should not permit itself to be used in lieu of continuous driver control of steering. Completely automated speed control is permitted (e.g., adaptive cruise control).

Supervised Automation

Supervised Automation: The vehicle controls speed and lane keeping. A human driver handles things the system is not designed to address.

• Human Role: Licensed driver keeps eyes on road, monitors for and intervenes in situations vehicle is not designed to handle, executes turns and other tasks beyond ordinary lane-keeping.
• Vehicle Role: Provides steady cruise functions of lane-keeping and speed control.

Technology normally provides a speed and lane-keeping "cruise" capability when feature is activated. A licensed human driver is responsible for continuous monitoring of driving and intervening when a situation is encountered beyond the design scope of the system. Human driver is responsible for  situations outside the stated design capabilities of the system. The design capabilities exclude turning at intersections and other scenarios beyond traversing the current roadway. Automation might not be capable of handling situations outside its stated capability, which the driver is aware of and accounts for in supervision. Driver is able to take over full control whenever appropriate.

An effective driver monitoring system is required to ensure driver remains situationally aware and is capable of taking over when required for safety. This does not have to mean hands on the wheel. Keeping hands on the wheel might be required for testing, and might be required in vehicles that do not have camera-based driver monitoring systems to ensure driver engagement. But the requirement for Supervised Automation is simply that the driver must be able to respond when needed, and it is up to the feature developer to determine how to accomplish that in an effective manner. In practice with current technology this is likely to mean a camera-based Driver Monitoring System (DMS).

Supervised automation should make it reasonable to expect a civilian driver without specialized training to achieve at least as good a safety record as would be the case without steering automation given comparable other vehicle capabilities and operational conditions.  This means that any such vehicle that is not as safe as a human driver (including not only crashes, but violating traffic laws or exhibiting reckless driving at an elevated rate) should be considered to have a defective design. The scope of design relevant to safety is not only the car, but also the human/driver interface.

As a practical matter, this limits use to highway and straight road-following cruise-control style applications where the vehicle does both lane keeping and speed/separation control. If the vehicle can make turns at intersections, with current technology it is beyond what is reasonably safe for civilian driver supervision, and instead is likely to be a road test vehicle. (This paragraph might be considered controversial. However it is the author's best estimate of what is feasible for safe road use by the full demographic of drivers on public roads, assuming an effective DMS can be deployed.)

Autonomous Operation

Autonomous Operation: The whole vehicle is completely capable of operation with no human monitoring.

• Human Role: No Human Driver; steering wheel optional depending on operational concept
• Vehicle Role: Responsible for all aspects of driving and driving-related safety.

The vehicle can complete an entire driving mission under normal circumstances without human supervision. If the operational design domain (ODD) is restricted, the vehicle is responsible for safely handling any exit from the ODD that might occur. 

If something goes wrong, the vehicle is entirely responsible for alerting humans that it needs assistance, and for operating safely until that assistance is available. Things that might go wrong include not only encountering unforeseen situations and technology failures, but also flat tires, a battery fire, being hit by another vehicle, or all of these things at once. People in the vehicle, if there are any, might not be licensed drivers, and might not be capable of assuming the role of “captain of the ship."

Examples of Autonomous vehicles might include uncrewed robo-taxis, driverless last mile delivery vehicles, and heavy trucks in which the driver is permitted to be asleep. A vehicle that received remote assistance would still be exhibiting Autonomous Operation if (a) the vehicle requests assistance whenever needed without any person being responsible for noticing there is a problem, and (b) the vehicle retains responsibility for safety even with assistance.  In some cases autonomous operation might change mode to remotely supervised operation if a remote operator becomes responsible for safety.

Achieving safety will depend on the autonomous vehicle being able to handle everything that comes its way, for example according to the UL 4600 safety standard with additional conformance to ISO 26262 and ISO 21448.

Vehicle Testing

Vehicle Testing: A trained safety driver supervises the operation of an automation testing platform.

• Human Role: Trained safety driver performs mitigates dangerous behaviors, and at times might perform driving.
• Vehicle Role: Automation being tested is expected to exhibit dangerous behaviors.

The vehicle is a test bed for vehicle automation features. Because it is immature technology, the driver must have specialized training and operating procedures to ensure public safety, for example according to the SAE J3018 road testing operator safety standard in accordance with a suitable Safety Management System (SMS).

Any vehicle which might exhibit dangerous behavior beyond the mitigation capability of an ordinary licensed driver encompassing the full driver demographic span, or that requires special qualification and care due to potentially dangerous behavior is an automation test platform. Anyone operating such a test platform is performing Vehicle Testing. (Alternately, such a platform is a defective Supervised Automation platform which should not be operating on public roads.) 

Driver Liability:

An advantage of this classification approach is that it provides a straightforward way to address driver liability.

• Driver Assistance: As with conventional vehicles.
• Supervised Automation: Absent vehicle defects, the driver is responsible for safe operation. Vehicle defects are activated when the automation does not perform as described to the driver, including incorrect responses to scenarios said to be handled automatically and also failure to respond to a situation the driver has been told is covered automatically. As an example, a vehicle that suddenly swerves into oncoming traffic while performing lane keeping is likely to have defective automation in the absence of other over-riding considerations.
• Autonomous Operation: The vehicle automation is responsible for safety.
• Vehicle Testing: The organization performing testing is responsible for safety in accordance with a Safety Management System that includes driver qualification, driver training, and testing protocols.

Other Considerations:

A single vehicle can operate in multiple modes during a single trip. For example a single trip can start in Driver Assistance mode on local roads, switch to Supervised Automation on a limited access highway, and then switch to Autonomous Operation on a designated portion of roads (federal highway, urban downtown, parking garage) as is compatible with its design restrictions.

All modes must have provisions for mitigating risk from foreseeable misuse and abuse. That includes ensuring operation of modes within their intended restrictions (e.g., enforcing the J3016 concept of an Operational Design Domain (ODD)).

Mode changes must be done safely. The principle should be that a human driver can take control in a situation for which that can be safely done, but a human driver can never be forced to assume control involuntarily. This implies, for example, that in Autonomous Operation the vehicle must safety stop in a reasonable location if it is unable to continue a mission without demanding human driver takeover. (A human driver, if present, might elect to assume control, but takeover cannot be required to ensure safety.)

Automation must make a best effort to ensure the highest level of safety it is capable of even without human intervention, but nonetheless is not responsible beyond best effort for dealing with aspects of vehicle and control beyond its currently active mode. The one exception is Vehicle Testing mode, which because it involves immature technology cannot be counted on to provide any automation function beyond a high integrity mechanism for the human test driver to assert vehicle control.

Mode confusion is a critical issue with system safety. There must be an effective scheme for ensuring that any driver is aware of the current vehicle mode. Changes in modes must also be safe. Mode changes should not be permitted without unambiguous determination that any human driver that might be involved has shifted their mental model of current mode to match the actual vehicle mode in effect after the transition and is capable of fulfilling the expected human driver mode for that role.

Graphic:  https://archive.org/details/2022-01-31-automation-modes

For an earlier version of this approach with more detail relevant to regulators, see Section V of this paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214

Updated 1/31/2022