Simplified Proposal for Vehicle Automation Modes

 Vehicle Automation Modes emphasize the responsibilities of a self-driving vehicle user

By: Dr. Philip Koopman, Carnegie Mellon University

Now that the AV industry has backed away from SAE Levels, especially the highly problematic Level 3, it's time for a fresh look at operating modes of vehicle automation technology.

Vehicle Automation Modes

If you follow self-driving car technology it’s likely you’ve encountered the SAE Levels of automation. The SAE Levels range from 0 to 5, with higher numbers indicating driving automation technology with more control authority (but not a linear progression, and not necessarily higher levels of safety). Unfortunately, in public discussions there is significant confusion and misuse (even abuse) of that terminology. In large part that is because the SAE Levels are primarily based on an engineering view rather than the perspective of a person driving the car.

We need a different categorization approach. One that emphasizes how drivers and organizations will deploy these vehicles rather than the underlying technology. Such an approach needs to emphasize the practical aspects of the driver’s role in vehicle operation.

If you doubt that another set of terminology is needed, consider the common informal use of the term “Level 2+,” which is undefined by the underlying SAE J3016 standard that sets the SAE Levels. Consider also the fact that different companies mean significantly different things when they say “Level 3.” In some cases Level 3 follows SAE J3016, meaning that the driver is responsible for monitoring vehicle operation and being ready to jump in — even without any notice at all — to take over if something goes wrong. In other cases vehicles described as Level 3 are expected to safely bring themselves to a stop even if the driver does not notice a problem, which is more like a “Level 3+” concept (also undefined by SAE J3016).

Even more importantly, the SAE Levels say nothing about all the safety relevant tasks that a human driver does beyond actual driving. For example, someone has to make sure that the kids are buckled into their car seats. To actually deploy such vehicles, we need to cover the whole picture, in which driving is critical but only a piece of the safety puzzle.

With the recent apparent removal of support for the SAE J3016 level system by the Autonomous Vehicle Industry Association, the time is ripe for revisiting how we talk about the different operational modes for vehicle automation. 

We start with the premise that for practical purposes all new vehicles will have some sort of active safety system such as Automated Emergency Braking (AEB) and so skip a category specifically for vehicles with no driver assistance. (One could use a "No Assistance" mode if desired, but it adds unnecessary clutter for most purposes.) We also include a distinct category for testing to help close the SAE Level 2 Loophole which let companies test immature technology without regulatory oversight simply by (improperly) claiming the presence of a safety driver makes an autonomous driving feature testbed SAE Level 2. There is no mapping to the SAE Levels, because that would import baggage that could compromise safety.

The Four Operational Modes

In creating a driver-centric description of capabilities, the most important thing is not the details of the technology, but rather what role and responsibility the driver is assigned in overall vehicle operation. We propose four categories of vehicle operation: 

• Driver Assistance
• Supervised Automation
• Autonomous Operation
• Vehicle Testing.

Driver Assistance:

Driver Assistance: A licensed human driver drives, and the vehicle assists.

• Human Role: Licensed driver performs driving task
• Vehicle Role: Active Safety, Driver Support, Driving Convenience

The technology’s job is to help the driver do better by improving the vehicle’s ability to execute the driver’s commands and try to mitigate potential harm from some types of impending crashes. Convenience features might also be provided, excluding sustained automated steering.

Capabilities included as driver assistance might include anti-lock brakes, stability control, cruise control, adaptive cruise control, and automatic emergency braking. The driver always remains in the steering loop, exerting at least some form of sustained control over lane keeping and turns to ensure active engagement and situational awareness. 

Momentary intervention by active safety and driver support functions in the steering function such as a steering wheel bump at lane boundaries is considered driver support rather than steering automation. Active safety might momentarily intervene in steering in response to a specific situation but should not permit itself to be used in lieu of continuous driver control of steering. Completely automated speed control is permitted (e.g., adaptive cruise control).

Supervised Automation

Supervised Automation: The vehicle controls speed and lane keeping. A human driver handles things the system is not designed to address.

• Human Role: Licensed driver keeps eyes on road, monitors for and intervenes in situations vehicle is not designed to handle, executes turns and other tasks beyond ordinary lane-keeping.
• Vehicle Role: Provides steady cruise functions of lane-keeping and speed control.

Technology normally provides a speed and lane-keeping "cruise" capability when feature is activated. A licensed human driver is responsible for continuous monitoring of driving and intervening when a situation is encountered beyond the design scope of the system. Human driver is responsible for  situations outside the stated design capabilities of the system. The design capabilities exclude turning at intersections and other scenarios beyond traversing the current roadway. Automation might not be capable of handling situations outside its stated capability, which the driver is aware of and accounts for in supervision. Driver is able to take over full control whenever appropriate.

An effective driver monitoring system is required to ensure driver remains situationally aware and is capable of taking over when required for safety. This does not have to mean hands on the wheel. Keeping hands on the wheel might be required for testing, and might be required in vehicles that do not have camera-based driver monitoring systems to ensure driver engagement. But the requirement for Supervised Automation is simply that the driver must be able to respond when needed, and it is up to the feature developer to determine how to accomplish that in an effective manner. In practice with current technology this is likely to mean a camera-based Driver Monitoring System (DMS).

Supervised automation should make it reasonable to expect a civilian driver without specialized training to achieve at least as good a safety record as would be the case without steering automation given comparable other vehicle capabilities and operational conditions.  This means that any such vehicle that is not as safe as a human driver (including not only crashes, but violating traffic laws or exhibiting reckless driving at an elevated rate) should be considered to have a defective design. The scope of design relevant to safety is not only the car, but also the human/driver interface.

As a practical matter, this limits use to highway and straight road-following cruise-control style applications where the vehicle does both lane keeping and speed/separation control. If the vehicle can make turns at intersections, with current technology it is beyond what is reasonably safe for civilian driver supervision, and instead is likely to be a road test vehicle. (This paragraph might be considered controversial. However it is the author's best estimate of what is feasible for safe road use by the full demographic of drivers on public roads, assuming an effective DMS can be deployed.)

Autonomous Operation

Autonomous Operation: The whole vehicle is completely capable of operation with no human monitoring.

• Human Role: No Human Driver; steering wheel optional depending on operational concept
• Vehicle Role: Responsible for all aspects of driving and driving-related safety.

The vehicle can complete an entire driving mission under normal circumstances without human supervision. If the operational design domain (ODD) is restricted, the vehicle is responsible for safely handling any exit from the ODD that might occur. 

If something goes wrong, the vehicle is entirely responsible for alerting humans that it needs assistance, and for operating safely until that assistance is available. Things that might go wrong include not only encountering unforeseen situations and technology failures, but also flat tires, a battery fire, being hit by another vehicle, or all of these things at once. People in the vehicle, if there are any, might not be licensed drivers, and might not be capable of assuming the role of “captain of the ship."

Examples of Autonomous vehicles might include uncrewed robo-taxis, driverless last mile delivery vehicles, and heavy trucks in which the driver is permitted to be asleep. A vehicle that received remote assistance would still be exhibiting Autonomous Operation if (a) the vehicle requests assistance whenever needed without any person being responsible for noticing there is a problem, and (b) the vehicle retains responsibility for safety even with assistance.  In some cases autonomous operation might change mode to remotely supervised operation if a remote operator becomes responsible for safety.

Achieving safety will depend on the autonomous vehicle being able to handle everything that comes its way, for example according to the UL 4600 safety standard with additional conformance to ISO 26262 and ISO 21448.

Vehicle Testing

Vehicle Testing: A trained safety driver supervises the operation of an automation testing platform.

• Human Role: Trained safety driver performs mitigates dangerous behaviors, and at times might perform driving.
• Vehicle Role: Automation being tested is expected to exhibit dangerous behaviors.

The vehicle is a test bed for vehicle automation features. Because it is immature technology, the driver must have specialized training and operating procedures to ensure public safety, for example according to the SAE J3018 road testing operator safety standard in accordance with a suitable Safety Management System (SMS).

Any vehicle which might exhibit dangerous behavior beyond the mitigation capability of an ordinary licensed driver encompassing the full driver demographic span, or that requires special qualification and care due to potentially dangerous behavior is an automation test platform. Anyone operating such a test platform is performing Vehicle Testing. (Alternately, such a platform is a defective Supervised Automation platform which should not be operating on public roads.) 

Driver Liability:

An advantage of this classification approach is that it provides a straightforward way to address driver liability.

• Driver Assistance: As with conventional vehicles.
• Supervised Automation: Absent vehicle defects, the driver is responsible for safe operation. Vehicle defects are activated when the automation does not perform as described to the driver, including incorrect responses to scenarios said to be handled automatically and also failure to respond to a situation the driver has been told is covered automatically. As an example, a vehicle that suddenly swerves into oncoming traffic while performing lane keeping is likely to have defective automation in the absence of other over-riding considerations.
• Autonomous Operation: The vehicle automation is responsible for safety.
• Vehicle Testing: The organization performing testing is responsible for safety in accordance with a Safety Management System that includes driver qualification, driver training, and testing protocols.

Other Considerations:

A single vehicle can operate in multiple modes during a single trip. For example a single trip can start in Driver Assistance mode on local roads, switch to Supervised Automation on a limited access highway, and then switch to Autonomous Operation on a designated portion of roads (federal highway, urban downtown, parking garage) as is compatible with its design restrictions.

All modes must have provisions for mitigating risk from foreseeable misuse and abuse. That includes ensuring operation of modes within their intended restrictions (e.g., enforcing the J3016 concept of an Operational Design Domain (ODD)).

Mode changes must be done safely. The principle should be that a human driver can take control in a situation for which that can be safely done, but a human driver can never be forced to assume control involuntarily. This implies, for example, that in Autonomous Operation the vehicle must safety stop in a reasonable location if it is unable to continue a mission without demanding human driver takeover. (A human driver, if present, might elect to assume control, but takeover cannot be required to ensure safety.)

Automation must make a best effort to ensure the highest level of safety it is capable of even without human intervention, but nonetheless is not responsible beyond best effort for dealing with aspects of vehicle and control beyond its currently active mode. The one exception is Vehicle Testing mode, which because it involves immature technology cannot be counted on to provide any automation function beyond a high integrity mechanism for the human test driver to assert vehicle control.

Mode confusion is a critical issue with system safety. There must be an effective scheme for ensuring that any driver is aware of the current vehicle mode. Changes in modes must also be safe. Mode changes should not be permitted without unambiguous determination that any human driver that might be involved has shifted their mental model of current mode to match the actual vehicle mode in effect after the transition and is capable of fulfilling the expected human driver mode for that role.

Graphic:  https://archive.org/details/2022-01-31-automation-modes

For an earlier version of this approach with more detail relevant to regulators, see Section V of this paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214

Updated 1/31/2022

Trust & Governance for Autonomous Vehicle Deployment / Keynote talk

Speaker: Professor Philip Koopman, Carnegie Mellon University

Video (33 minutes):   YouTube | Archive.org
Slides: pdf format

For a related pager with a lot more detail, see: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214

Abstract:

Governance of who decides when and where to test and deploy autonomous vehicle (AV) technology--and on what basis--is a pressing problem. At the moment, an essentially opaque governance model is being run by the AV industry in the US. With few exceptions, companies do not follow industry safety standards, and are not required to. Public sentiment has soured to a degree as timelines have stretched and high profile crashes with injuries and deaths attributed to the technology mount. The industry frequently states that they cannot succeed without public trust, and indeed trust will be essential to riding through future adverse news cycles that will inevitably arrive.

Nonetheless, the industry persists in practices that erode trust, including safety opacity, promoting misleading public talking points, adversarial relations with regulators, some reckless road testing practices, and over-hyped promises of a road safety utopia being just beyond an ever-receding horizon. Hopefully the industry will recognize the cognitive dissonance of their current approach and shift to a collaborative governance model to build trust before the current situation catches up with them.

Speaker information:

Professor Philip Koopman is an internationally recognized expert on Autonomous Vehicle (AV) safety whose work in that area spans over 25 years. He is also actively involved with AV policy and standards as well as more general embedded system design and software quality. His pioneering research work includes software robustness testing and run time monitoring of autonomous systems to identify how they break and how to fix them. He has extensive experience in software safety and software quality across numerous transportation, industrial, and defense application domains including conventional automotive software and hardware systems. He was the principal technical contributor to the UL 4600 standard for autonomous system safety issued in 2020. He is a faculty member of the Carnegie Mellon University ECE department where he teaches software skills for mission-critical systems. In 2018 he was awarded the highly selective IEEE-SSIT Carl Barus Award for outstanding service in the public interest for his work in promoting automotive computer-based system safety.

Comments on PA SB-965 Regulating Autonomous Vehicles

This posting lists a number of major issues and concerns with the Pennsylvania legislation introduced in January 2022 to change how the Commonwealth regulates Highly Automated Vehicles (HAV). It deals with technical issues with the bill and especially aspects of safety. The short version is that this bill should NOT be passed without SIGNIFICANT changes.

• Pittsburgh Post-Gazette Op Ed covering J3018 and a brief mention of preemption: Autonomous vehicle safety bill needs improvement
• Pittsburgh Post-Gazette article covering bill, including City of Pittsburgh opposition to preemption.
• The bill: PA SB 965
• The press conference by bill supporters (Jan 5, 2022)
• User Guide to SAE J3016
• "SSRN paper" is Widen & Koopman, Autonomous Vehicle Regulation and Trust, SSRN, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214
• A newer paper by Prof. Widen points out a slew of legal issues with the proposed bill, including discrimination against the poor: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4016783

• News article: AV company's point of view 
• Emphasizes: economic opportunity, jobs. Some talk about safety but no substance (just as there is no safety substance in the bill).
• "incredibly collaborative" -- but collaborators limited to AV companies, bill sponsors, PennDOT -- not inclusive of Pittsburgh city government, safety advocates, consumer advocates stakeholders

Green text has been added or modified since initial post responsive to comments.

As of afternoon of 1/26/2022 the bill has been amended and committee sent to PA Senate. This writeup has not yet been updated to reflect any substantive changes.

OVERVIEW:

Overall, this bill suffers from a significant imbalance regarding the tradeoff between risks and benefits to Pennsylvania residents. Companies stand to benefit enormously by using public roads as a living laboratory to assist in developing automated vehicle technology. However, other road users are not afforded commensurate safety and compensation assurances for the risks they take from sharing the road with immature technology that presents a real and present danger to other road users. Especially of concern is the palpable risk to vulnerable road user (pedestrians, cyclists, etc.).

Additionally, much of the bill is drafted in a way that creates loopholes and exploitable ambiguities. Regardless of root cause and intent, the net effect is to weaken the bill dramatically, to the point that it amounts to not much more than a free pass of HAV companies to do whatever they want. (Even insurance requirements, such as they are, would be scant deterrent to a company chasing a trillion dollar market. Even a $5 million payout to an injured party is too easily characterized as a cost of doing business when development involves a multi-billion dollar war chest.)

This bill, if passed as-is, will dramatically weaken what safety protection Commonwealth residents have under the current HAV regulatory posture. Passing this bill as it is now will be actively harmful -- it is worse than doing nothing. 

The press conference made a big point that local HAV companies were consulted for the bill -- but not public interest stakeholders. This entire situation really brings into question whether those companies should be considered trustworthy in terms of sincere regard for the public interest vs. their own profit motives (see SSRN paper sections II, IV, and Conclusions).

There is no simple fix for this bill. It needs a major overhaul. 

EXAMPLE ISSUES:

Below are some (potentially aggressive) interpretations of this bill to point out how problematic it is in its current form:

• This bill permits HAV testing with absolutely no oversight by PennDOT, no permit, and no "license test" of the HAV so long as an insurance minimum coverage of $1 per incident is met. 
• This bill permits an unlicensed 12-year-old to act as a Level 3 fallback/safety driver in a heavy truck HAV transporting radioactive waste through local towns and urban centers, with no possibility for municipalities to prohibit that activity.
• HAVs are not required to follow most traffic laws that apply to human drivers, and cannot be pulled over by anyone other than PA State Police for traffic violations. Except they don't have to pull over for even PA State Police, because they are not required to yield to emergency vehicles, nor to stop when pulled over.
• An HAV operator could hire judgement proof remote vehicle operators outside the US who are immune to state law sanctions and incentives. Traffic ticket points would be meaningless, as would responsibility for criminal driving behavior such as driving under the influence. (For that matter, how would you give someone in a foreign country a breathalyzer test?)  Companies could just replace that driver after every infraction -- assuming that police can even identify who the remote operator is, which is not provided for in the bill.
• A passenger in the back seat of a robotaxi might be found culpable for an injury or death caused by a crash. The mechanism would be designating the robotaxi as Level 3, with a click-through rider agreement (that is not read in practice) making passengers responsible for pressing an obscurely placed red panic button in event of a control malfunction. Such robotaxis could operate even if that passenger is a minor, impaired, or does not have a driver license.

It seems likely that at least some of the above is due to drafting errors and internal inconsistencies. For example, the $1 insurance limit might be intentional, or might just be worded incorrectly as a maximum instead of a minimum $5 million insurance requirement. One would hope that a responsible company would not act so egregiously -- but not all companies are responsible, and mere hope is not a responsible plan for public safety.

These issues illustrate the point that this bill doesn't just need a little clean-up of loose ends. It has major issues and should be entirely revisited.

DETAILED NOTES:

Below are notes on issues found in a review of the bill language:

1. Page 2, lines 7-15: This invokes SAE J3016 levels 3, 4, and 5 for coverage of the bill. This leaves open the "Level 2 Loophole" by which any company can attempt to claim they are simply putting a driver assistance feature on the road when they are really testing dangerously immature highly automated driving features. Tesla is already doing this with FSD (SSRN paper Section I.B). The bill instead should also apply to any testing of pre-series production features that control vehicle steering and require a test driver.  See: SSRN paper section V.
2. Page 2, lines 20-21: does not put any requirements on what it means to be an "authorized affiliate." For example, Tesla FSD beta testing using owners who are not trained as testers could be considered "authorized" affiliates. Anyone operating a test vehicle should be qualified per SAE J3018 and/or have a special tester license issued by PennDOT.
3. Page 3 line 13: requires annotating "highly automated vehicle" status on a title. While that is reasonable, in the context of this bill it seems that the only requirement to begin testing is to find a way to get this on a title. There is no permitting or approval process mentioned for anything except truck platoons. There should be a formal permitting process with follow-up monitoring of operational safety given the immature state of the technology.
4. Page 3 line 29 – page 4 line 3: authorizes platoons of no more than 2 total vehicles with second vehicle having no driver. At the moment this is more stringent than the rest of the bill, but should be revised per other notes to take into account that if the nonlead vehicle loses track of the lead vehicle (for example, lead vehicle exits the roadway for some reason) the nonlead vehicle is now an HAV operating solo.
5. Page 4 lines 4-11: authorizes platoon operations by filing and reviewing a plan with PennDOT. All HAV operations should file a plan, not just platoons. PennDOT should be required to review all plans at submission and periodically (not less frequently than one year), with ability to revoke licenses at any time for safety concerns.
6. Page 4 line 16 – Page 7 line 8: requires HAV to stop at accident scenes and owner/registrant report to police with insurance information. (This is repeated in different variations). An additional requirement should be added to ensure that the identity of any remote person who might be held responsible for or have contributed to an accident is promptly identified and made available (e.g., via video conference) to police at the scene just as if they were physically present in the car.
7. Page 7 lines 12-19: Removes the requirement for following regulations that apply to a (human) driver in the vehicle without imposing an obligation for equivalent means for the ADS to meet the intent of any relevant laws or regulations.
• As a simple example, autonomous trucks carrying fuel, explosives, or radioactive materials would not be required to stop at railroad crossings per PA Code Title 75 Ch 33, 3342, because that requires the driver to stop, not the vehicle. This issue seems likely to be pervasive, and might result in HAVs not being required to follow traffic laws in instances they are phrased as driver actions rather than vehicle actions.
• As another example HAVs would not be required to yield right of way to emergency vehicles, because that is a driver responsibility per PA Code Title 75 Ch 33, 3325.
• The bill also exempts any requirement "not relevant for an ADS," which is a subjective determination that impairs certainty of interpretation. If an ADS thinks it can ignore a red traffic signal or stop sign without a collision because it has a 360 degree field of view from a roof-mounted lidar, one could say that obeying such traffic signals (or performing full and complete stops) are "not relevant" for that ADS.
• The wording is confusing and ambiguous. If the ADS computer box is physically located in the driver seat as a matter of convenience, is that a "driver seated in the vehicle" since the ADS is considered the driver? 
• An issue that needs to be resolved is on the one hand saying an ADS does not need to follow human driver rules (pg 7 lines 17-18), but then saying the ADS might be the driver (pg 11 lines 20-24). So is the ADS the driver just for liability? Does it actually need to follow parts of traffic laws that apply to human drivers even though the bill says human driver rules don't apply to an ADS? Needs to be resolved.
8. Page 8 line 25: quotes J3016 definition of DDT. Given the history of J3016 “lateral vehicle motion” is not the same as “turning" but rather can be interpreted to mean lane-keeping. This ambiguity can be exploited as part of the Level 2 Loophole. If "turning" at intersections is meant to be included in the DDT and permit Level 2 systems to act in a way indistinguishable from Level 3 systems, that should be stated. It would be better to limit Level 2 systems to those that are not capable of making turns at intersections. (See SSRN paper section V.)
9. Page 9 lines 15-18: incorrectly characterizes the Level 3 intervention wording. SAE Level 3 does not require notification by the ADS for "evident" failures. So any assumption that a Level 3 ADS will always notify the driver when to take over is false. Nor does it deal with the issue of how a remote teleoperator is supposed to detect "kinesthetically apparent" failures if not in the vehicle.  This definition should be changed to require driver notification by the ADS of all failures relevant to ability to safely drive the vehicle. See SAE J3016 Myth #6 here.
10. Page 9 line 29 - Page 10 line 3.: This section gives permission for operation with no human driver on board, but does so in an overly broad manner. Given the wording, one might assume that  the DDT Fallback operation is NOT considered a "driver" since DDT Fallback is by J3016 different distinct from performing the Dynamic Driving Task (DDT). 
11. Page 9 line 29 - Page 10 line 3.: Part (1) requires "capable of operation" in compliance with regulations, but does not actually require it to operate in compliance with traffic laws, regulations, and relevant ordinances.
12. Page 10 lines 11-15: requires a minimum risk condition (MRC) be achieved in case of an ADS failure. It does not require an MRC if a failure of non-ADS equipment renders the vehicle unsafe to drive, but should. It does not require an MRC if the vehicle is forced out of its ODD e.g., due to a sudden rain squall it is not designed to handle, or entering a construction zone it is not designed to handle. (This is related to the "evident failure" exclusion of Level 3 previously discussed.)
13. Page 10 lines 11-15: does not require that the MRC be free of unreasonable risk. An MRC can do a panic stop in front of a heavy truck, or stop in the proverbial railroad crossing with an oncoming train. This should be changed to further require the MRC to be be free of unreasonable risk to the maximum degree practicable given vehicle and environmental conditions.
14. Page 10 lines 16-18: require a licensed driver. However, that licensed driver might be abroad in a foreign labor market (for example, in Central America). There are numerous issues raised by this that need to be dealt with such as what happens if such a driver behaves in a reckless or intentionally malicious manner with a vehicle on Commonwealth roads. How do you give such a driver a breathalyzer test (even a teleoperator who is within PA)? How do you arrest such a driver when called for? At the very least, PennDOT should be given broad latitude to restrict and license teleoperator drivers, as well as require some mechanism to assure that companies will be held responsible for the behavior of their remote drivers.
• The bill includes the phrase "The highly automated vehicle driver on board must be properly licensed under this title" which might or might not be interpreted to be a PA state license, but it is unclear if this is the case. Without clarification, AV operators might find it easier to argue in support of foreign teleoperators to exploit this situation. 
• For Levels 4 & 5 there is no requirement for a driver, nor a requirement that the vehicle be safe (per SAE J3016), so the requirement for a a human driver to be licensed can be disclaimed simply by avoiding calling a vehicle a "test" vehicle.
• For Level 3, a "highly automated vehicle driver" (page 2 lines 16-19) drives or is in physical control of a vehicle, which describes the DDT, but not necessarily the Fallback task required in a level 3 vehicle.
• Since the bill does not distinguish test vehicles, the situation is further muddled.
15. Page 10 line 24 – page 11 line 5: this exempts school buses, which is good. Placarded loads should also be excluded. Per SAE J3016 the ADS has no responsibility whatsoever to monitor vehicle condition or other aspects of vehicle safety (e.g., loose loads, cargo fires, tires on fire, lost wheels). Hazardous loads should not be carried without human supervision at this early stage of deploying technology.
16. Page 11 lines 6-17: this authorizes transportation network services without safety drivers. Such services should require SAE Level 4 or 5, and exclude SAE Level 3. Operating a transportation network service at Level 3 creates significant risk of using passengers that are not able to ensure safety in practice as a "moral crumple zone" -- blaming them for failing to avoid a crash in situations for which it is unreasonable to expect them to do that. At the very least, minors should be prohibited from riding unescorted in a Level 3 transportation network vehicle.
17. Page 11 lines 20-26: declares the ADS is the driver if there is no safety driver for the purpose of licensing drivers. This is an exceptionally bad idea.
• Is PennDOT required to administer driver tests to an ADS?  This needs to be clarified. Especially since in practice there is likely to be a requirement to re-license after every software update, which might occur daily. A driver license test cannot ensure HAV safety. Rather conformance to industry standard safety requirements proposed by the NHTSA ANPRM on Framework for Automated Driving System Safety should be required (including at least ISO 26262, ISO 21448 and ANSI/UL 4600) for any production ADS (one without a human safety driver monitoring its operation as a test platform).
•  Any unlicensed driver, including a minor, could load ADS software onto a cell phone and start driving themselves around the city with no need for a driver license, and no HAV testing permit. This is not just a theoretical possibility. Such systems are for sale now that advertise compatibility with over 150 vehicle types starting at $1100: https://comma.ai/ and could plausibly be claimed as SAE Level 3 systems by someone registering a vehicle in PA. (To be clear, these systems are likely to be unsafe if operated as Level 3 systems, but J3016 does not require safety operation or even driving competence for an ADS to be assigned a particular automation level. Such a Level 3 or even Level 4 claim could be made within the scope of SAE J3016 invoked by the bill either now or in the readily foreseeable future.)
• It is unclear what effect this would have on insurance issues, but it might result in literally having no natural person and no company to pursue for recompense after a major injury or fatality to a vulnerable road users who does not carry automotive insurance (and indeed neither owns nor drives a car). I point out a concern here, and defer to legal experts on this matter.
18. Page 11 lines 20-26: directs police to cite the HAV "owner or registrant".
• Why should a vehicle owner who puts their current-unused vehicle into a transportation network pool be held responsible for traffic violations committed by an ADS, when they likely have no understanding of how it has been programmed and certainly have no control over its driving? Either this clause assumes only large sophisticated companies will own or register vehicles (which is likely to be false soon for Level 3), or this is a clause intended to transfer liability onto hapless vehicle owners who have no practical ability to control the actions of their HAV's software.
• And what if the "registrant" is an anonymous series LLC in another state (or country) with no assets other than the car, while the citation is for a serious offense such as vehicular manslaughter? How are those whose actions, or perhaps even negligence, might have contributed to easily avoidable harm held accountable?
• A different approach should be used to hold a party accountable for driving behavior who has an actual understanding and/or ability to control the behavior and software quality of the HAV.
• Again, I point out concerns, but defer to legal experts on this matter.
19. Page 11 line 27- page 12 line 3: If there is a remote safety driver, police citations are issued to that remote safety driver. Again, what if the safety driver is outside the US? What is to stop a company from employing semi-disposable safety drivers that are simply fired after an infraction with another driver hired to replace them after each traffic ticket? (This is another potential manifestation of moral crumple zone abuse.) PennDOT should be given broad latitude here to, for example, create a point assessment system book-kept against a large company designing and/or operating an AV rather than solely against a safety driver for illegal behavior.
20. Page 12 lines 9-17:  $5M insurance seems grossly inadequate. (I'm aware other states do this, but they are historically dealing with testing with human safety drivers, not un-crewed HAVs.) A few issues that come to mind are listed here, and there are no doubt others.
• The $5 million amount is a "not to exceed." This implies it can be lower, and the bill prohibits carrying more insurance than that. As written, this requirement permits either state minimums or $1 of insurance, depending on whether this is considered to preempt other insurance wording, and what other actions PennDOT might take.
• There is no insurance requirement for the Fallback operator, who might be found at fault by police but not qualify as a "driver." Thus in a mishap there may be no insurance coverage to pursue.
• There is no mechanism for vulnerable road users without their own automotive insurance policy to collect without initiating a lawsuit (e.g., for hospital co-pays for injuries they have suffered). Perhaps HAV policies should treat such crash victims as named insured parties.
• The US DOT sets a statistical value of human life at $11.6 million for 2020, with yearly increases. The insurance minimum should be this amount, and should be per person, not per incident as seems implied by the bill wording.
21. Page 12 lines 18-25: this is incredibly broad preemption language. It should be entirely removed so that municipalities such as the City of Pittsburgh can adopt further ordinances to ensure safety responsive to local conditions. Some concerns include:
• This could be interpreted to mean that HAVs do not have to obey local traffic rules.
• This could be interpreted to mean that HAVs do not have to obey traffic directions from local police or school crossing guards.
• This could be interpreted to mean that HAVs do not have to obey local police stops.
• In general this clause is an egregious example of autonomandering (see SSRN paper section IV)
22. Page 12 lines 27-30: gives PennDOT permission to regulate or publish guidance “consistent with this title.” PennDOT should additionally have the ability to issue at least temporary rules more strict than this bill in the interest of public safety, and should be explicitly given permission to require operating permits. As it stands, since no permitting process is required, PennDOT would seem to have no authority to stop reckless testing being conducted by a bad actor, and neither would cities due to the preemption clause. PennDOT should be given authority to issue operating permits subject to quarterly review, along with establish data reporting requirements so that it can monitor that HAV operation is free from unreasonable risk.
23. In contrast to a statement made at the press conference, SAE J3016 is not a safety standard in any way, shape, or form. Rather, conforming to the letter of the standard for assigning SAE Levels (and no more) is guaranteed to result in vehicles that are unsafe in practice. (For example, J3016 does not require driver monitoring for safety drivers.) Instead, conformance to SAE J3018 for safe road testing should be required.
24. There is no requirement anywhere for HAVs to be safe. There should be a requirement that the company self-certify they have a credible safety case that their vehicles will be at least as safe as an average unimpaired human driver in a comparable ODD, and that they will update the safety case and re-certify that statement to PennDOT quarterly.
25. Page 7 line 28 - page 8 line 7: a "set of devices or components" that produces a rear view image is considered a mirror for Federal laws. For context, this is referring at least to FHWA width requirements, which exclude mirrors from truck width. Big rig truck mirrors stick out at a height well above cars, and are not hugely threatening compared to say a wider truck, so as a practical matter this rule makes sense. Replacing side view mirrors with cameras is also sensible, but as drafted this provision seems to have issues:
• It appears to over-ride Federal regulations. Is it really PA's place to tell FHWA what does and does not count as a mirror? Why not kick this to PennDOT and get a ruling from FHWA as a matter of enforcement policy rather than law?
• As a practical matter, the FBI does not (as far as I know) enforce truck equipment regulations. So this seems to have the effect of telling PA State Police not to enforce truck width requirements so long as the driver can claim whatever sticking out from the truck is "similar" in function to a mirror -- without restriction. One wonders if this is an attempt to normalize evading FHWA regulations for truck size requirements at the state-by-state level. 
• The apparent reason for a width rule is to set a boundary for switching to "wide load" procedures, which has to be set at some definite number -- 2.6 meters as it turns out. With this loophole, you could put two-foot-wide armor-plated camera box sticking out the side of a truck down where it is at eye level to passenger vehicle drivers (wouldn't want expensive cameras to get damaged), effectively making trucks wider than they would otherwise be. Even if somehow it is OK to override FHWA rules, there should be a restriction that such a device should be comparable in size, placement, and crash risk to a 3-D footprint to a mirror, or the like.

Some issues might be fixed easily, but others will likely require more pervasive changes. Be that what it may, a bill that has this much of an effect on public safety should be easy to understand for all stakeholders. And certainly should not be mispresented by its sponsors (as was done, for example, by claiming SAE safety standards had been incorporated).

It seems likely that bill advocates will argue that deficiencies in the bill might be made up for by PennDOT guidance and policies. Maybe so -- and maybe not.  

It is also possible other sections of the vehicle code interact with this bill in counter-intuitive ways. Still, since we've already heard clearly inaccurate statements about the bill, we need some more transparency as to the actual regulatory outcome will be, especially since the bill was developed without broad stakeholder input.

Regardless, it seems ill advised to put out a bill with known deficiencies in hopes that they might be made up for later. Why not get it right in the first place?

The Paris Tesla Taxi Crash and why crash investigations are broken

The tragic Tesla Paris crash of December 2021 highlights a completely dysfunctional vehicle defect investigation process. This story is well done, but highlights the usual broken investigation narrative (not fault of reporters):

The Paris government says: "no indication" of a "technical fault"
Well, did they look at software as a potential cause? Almost certainly not. Does Tesla follow international software safety standards (e.g., ISO26262)? Probably not. So it is a least possible this is a software defect.

What kind of software defect you ask? Well what if a software defect misreports the position of the accelerator pedal? What if there is a botched redundancy pattern in electronic controls? What if flaky code turns on AP? Mistakes like this are in other production vehicles.

Well, NHTSA exonerated 200+ Teslas earlier this year, right? Read the report. NHTSA DID NOT CONSIDER SOFTWARE DEFECTS at all in their investigation. If you don't even look, you won't find it. Read it for yourself!

In fact, there is a multi-decade pattern of NHTSA ignoring potential software defects in investigations unless they are readily reproduced. Instead, if it isn't readily reproduced, it must be the human driver. Scholarly paper on this history here.

We still don't know what happened here. But if it is a software problem across multiple vehicles, we're not going to fix it by ignoring it. Historically, the industry and NHTSA blame the driver to avoid having to deal with software safety.

I hope the French can do better here.

I hear so often "no other cars have that type of defect," or "it's not possible software is that bad." That's why I compiled this list of 100+ dangerous software defects that have resulted in recalls (i.e., the ones companies actually fixed): https://betterembsw.blogspot.com/p/potentially-deadly-automotive-software.html

(For those who aren't software safety experts, saying "we tested a car for a few miles and the accelerator worked fine, so no software error in Teslas" is the same as saying "your phone software crashed that one time but you can't make it happen again, so it can't be true.")

Automated Vehicle Regulatory Launch Page

 I've worked on a number of pieces that are relevant to automated/autonomous vehicle (AV) regulations lately. Here is a resource page to organize the material all in one place.

• Round table discussion on AV ethics, safety, and trust (good starting point for orientation to issues; should be no paywall despite being IEEE)
• Smerconish general audience piece on issues with AV industry public road testing (both Tesla and rest of industry)
• Video of SiriusXM Radio interview: Michael Smerconish and Phil Koopman
• Five Principles for regulation of highly autonomous vehicles
• Keynote talk on autonomous vehicle safety engineering
• AV industry and trust
• SSRN Autonomous Vehicle Regulation and Trust (detailed scholarly publication preprint for legal journal).  Several topics in this paper appear in shorter versions elsewhere on this list
• Autonomous Vehicle Myths: The Dirty Dozen  {AV policy myths, disinformation campaign}
• EE Times short version 
• Longer, more detailed version 
• Video talk on the dirty dozen
• Letter sent to Washington State rebutting the type of disinformation that is being circulated by AV industry groups in this case about the UL 4600 standard
• General regulatory playbook for state & municipal DOT/DMV organizations
• Regulatory playbook 
• Why disengagements are the wrong metric.  
• Proposal for regulating AV testing that addresses the Level 2 loophole
• Tesla FSD issues
• Tesla FSD is functionally a Level 4 test bed (short version)
• Why Tesla FSD should be treated as a Level 4 test deployment by regulators (long version)
• J3016 information
• What it would take to make Level 2 & Level 3 systems safe beyond J3016 
• SAE J3016 User Guide. (Important: J3016 is not and never has been a safety standard.)
• An alternative to J3016 levels for communicating to drivers
• Video talk on J3016
• Ethics, safety & trust for AVs:
• Lessons learned from the Tempe/Uber testing fatality
• Ethical issues that matter (hint: not the trolley problem)
• Safety case for public road testing
• Software safety for vehicle automation short course (part of a grad course I teach at Carnegie Mellon University)
• More general coverage podcasts:
• AV safety metrics podcast series
• Autonocast: AV safety engineering
• No Parking Podcast: building trust & reliability in autonomous vehicles. (Older material from 2019)

A quick note on standards:

• SAE J3018 is relevant to public road testing in which a human driver ensures safety
• Important: it is about driver capability, not automation capability
• When there is no human driver continuously monitoring safety, then automated vehicle technical standards become relevant: ISO 26262, ISO 21448, ANSI/UL 4600
• One slide showing how the standards fit together when there is no safety driver
• More details about ANSI/UL 4600

Regulating Automated Vehicles with Human Drivers

Summary 

Regulatory oversight of automated vehicle operation on public roads is being gamed by the vehicle automation industry via two approaches: (1) promoting SAE J3016, which is explicitly not a safety standard, as the basis for safety regulation, and (2) using the "Level 2 loophole" to deploy autonomous test platforms while evading regulatory oversight. Regulators are coming to understand they need to do something to reign in the reckless driving and other safety issues that are putting their constituents at risk. We propose a regulatory approach to deal with this situation that involves a clear distinction between production "cruise control" style automation that can be subject to conventional regulatory oversight vs. test platforms that should be regulated via SAE J3018 use for testing operational safety.

Video showing Tesla FSD beta tester unsafely turning into oncoming traffic.

https://www.youtube.com/watch?v=Fmj5MkyUD08&t=405s

Do not use SAE J3016 in regulations

The SAE J3016 standards document has been promoted by the automotive industry for use in regulations, and in fact is the basis for regulations and policies at the US federal, state, and municipal levels. However, it is fundamentally unsuitable for the job. The issues with using SAE J3016 for regulations are many, so we provide a brief summary. (More detail can be found in Section V.A of our SSRN paper.)

SAE J3016 contains two different types of information. The first is a definition of terminology for automated vehicles, which is not really the problem, and in general could be suitable for regulatory use. The second is a definition of the infamous SAE Levels, which are highly problematic for at least the following reasons:

• The defined levels are arbitrarily selected, and do not fit vehicles being actually produced that well. (If they did, you would not see companies advertising "Level 2+" when fractional levels are forbidden by J3016.)
• The defined levels have lower but not upper bounds on capabilities, making it straightforward to game levels to avoid regulations if desired by leaving small intentional holes in capabilities (in essence, a notional Level 2.99 might be used as Level 3 or 4 in practice by a customer, but not regulated because it rounds down to Level 2.)
• Vehicles designed to implement Levels 2 and 3 -- and no more -- can be expected to be unsafe in practice, for example due to lack of mandatory driver monitoring.
• Levels are set by manufacturers according to "design intent," meaning that a manufacturer can attempt to game regulations simply via disingenuous declarations of intent.
• Overall, the structure and details of J3016 are vague and subjective, degrading regulatory certainty.

In practice, one of two big issues is the "Level 2 Loophole" in which a company might claim that the fact there is a safety driver makes its system Level 2, while insisting it does not intend to ever release that same automated driving feature as a higher level feature. This could be readily gamed by, for example, saying that Feature X, which is in fact a prototype fully automated driving system, is Level 2 at first. When the company feels that the prototype is fully mature, it could simply rebrand it Feature Y, slap on a Level 4 designation, and proceed to sell that feature without ever having applied for a Level 4 testing permit. We argue that this is essentially what Tesla is doing with its FSD "beta" program that has, among other things, yielded numerous social media videos of reckless driving despite claims that its elite "beta test" drivers are selected to be safe (e.g., failure to stop at stop signs, failure to stop at red traffic signals, driving in opposing direction traffic lanes).

The second big practical issue is that J3016 is not intended to be a safety standard, but is being used as such in regulations. This is making regulations more complex than they need to be, stretching the limits of the esoteric technical expertise in AVs required of regulatory agencies, especially for municipalities. This is combined with the AV industry promoting a series of myths as part of a campaign to deter regulator effectiveness at protecting constituents from potential safety issues. The net result is that most regulations do not actually address the core safety issues related to on-road testing of this immature technology, in large part because they aren't really sure how to do that.

For road testing safety purposes, regulators should focus on both the operational concept and technology maturity of the vehicle being operated rather than on what might eventually be built as a product. In other words "design intent" isn't relevant to the risk being presented to road users when a test vehicle veers into opposing traffic. Avoiding crashes is the goal, not parsing overly-complex engineering taxonomies.

The solution is to reject SAE J3016 levels as a basis for regulation, instead favoring other industry standards that are actually intended to be relevant to safety. (Again, using J3016 for terminology is OK if the terms are relevant, but not the level definitions.)

Four Regulatory Categories

We propose four regulatory categories, with details to follow:

• Non-automated vehicles: These are vehicles that DO NOT control steering on a sustained basis in any operational mode. They might have adaptive speed control, automatic emergency braking, and active safety features that temporarily control steering (e.g., an emergency swerve around obstacles capability, or bumping the steering wheel at lane boundaries to alert the driver).
  

• Low automation vehicles: These are vehicles with automation that CAN control steering on a sustained basis (and, in practice, also vehicle speed). They are vehicles that ordinary drivers can operate safely and intuitively along the lines of a "cruise control" system that performs lane keeping in addition to speed control. In particular, they have these characteristics:
• Can be driven with acceptable safety by an ordinary licensed driver with no special training beyond that required for a non-automated version of the same vehicle type.
• Includes an effective driver monitoring system (DMS) to ensure adequate driver alertness despite inevitable automation complacency
• Deters reasonably foreseeable misuse and abuse, especially with regard to DMS and its operational design domain (ODD)
• Safety-relevant behavioral inadequacies consist of omissive behaviors rather than actively dangerous behavior
• Safety-relevant issues are both intuitively understood and readily mitigated by driver intervention with conventional vehicle controls (steering wheel, brake pedal)
• Automation is not capable of executing turns at intersections.
• Field data monitoring indicates that vehicles remain at least as safe as non-automated vehicles that incorporate comparable active safety features over the vehicle life.

• Highly automated vehicles: These are vehicles in which a human driver has no responsibility for safe driving. If any person inside the vehicle (or a tele-operator) can be blamed for a driving mishap, it is not a highly automated vehicle. Put simply, it's safe for anyone to go to sleep in these vehicles (including no requirement for a continuous remote safety driver) when in automated operation.

• Automation test platforms: These are vehicles that have automated steering capability and have a person responsible for driving safety, but do meet one or more of the listed requirements for low automation vehicles. In practical terms, such vehicles tend to be test platforms for capabilities that might someday be highly automated vehicles, but require a human test driver -- either in vehicle or remote -- for operational safety.

Non-automated vehicles can be subject to regulatory requirements for conventional vehicles, and correspond to SAE Levels 0 and 1. We discuss each of the remaining three categories in turn.

Low automation vehicles

The idea of the low automation vehicle is that it is a tame enough version of automation that any licensed driver should be able to handle it. Think of it as "cruise control" that works for both steering and speed. It keeps the car moving down the road, but is quite stupid about what is going on around the car. DMS and ODD enforcement along with mitigation of misuse and abuse are required for operational safety. Required driver training should be no more than trivial familiarization with controls that one would expect, for example, during a car rental transaction at an airport rental lot.

Safety relevant issues should be omissive (vehicle fails to do something) rather than errors of commission (vehicle does the wrong thing). For example, a vehicle might gradually drift out of lane while warning the driver it has lost lane lock, but it should not aggressively turn across a centerline into oncoming traffic. With very low capability automation this should be straightforward (although still technically challenging), because the vehicle isn't trying to do more than drive within its lane. As capabilities increase, this becomes more difficult to design, but dealing with that is up to the companies who want to increase capabilities. We draw a hard line at capability to execute turns at intersections, which is clearly an attempt at high automation capabilities, and is well beyond the spirit of a "cruise control" type system.

An important principle is that human drivers of a production low automation vehicle should not serve as Moral Crumple Zones by being asked to perform beyond civilian driver capabilities to compensate for system shortcomings and work-in-progress system defects. If human drivers are being blamed for failure to compensate for behavior that would be considered defective in a non-automated vehicle (such attempting to turning across opposing traffic for no reason), this is a sign that the vehicle is really a test platform in disguise.

Low automation vehicles could be regulated by holding the vehicles accountable to the same regulations as non-automated vehicles as is done today for Level 2 vehicles. However, the regulatory change would be excluding some vehicles currently called "Level 2" from this category if they don't meet all the listed requirements. In other words, any vehicle not meeting all the listed requirements would require special regulatory handling.

Highly automated vehicles

These are highly automated vehicles for which the driver is not responsible for safety, generally corresponding to SAE Levels 4 and 5.  (As a practical matter, some vehicles that are advertised as Level 3 will end up in this category in practice if they do not hold the driver accountable for crashes when automation is engaged.)

Highly automated vehicles should be regulated by requiring conformance to industry safety standards such as ISO 26262, ISO 21448, and ANSI/UL 4600. This is an approach NHTSA has already proposed, so we recommend states and municipalities simply track that topic for the time being. 

There is a separate issue of how to regulate vehicle testing of these vehicles without a safety driver, but that issue is beyond the scope of this essay. 

Automation test platforms

These are vehicles that need skilled test drivers or remote safety monitoring to operate safely on public roads.  Operation of such vehicles should be done in accordance with SAE J3018, which covers safety driver skills and operational safety procedures, and should also be done under the oversight of a suitable Safety Management System (SMS) such as one based on the AVSC SMS guidelines.

Crashes while automation is turned on are generally attributed to a failure of the safety driver to cope with dangerous vehicle behavior, with dangerous behavior being an expectation for any test platform. (The point of a test platform is to see if there are any defects, which means defects must be expected to manifest during testing.)

In other words, with an automation test platform, safety responsibility primarily rests with the safety driver and test support team, not the automation. Test organizations should convince regulators that testing will overall present an acceptably low risk to other road users. Among other things, this will require that safety drivers be specifically trained to handle the risks of testing, which differ significantly from the risks of normal driving. For example, use of retail car customers who have had no special training per the requirements of SAE J3018 and who are conducting testing without the benefit of an appropriate SMS framework should be considered unreasonably risky.

This category covers all vehicles currently said to be Level 4/5 test vehicles, and also any other Level 2 or Level 3 vehicles that make demands on driver attention and reaction capabilities that are excessive for drivers without special tester training.

Regulating automated test platforms should concentrate on driver safety, per my State/Municipal DOT regulatory playbook. This includes specifically requiring compliance with practices in SAE J3018 and having an SMS that is at least as strong as the one discussed in the AVSC SMS guidelines.

Wrap-up

Automated vehicles regulatory data reporting at the municipal and state levels should concentrate on collecting mishap data to ensure that the driver+vehicle combination is acceptably safe. A high rate of crashes indicates that either the drivers aren't trained well enough, or the vehicle is defective. Which way you look at it depends on whether you're a state/municipal government or the US government, and whether the vehicle is a test platform or not. But the reality is that if drivers have trouble driving the vehicles, you need to do something to fix that situation before there is a severe injury or fatality on your watch.

The content in this essay is an informal summary of the content in Section V of: Widen, W. & Koopman, P., "Autonomous Vehicle Regulation and Trust" SSRN, Nov. 22, 2021. In case of doubt or ambiguity, that SSRN publication should be consulted for more comprehensive treatment.

-----

Philip Koopman is an associate professor at Carnegie Mellon University specializing in autonomous vehicle safety. He is on the voting committees for the industry standards mentioned. Regulators are welcome to contact him for support.

Debunking AV Industry Positions on Standards and Regulations

(This is the expanded version of: Philip Koopman, "Autonomous Vehicle Myths: The Dirty Dozen," EE Times, Oct. 22, 2021.)

Too often, I’ve read documents or listened to panel sessions that rehash misleading or just plain incorrect industry talking points regarding autonomous vehicle standards and regulations. The current industry strategy seems to boil down to “Trust us, we know what’s best,” “Don’t stifle innovation,” and “Humans are bad drivers, so computers will be better.”

As far as I can tell, what’s really going on is that automated-vehicle companies are saying what they say both to avoid being regulated and to avoid having to follow their own industry safety standards. That strategy has not yielded long-term safety in other industries that have tried it, however, and I predict that in the long term it will not serve the automotive industry well either. It certainly does not encourage trust. 

In this essay, I address the usual industry talking points and provide summary rebuttals. I’m intentionally simplifying and generalizing each talking point for clarity. It is my hope that other stakeholders, policymakers, and regulators can use this information to encourage AV companies to talk about the things that matter, such as ensuring the safety of all road users. We need more transparency and honest discussion — not a continuation of the current, empty rhetoric.

It is important to be clear that, from everything I’ve seen, the rank-and-file engineers, and especially the safety professionals — if the company has any — are trying to do the right thing. It is the government relations and policy people, not the engineers, who are providing the facile talking points. And it is the high-level managers — the ones who set budgets, priorities, and milestones — who affect whether safety teams have sufficient resources and authority to build an AV that will in fact be acceptably safe. So this essay is directed at them, not at the engineers.

For more detailed guidance to state and municipal DOT and DMV regulators, see this blog post.

(I've numbered these points for easier reference. Making yourself a bingo card and bringing it to the next regulatory/policy panel session you attend to keep score is optional.)

• Myth #1: 94% of crashes are due to human driver error, so AVs will be safer.

I’ve seen this myth stretched to suggest that 94% of crashes are due not just to human error but to “bad driver choices” (implying driving drunk and texting, perhaps). Sometimes, the cited percentage is 90%. Regardless of the particular spin, the usual, unstated implication is that AVs will be about 10 to 20 times safer by not making those same mistakes.

However, the 94% number is a misrepresentation of the original source. A vocal proponent of this myth, notably under the previous administration, has been the U.S. DOT, which ironically was the source of the study being misrepresented.

What the study data actually shows is that 94% of the time, a human driver might have helped avoid a bad outcome. The source explicitly says that this is not 94% “blame” on the human driver. (Not discussed is the astonishing effectiveness of human drivers in mitigating technical failures. It is more about pointing out that about once in a while, they don’t get it right.)

Sometimes, the bad outcome is due to an overt mistake or to driving impaired or distracted. Often, failure to wear a seatbelt turns what would otherwise be an injury crash into a fatality. 

But some crashes occur simply because the human driver incorrectly guessed the intent of another road user or misunderstood an unusual situation in the roadway. Those are mistakes an AV can make as well. The 94% number also ignores the possibility that roadway and other infrastructure improvements could increase safety even for human drivers.

Beyond the 94% number being more complex than just “driver error,” AVs will make different mistakes. This should be abundantly clear to anyone who’s watched automation road test videos. Yes, the technology will improve. But there is no evidence I’ve seen that proves an AV will be safer than a human driver anytime soon. The safety benefit is aspirational for now. And Tesla data doesn’t count, because Tesla blames the human driver for crashes, and a deployed fully automated vehicle does not have a human driver to blame. (See: "A Reality Check on the 94 Percent Human Error Statistic for Automated Cars")

To be sure, with something like $100 billion of investment chasing the problem, we could get there. Hardworking engineers at the AV companies are trying to make sure we do get there. But we’re still on the journey, not at the destination.

• Myth #2: You can have either innovation or regulation, not both.

This is a false dichotomy. You can easily have both innovation and regulation, if the regulation is designed to permit it.

To consider a simple example, you could regulate road testing safety by requiring conformance to SAE J3018. That standard is all about making sure that the human safety driver is properly qualified and trained. It also helps ensure that testing operations are conducted in a responsible manner consistent with good engineering validation and road safety practices. It places no constraints on the autonomy technology being tested. (Adaptation would be needed for a safety driver in a chase car if it was deemed impractical to have a safety driver sit in a prototype vehicle for road testing;  see this blog post.) 

For more general approaches, you can switch from the current approach of track and road testing to more goal-based testing. For example, a regulation that tells you what symbol to put on a dashboard to tell the driver sitting in a driver seat that there is low tire pressure (Federal Motor Vehicle Safety Standard [FMVSS] 138) does indeed constrain design by requiring a light, a dashboard, a driver seat, and a driver. But the lighted symbol isn’t the point; getting the low tire pressure addressed is the point, and regulations can focus on that instead (see this blog post).  To be sure, this requires a change in regulatory structure. But the choice isn’t between innovation and regulation; it is between old regulation and new regulation, and that is a far different matter — especially if the new elements of the regulatory approach are based on conforming to standards the industry itself wrote.

The primary industry standards for deployed AVs in the  U.S. are ISO 26262 (functional safety), ISO 21448 (safety of the intended function, or SOTIF), and ANSI/UL 4600 (system-level safety). Indeed, the current  US DOT proposal for regulation as of this writing is to get the AV industry to conform to precisely those standards. None of the standards stifle innovation. Rather, they promote a level playing field so companies can’t skimp on safety to gain a competitive timing advantage while putting other road users at undue risk.

If a company states that safety is their #1 priority, how can that possibly be incompatible with regulatory requirements to follow industry consensus safety standards written and approved by the industry itself?

• Myth #3: There are already sufficient regulations in place (for example in California).

Existing regulations (with one exception) do not require conformance to any industry computer-related safety standard, and do not set any level of required safety. At worst, it is the "Wild West." At best, there are requirements for driver licensing, insurance, and reporting. But requirements on assuring safety, if any, are little more than taking the manufacturer's word for it.

The one exception is the New York City DOT’s rule to require the SAE J3018 road testing safety standard, and to attest that road testing will not be more dangerous than a normal human driver. (See: https://rules.cityofnewyork.us/rule/autonomous-self-driving-vehicles/ . For bonus points, see if you can find any of these myths in the comments submitted in response to that standard, or in responses to the DOT proposal referenced under Myth #2.)

• Myth #4a: We don't need proactive AV regulation because of existing safety regulations.

The current Federal Motor Vehicle Safety Standard (FMVSS) regulations do not cover computer-based functionality safety. They are primarily about whether brakes work, whether headlights work, tire pressure, seat belts, airbags, and other topics that are basic safety building blocks at the vehicle behavior level. As the National Highway Traffic Safety Administration would tell you, merely passing FMVSS mandates is not enough to ensure safety on its own; it is simply a useful and important check to weed out the most egregious safety problems based on experience. 

There are no FMVSS or other regulatory requirements for automotive software safety in general, let alone for AV-specific software safety.

Safety regulators should think hard about an approach in which “safety” means requiring insurance to compensate the next of kin after a fatality. With multi-billion dollar development war chests, a few million dollars of payout after a mishap might not be sufficient deterrent to taking safety shortcuts in the race to autonomy.

• Myth #4b: We don’t need proactive regulation because of liability concerns and NHTSA recalls.

The National Highway Traffic Safety Administration generally operates reactively to bad events. Sometimes, car companies voluntarily disclose a problem. Other times, a number of people have to die or be seriously injured before NHTSA forces some action (for example, eleven crashes involving Tesla driver assistance "autopilot" with emergency vehicles occurred over a 3.5-year period before action was initiated, with an expectation of many months to resolution). For mature technology, maybe this is OK — if one makes the assumption that the industry is populated with only good faith actors. But even with that assumption, it isn’t enough for immature AV technology and manufacturers new to automotive safety.

Aircraft safety regulation used to wait for crashes, but air travel got a lot safer when FAA and airlines became proactive. Most importantly, a regulatory policy that waits for loss of life and limb before taking action can result in a process that takes years to solve problems, even as the loss events continue.

It would be better if companies voluntarily commit to follow their own industry’s safety standards. If not, we might be only one big news event from a regulatory hammer coming down.

• Myth #5: Existing safety standards aren't appropriate because (pick one or more):
• they are not a perfect fit;
• no single standard applies to the whole vehicle;
• they would reduce safety because they prevent the developer from doing more;
• they would force the AV to be less safe;
• they were not written specifically for AVs.

These statements misrepresent how the real standards work. ISO 26262, ISO 21448, and ANSI/UL 4600 all permit significant flexibility to be used in a way that makes sense. All three work together to fit any safe AV. 

ISO 26262 can apply to any light vehicle on the road for the parts that aren’t the machine-learning–based mechanisms. Cars still have motors, brakes, wheels, and other non-autonomous features that have to be safe. The hardware on which the autonomy software runs can still conform to ISO 26262. All of these are covered by ISO 26262, and the standard specifically permits extension to additional scope.

ISO 21448 is explicitly scoped for AVs in addition to ADAS. Its origin story includes being proposed as an addition to ISO 26262, and it is written to be compatible with that standard.

ANSI/UL 4600 is specifically written for AVs. It applies to the whole vehicle as well as support infrastructure. The voting committee includes experts on ISO 26262 and ISO 21448, so it is compatible with those standards, and in fact it leads naturally to using all three of these standards and not just a “single” standard. (Anyone who knows standards knows it is improbable that any safety-critical system would involve use of just a single standard.)

There is no reason not to conform to these standards, and U.S. DOT has already proposed this set for the United States. All of these standards allow developers to do more than required. All of them are flexible enough to accommodate any AV. None of them force a company to be less safe (really, that argument is laughable). None of the standards constrain the technical approach used.

• Myth #6: Local and state regulations need to be stopped to avoid a “patchwork” of regulations that inhibits innovation.

A significant reason that local and state regulations are a so-called patchwork is that in each jurisdiction, the AV companies play hardball to minimize regulation. Typically these negotiations involve statements that if regulation is too stringent, companies will take their jobs and spending elsewhere, and the jurisdiction in question will get a reputation for being hostile to innovation and technology. The outcome of each negotiation is different, resulting in somewhat different regulations or voluntary guidance from place to place.

If the industry changed its stance on avoiding regulation at all costs, the patchwork could be resolved via the same uniform state law-making mechanism that standardizes other driving laws. That would make things as uniform as is practical (“right turn on red” rule differences were around before AVs).

Moving to regulation based on industry standards would actually help in this regard, because national and international standards don’t change from city to city. 

A federal regulation that requires conformance to standards would help address this issue. A federal regulation that prevents states from acting but does not itself ensure safety would be worse than nothing.

• Myth #7: We conform to the “spirit” of ISO 26262, etc.

AV developers typically justify their “in the spirit of” statements by advancing the theory that there might be a need for deviation from the standard (beyond any deviations that the standards already permit). The statements never specify what the possibly required deviations might be, and I’ve never heard a concrete example at any of the many standards meetings I have attended. (I’m on the U.S. voting committees for all the standards listed in this essay.)

I’ve never heard an AV company argue, when making its case, that it conforms to the intent of a standard, just to the spirit of the standard — whatever that means. Indeed, any “in the spirit” statement is meaningless, because the standards I’ve mentioned are all flexible enough that if you actually conform to the spirit and intent of the standard, you can conform to the standard. The standards explicitly permit not doing things that don’t apply and deviating from inapplicable clauses with appropriate decision processes. Doing that still permits conformance to the actual standard.

I worry that AV companies’ “spirit” claims are really code for cutting corners on safety when they think it is economically attractive to do so, or they’re in a hurry, or both.

A reasonable alternative explanation is simply that lawyers might want to avoid committing to something if nobody is forcing them to do so. That is understandable from their point of view, but it impairs transparency. The dark side of the strategy is that it provides cover for companies that are not the best actors to hide cutting corners. If companies are worried that they’ll be called out for not following a standard after a crash, they should spend the resources to actually follow the standard. Or they should not spend so much effort making public claims about safety being their top priority.

Companies that are truly doing their best on safety should be transparent about conforming to consensus standards to raise the bar for others.

Consider whether you would ride in an airplane whose manufacturer said, “We conform to the spirit of the aviation safety standards, but we’re very smart and our airplane is very special so we skipped some steps. It will be fine. This aircraft type hasn’t killed anyone yet, so trust us.” 

Now ask yourself if you’d want to share the road with a test AV whose developer has said it wants the flexibility of not conforming to the industry standards for road testing that the developer itself helped write.

• Myth #8: Government regulators aren’t smart enough about the technology to regulate it, so there should be no regulations. Industry is smarter and should just do what it thinks is best.

Following the proposed  U.S. DOT plan to invoke the industry standards mentioned earlier makes sense, because it addresses precisely this concern. Even two years ago, the standards weren’t really there, but now they are. Industry decided which standards made sense, and then they created them.

If we could trust industry — any industry — to self-police safety in the face of short-term profit incentive and organizational dysfunction, we wouldn’t need regulators. But that isn’t the real world. Trusting the automotive industry to regulate development with immature, novel technology is unlikely to work. It’s possible, and important, for the industry to achieve a healthy balance between taking responsibility for safety and accepting regulatory oversight. Near-zero regulatory influence until after the crashes start piling up is not the right balance.

Near-zero regulatory influence until after the crashes start piling up is not the right balance.

It’s difficult to understand why it is a bad idea for government regulators to say to the AV industry: We want you to follow your own safety standards, just as all the other industries do.

• Myth #9: Disclosing testing data gives away the secret sauce for autonomy.

Road testing safety is all about whether a human safety driver can effectively keep a test vehicle from creating elevated risk for other road users. That has nothing to do with the secret-sauce autonomy intellectual property. It is about the effectiveness of the safety driver.

Companies sometimes say it would be too difficult or expensive to get or provide data. If companies don’t have data to prove that they are testing safely, they shouldn’t be testing. If they think that providing testing safety data is too expensive, they can’t afford the price of admission for using public roads as a testbed.

Testing data need not include anything about the autonomy design or performance. An example would be revealing how often test drivers fall asleep while testing. A non-zero result might be embarrassing, but how does that divulge secret autonomy technology data?

By the same token, regulators should not be asking for autonomy performance data such as how often the system “disengages” because of an internal fault or software issue. They should be asking how often other road users are placed at risk, which is an entirely different story. So the miles and locations tested, along with the collisions and near-miss situations that occur, make sense as measures of public exposure to risk. But metrics related to the quality of the autonomy itself do not, unless and until that data is being used to justify testing without a safety driver.

• Myth #10: Delaying deployment of AVs is tantamount to killing people.

The safety benefits of AVs are aspirational and targeted for sometime in the future. Every year, that “sometime” seems to get further away. Given the track record of promises and delays, nobody really knows how far in the future. Moreover, there is no real proof to show that AVs will ever be safer than human-driven vehicles, especially with human-driven vehicles becoming safer via active safety systems such as automated emergency braking (AEB). 

With something like $100 billion being spent on AV technology, it seems likely they will eventually be safer for appropriately restricted operational design domains (ODDs). But the “when” still remains a question mark.

Ignoring industry best practices to put vulnerable road users at risk today should not be permitted in a bid to maybe, perhaps, someday, eventually save potential later victims if the technology proves viable.

Even the famous RAND study urging early deployment was careful to say that AVs should be safer than human drivers initially. The discussion is not about whether AVs should be safer than humans when deployed, but rather how much safer (or, as RAND puts it, deploying while good rather than waiting for nearly perfect.) Deploying vehicles that aren’t clearly shown to be safer than an unimpaired human driver in a similar ODD violates this principle. So does adopting test practices that result in risk worse than that presented by an unimpaired human driver operating on the same roads.

If saving lives on the road today is indeed the No. 1 priority, then a small fraction of the tens of billions of dollars being spent on AVs could be spent on reducing the drunk driving rate (embarrassingly high in the U.S. compared with Europe), improving roadway infrastructure, improving speed limit strategies, installing safer pedestrian crossings and bikeways, and so on — not on increasing near-term risk via premature deployment or irresponsible testing.

Don’t forget that bad press from a high-profile mishap can easily sets the whole industry back. No company should be rolling the safety-shortcut dice to hit a near-term funding milestone while risking both people’s lives and the reputation of the entire industry.

• Myth #11: We haven't killed anyone, so that must mean we are safe.

Often this amounts to arguing, “We’ve gotten lucky so far, so we plan to get lucky in the future.” If there is no evidence of robust, systematic safety engineering and operational safety practices, this amounts to a gambler on a winning streak claiming they will keep winning forever. 

Think about the implications of accepting this argument. That means that every AV tester gets to operate however they like until they kill someone. This was effectively the dynamic in at least parts of the AV industry until  when a pedestrian was killed during a testing mishap in Tempe Arizona. We should not be giving developers a free pass on safety, looking into the matter only after they have killed someone.

The one possible exception to this argument might be claimed if the company has a statistically significant basis for showing safety. For fatalities, that is perhaps 300 million miles of operation with zero fatalities against a 100 million-mile average fatality rate. In practice, however, even this argument doesn’t really work, because it requires nothing to change for a vehicle that is still being tested. Changes to vehicle software, changes to roads, different operational environments, different driver demographics, and so on all reset the test odometer, so to speak, and invalidate any safety claims being made. In reality, an argument based on history alone does not prove safety. And it begs the question of how safety can be ensured while 300 million miles of operational evidence are accumulated to support the claim.

• Myth #12: Other states/cities let us test without any restrictions; you should too.

Whether regulators are willing to put their constituents at increased risk in exchange for some economic benefit is a decision they are permitted to make for themselves. But the hard reality is that any tester who is not at least doing as well as SAE J3018 for road testing safety is not following accepted practices and is likely putting the local population at unnecessary risk.

We all know what happened in the 2018 Tempe Arizona testing fatality. The NTSB chair pointed out at that investigation hearing that other companies didn’t need to have a similar crash to learn the lessons of this one. One result of learning those lessons was the starting document that became the newest revision of SAE J3018 for road testing safety. If testers won’t follow that consensus industry standard, they haven’t really taken that lesson to heart.

The responsibility of safety regulators is to promote safety. Vulnerable road users should not act as unwitting test subjects for AV road testers who can’t even be bothered to commit to following accepted industry safety practices. Regulators should not feel inhibited from merely asking developers to follow the industry safety standards that, in most cases, they themselves helped write.