Safe Autonomy: Proposal for putting the "Safety" back in the "S" of NHTSA

Summary: Overview of my comments on how NHTSA can effectively engage with the Automated Vehicle industry to ensure safety without inhibiting responsible innovation.

The National Highway Traffic Safety Administration is tasked with regulating non-commercial vehicle safety in the US. Thus far, their approach to highly automated vehicles has been conspicuously hands-off with regard to safety, with more emphasis on not inhibiting innovation rather than ensuring that the innovation proceeds with reasonable safety. In late 2020 NHTSA put out an ANPRM (Advanced Notice of Proposed Rule Making) proposing an approach to changing this situation. This approach is significantly based on adopting existing industry standards. Here is my high level response. (Link at the end of this to full document.)

A Federal Safety Framework for ADS should encompass the following elements. These elements have been selected to make use of existing industry efforts and provide a level playing field for an implementation-neutral approach to establishing a baseline for and continually improving safety.

1. Industry standards. NHTSA should encourage conformance to safety standards written by the automotive industry and stakeholders themselves, and issued as normative standards by accredited standards organizations (e.g., ISO, ANSI/UL, SAE). This includes but is not limited to ISO 26262, ISO 21448, ANSI/UL 4600, and safety-relevant security standards.

a. This should include full self-disclosure of standard conformance status for every highly automated vehicle operating on public roads, including aspects of the vehicle for which conformance is declared. (The sole exception should be test vehicles under the immediate control of a qualified safety driver as part of a publicly declared testing effort.) This would not necessarily be a requirement for conformance, but rather a requirement to be transparent and forthcoming about conformance with industry-created standards (or lack thereof). If no safety standards are conformed to, that should be so stated. A clear and unambiguous statement should be required (e.g., “we conform to ISO 26262”) rather than a vague statement such as “we use approaches inspired by [standard]” or “we adopt techniques drawn from [list of standards].”

b. It is important to note that such self-disclosure does not require public disclosure of sensitive proprietary technical information. For example, conformance to ANSI/UL 4600 does not require disclosing any technical information to any organization external to the organization declaring conformance.

c. I note that in industries other than automotive there is either required or voluntary conformance with comparable domain-specific safety standards. It is difficult to understand how the ADS industry, which justifies its need for regulatory breathing room by promising to make things safer, can at the same time fail to follow industry consensus safety standards for applicable aspects of their vehicles.

2. Transparency. NHTSA should act to increase transparency with regard to safety in the automated vehicle industry.

a. Specific steps should include updating the NHTSA-defined VSSA guidance scope to include all major aspects of ANSI/UL 4600 compared to the current subset of topics covered. (In fairness, the VSSA guidance was created before the April 2020 issuance of ANSI/UL 4600, so this should be considered an evolution of the VSSA guidance to track evolving issued industry standards.)

b. NHTSA should also increase industry participation rates in releasing technically substantive VSSAs. A properly formed VSSA document should in fact be a high level but technically substantive disclosure of the relevant safety case, and should be issued by every company putting a vehicle on public roads. This should include companies testing on public roads publishing a VSSA scoped to address the safety of the testing effort.

c. The release of some recent, technically substantive VSSAs and the public Web posting of the Uber ATG safety case framework demonstrate that significantly more transparency is viable without undue disclosure of sensitive proprietary information.

d. NHTSA should define and strongly encourage reporting safety outcomes (lagging metrics) in a uniform and transparent manner to demonstrate via data that ADS technology results in safer roads. This information should be supplied by manufacturers and operators rather than solely relying upon, for example, police reports. (Note that the industry itself could drive this standardization; it need not be a NHTSA-defined standard.)

e. A specific concern is ensuring that potential safety issues in one mode of operation (e.g., driver supervision) should not be buried in aggregate data (e.g., by mixing less safe mode data with safety improvements from active safety features during manual driving). An additional concern is that metrics should drive improved safety for road users rather than be used as a score card that is gamed to show progress in a “race to autonomy” (e.g., disengagement metrics are problematic for this reason).

3. Safety First. NHTSA should encourage the industry to collaborate on safety and compete on factors other than safety.

a. Safety should be a given. As with the airline industry, achieving industry-wide safety should involve cooperation among all stakeholders. NHTSA is in a unique position to foster such cooperation, potentially with support from neutral organizations.

b. A starting point can be a shared repository of potential hazards to be addressed when relevant to an ADS-equipped vehicle’s ODD.

c. NHTSA should facilitate a dialog on the topic of how safe is safe enough, including all stakeholders. This should address issues such as relevant metrics, risk transfer, taking credit for safety improvements to offset higher-risk operating modes within vehicle fleets, and degree to which near term risk can (or even should) be traded off against potential long-term aspirational safety improvements.

d. A longer term goal should be a set of ODD-specific lagging metric safety performance indicators and baseline minimum targets based on human driver performance to set a level playing field for safety performance reporting and outcome assessment.

4. Human Operators. NHTSA should ensure that the division of tasks between human operators and automated vehicles results in acceptable safety.

a. This should include monitoring deployed vehicles for an unsafe division of responsibility (e.g., systems overly prone to automation complacency that results in elevated mishap rates) as well as longer term research into driver monitoring effectiveness at ensuring operational safety.

b. NHTSA should encourage the industry to develop standards for measuring driver engagement in the context of driver monitoring systems and their effectiveness in naturalistic driving situations.

c. NHTSA should address all outstanding NTSB recommendations, especially in the area of driver engagement. (See: https://www.regulations.gov/comment/NHTSA-2020-0106-0617 )

5. Safety Cases. Longer term, NHTSA should transition from a test-based posture to a safety case-based posture that includes testing as a component.

a. For some aspects of safety, a test-centric approach is appropriate. However, in essentially all areas of large-scale computer-based system safety, testing is necessary but insufficient to ensure acceptable safety. Given the unique nature of machine-learning based technology incorporated into typical ADS equipment, process-based metrics and leading indicator metrics based on field engineering feedback will be essential to demonstrate and improve safety over the course of deployment.

b. A safety case-based NHTSA posture should involve asking ADS-equipped vehicle makers to use safety cases and (a) define what they mean by safe, (b) explain what reasoning is being used to argue they are safe, and (c) explain the basis of evidence to support that reasoning.

c. A critical part of this will be to ensure not only that ADS equipped vehicles send back field data to ensure that the safety case is valid in practice, but also that a metric-based approach ensures that the ADS design and deployment organizations are actually paying attention to and taking action upon data that indicates potential safety issues before loss events occur.

d. While good engineering, sound data collection practices, simulation, closed course testing, and safe road testing will all play a part in ensuring safety, the precise role of each of these is still open for ADS technology. Therefore, NHTSA should concentrate on ensuring that manufacturers have a coherent story to tell about safety rather than mandating what that story actually is. ADS equipped vehicles should only be deployed when they are demonstrably safe, but the form of the demonstration (which will need to include more than driving an actual vehicle) should be informed by the specific safety case involved.

6. Safety Critical Computer System Skills. NHTSA should significantly increase their staffing strength in computer-based system skills, especially in the area of software.

a. NHTSA has historically under-staffed in the area of computer-based system safety, and especially software safety. However, in recent years automobiles have transformed from electromechanical systems to computers-on-wheels. Especially in electric vehicles, there is simply no way to understand whether a vehicle is acceptably safe without understanding computer technology.

b. Currently, NHTSA reports routinely do not rule in computer-based system defects (and especially software) when considering potential root causes of mishaps. Yet there is a dramatic rise in software-related recalls. The writing is on the wall: significantly more capability is required in the area of safety critical software if NHTSA wants to remain relevant to actual safety outcomes. It is recognized that budgets are limited, but this is an area that simply cannot be neglected.

Links: