Friday, October 28, 2022

Talk: Autonomous Vehicles Standards & Open Challenges

 Here is my talk from the October 2022 ISO 26262/SOTIF conference.

Assuming you follow the relevant standards (ISO 26262, ISO 21448, ANSI/UL 4600) in practice teams are finding the following topics difficult:

  • Fail operational architecture
  • Building an accurate, predictive world model
  • Safety beyond the driving task (system safety, traffic system interactions)
  • Determining how safe is safe enough in an equitable way

Link to slides

(Slides only at this time)

Friday, October 21, 2022

AV Safety with a Telepresent Driver or Remote Safety Operator

Some teams propose to test or even operate autonomous vehicles (AVs) with a telepresent driver or remote safety operator.  Making this safe is no easy thing.

Woman wearing VR goggles holding steering wheel

Typically the remote human driver/supervisor located at a remote operating base, although sometimes they will operate by closely following the AV test platform in a chase vehicle for cargo-only AV configurations.

Beyond the considerations for an in-vehicle safety driver, telepresent safety operators have to additionally contend with at least:

·        Restricted sensory information such as potentially limited visual coverage, lack of audio information, lack of road feel, and lack of other vehicle physical cues depending on the particular vehicle involved. This could cause problems with reacting to emergency vehicle sirens and reacting to physical vehicle damage that might be detected by a physically present driver such as a tire blow-out, unusual vibration, or strange vehicle noise. Lack of road feel might also degrade the driver’s ability to remotely drive the vehicle to perform a fallback operation in an extreme situation.

·        Delayed reaction time due to the round-trip transmission lag. In some situations, tenths or even hundredths of seconds of additional lag time in transmissions might make the difference between a crash and a recovery from a risky situation.

·        The possibility of wireless connectivity loss. Radio frequency interference or loss of a cell tower might interrupt an otherwise reliable connection to the vehicle. Using two different cell phone providers can easily have redundancy limitations due to shared infrastructure such as cell phone towers,[1] cell tower machine rooms (for some providers), and disruption of shared backhaul fiber bundles.[2] A single infrastructure failure or localized interference can disrupt multiple different connectivity providers to one or multiple AVs.

Role of remote safety operator

Achieving acceptable safety with remote operators depends heavily on the duties of the remote operator. Having human operators provide high-level guidance with soft deadlines is one thing: “Vehicle: I think that flag holder at the construction site is telling me to go, but my confidence is too low; did I get that right? Operator: Yes, that is a correct interpretation.” However, depending on a person to take full control of remotely driving a vehicle in real time with a remote steering wheel at speed is quite another, and makes ensuring safety quite difficult.

A further challenge is the inexorable economic pressure to have remote operators monitoring more than one vehicle. Beyond being bad at boring automation supervision tasks, humans are also inefficient at multitasking. Expecting a human supervisor to notice when an AV is getting itself into a tricky situation is made harder by monitoring multiple vehicles. Additionally, there will inevitably be a situation in which two vehicles under control of a single supervisor will need concurrent attention when the operator can only handle one AV in a crisis at a time.

There are additional legal issues to consider for remote operators. For example, how does an on-scene police officer give a field sobriety test to a remote operator after a crash if that operator is hundreds of miles away – possibly in a different country? These issues must be addressed to ensure that remote safety driver arrangements can be managed effectively.

Any claim of testing safety with a telepresent operator needs to address the issues of restricted sensory information, reaction time delays, and the inevitability of an eventual connectivity loss at the worst possible time. There are also hard questions to be asked about the accountability issues and law enforcement implications of such an approach.

Active vs. passive remote monitoring

A special remote monitoring concern is a safety argument that amounts to the vehicle will notify a human operator when it needs help, so there is no need for any human remote operator to continuously monitor driving safety. Potentially the most difficult part of AV safety is ensuring that the AV actually knows when it is in trouble and needs help. Any argument that the AV will call for help is unpersuasive unless it squarely addresses the issue of how it will know it is in a situation it has not been trained to handle.

The source of this concern is that machine learning-based systems are notorious for false confidence. In other words, saying an ML-based system will ask for help when it needs it assumes that the most difficult part to get right – knowing the system is encountering an unknown unsafe condition –  is working perfectly during the testing being performed to see if, in fact, that most difficult part is working. That type of circular dependency is a problem for ensuring safety.

Even if such a system were completely reliable at asking for help when needed, the ability of a remote operator to acquire situational awareness and react to a crisis situation quickly is questionable. It is better for the AV to have a validated capable of performing Fallback operations entirely on its own rather than relying on a remote operator to jump in to save the day. Before autonomous Fallback capabilities are trustworthy, a human safety supervisor should continuously monitor and ensure safety.

Any remote operator road testing that claims the AV will inform the remote operator when attention is needed should be treated as an uncrewed road testing operation as discussed in book section 9.5.7. Any such AV should be fully capable of handling a Fallback operation completely on its own, and only ask a remote operator for help with recovery after the situation has been stabilized.

[1] For example, a cell tower fire video shows the collapse of a tower with three antenna rows, suggesting it was hosting three different providers. 

[2] While it is difficult to get public admissions of the mistake of routing both a primary and backup critical telecom service in the same fiber bundle, it does happen.

Friday, October 14, 2022

The Software Defined Vehicle Is Still More Wish Than Reality

Here is a Software Defined Vehicle video that covers a lot of ground. Car companies are all talking a big game about adding software to their vehicles, including big data, software updates, connectivity, and more. The possibilities are exciting, but you only have to read the news to know that the road to get there is proving bumpier than they'd like. (See this story too.)

Getting the mix of Silicon Valley software + automotive system integration + vehicle automation technology right is still a big challenge. This video talks about the possibilities. But to get there, OEMs still have a lot of work to do achieving a viable culture that addresses inherent tensions:
  • Cutting edge cloud software vs. life critical embedded systems
  • Role of automation vs. realistic expectations of human drivers
  • A shift from "recall" mentality to continuous improvement processes
  • Fast updates vs. assured safety integrity
  • Role of suppliers vs. OEM, especially for autonomous vehicle functions
  • Monetizing data vs. consumer rights
  • OEMs stepping up to the system integration challenges
  • Getting a regulatory approach that balances risks and benefits across all stakeholders
(Sadly, the video includes an incorrect statement that "95% to 96% of the accidents happen because of distracted driving" in the context of fatalities. Drivers are not perfect, but distracted driving only contributes to about 9% of fatalities per US DOT, about one-tenth of what was stated.)

Friday, October 7, 2022

Enhanced personal safety for autonomous vehicles

AV safety discussions often get quite technical. But there are aspects of safety that have a lot more to do with personal safety concerns. It is important that AV technology deployments enhance rather than degrade personal safety.

Person in parking lot at night -- Dall-e
Do autonomous vehicles improve personal safety compared to alternatives?

An important feature of a personally owned human-driven vehicle is having more control over personal safety. A locked private vehicle provides a measure of physical protection against potential threats to personal safety. In a single occupancy conventional vehicle the driver can make personal safety choices beyond the obvious one of not sharing a vehicle with a stranger as would be the case in a taxi or ride-share vehicle.[1] The availability of a single-occupancy AV might extend this safety benefit to those who cannot drive or do not have resources to own a private vehicle.

Example safety choices beyond just riding solo include debarking in an escort-provided portion of a parking lot, selecting routes that seem to present lower personal risk, and deciding not to exit the vehicle at a preselected destination location that turns out to look dangerous. To the degree that a single-occupant AV provides similar personal risk management features, riding solo in an AV might be safer than in a shared vehicle, including one with a human driver.

Personal safety is especially important to more vulnerable demographic segments, particularly when traveling alone, such as women, the elderly, and children. Also potentially at risk are identifiable minority groups in areas prone to abusive behavior based on race, gender, ethnicity, religion, or other factors. Beyond that, any AV user might have personal safety concerns, especially in areas with high crime rates.

Personal safety on shared AV mass transit vehicles will be an obvious concern as it is with crewed transit. On crewed mass transit the crew members can provide an additional measure of social supervision and deterrence. A potential move to smaller AV shared transit vehicles increases the opportunity for a passenger being isolated with a potential bad actor in a travel module, and complicates remote surveillance by multiplying the number of small passenger compartments being managed rather than fewer large compartments. Supervising dozens or hundreds of people on a single fully automated passenger train seems a more tractable problem (e.g., done with an on-train conductor) than remotely supervising dozens or hundreds of robotaxis shared by strangers.

Beyond the ride itself, there are also safety issues related to waiting for transport arrival and offloading. In particular, it will be important for vulnerable passengers to be able to change their destination at the end of the trip if local conditions at the destination seem too dangerous. Consider a city that requires using designated drop-off points of AV robotaxis. What should a passenger do if they do not like the looks of a group of people, potentially armed, waiting at the stop for them to get out?

A simple argument is to say that every automated low-speed shuttle will have an attendant. While attendants might be desirable and might prove necessary for a variety of reasons, requiring full-time staff on an automated vehicle that is smaller than a mass transit vehicle is largely at odds with the argument that AVs provide economic benefits due to not having to pay a person to be on board.

The question here is: will riding on an automated vehicle be as safe as riding in a vehicle with a human driver from a personal safety point of view?

[1] A popular meme goes something like this: Years ago we were told not to get into cars with strangers and not to talk to strangers on the Internet. Now we literally contact strangers via the Internet so we can get into their cars.            
While ride-share companies recognize that personal safety is a key issue, and put effort into improving it, personal safety needs more work. See Marshall 2019:
Also Saddiqui 2021:

Wednesday, October 5, 2022

Gatik Announcement -- Is it real safety? Or just AV safety theater?

Gatik just announced it has completed an extensive third-party safety review of its system as part of deploying fully driverless commercial operations in Canada. But the announcement raises many questions as to how much it really assures safety.

Gatik truck in Walmart livery

The autonomous vehicle safety arena is full of misinformation, disinformation, safety theater -- and players earnestly trying to do the right thing. Companies routinely employ ambiguous language, half-truths, and outright propaganda to deploy safety theater. But some companies use unambiguous statements of conformity to safety standards to show they are really doing safety. Which bucket does Gatik fall into?  Let's take a look at the signs from their press release.

Gatik claims that their third-party review covers safety and security. This was done with "a team of third-party experts." No mention of who these experts might have been, nor their qualifications. The gold standard is an accredited third party assessor such as TUV SUD (there are quite a few others as well).

For security they mention reasonable standards including SAE J3061, ISO/SAE 21434, and UNECE R155. It would be better to see them state "conformance" with these standards instead of just saying they were "covered" by the review. (Maybe they failed to conform as a result -- who knows?)  But at least this statement shows that the experts knew enough to look at these standards. So maybe OK, but hard to say.

For safety the only standard mentioned is SAE J3016 -- which is not a safety standard. In fact, only meeting the minimum requirements for the SAE Levels is not safe in practice (e.g., driver monitoring is not required, nor is any notification to the human driver that takeover is required after some types of failures). The safety analysis, such as it is, is clearly patterned after J3016, mentioning ODD and OEDR. 

There is a statement that "where they apply, the vehicle and ADS comply with safety relevant standards and best practices, such as those developed by SAE International and the International Organization for Standardization (ISO)."  No mention of ANSI/UL 4600, which was included by NHTSA as a highly relevant standard. Also, what do they mean by "where they apply" exactly? Other companies have gone on the record saying none of the AV-specific safety standards apply to their AV. So maybe Gatik means they aren't following safety standards at all.  Not even SAE J3018 for testing safety.

What I get out of reading the announcement is they hired some unidentified experts of unknown reputation, who likely had better credentials in security than safety. (Any bona fide safety experts would never pronounce that a system was "safe" based solely on testing results as indicated in the press release.) They took a look and say "sure, looks like it works." That's about it. (If there is more, we'd expect them to brag about it with some specificity, right?)  

If there is one thing I've learned in this industry is that companies will claim the strongest thing they think they can. A weak claim means a weak result. This is a very weak safety claim.

While I appreciate that Gatik publicly messages “Safety is at the heart of everything we do," this Gatik press release fairly screams safety theater. If they want us to believe their message is compelling, they need to do better. Some examples of ways they could provide a better statement of safety:

  • What exactly do you mean by "acceptable" safety?
  • Name the safety standards they "considered."   Were they ISO 26262, ISO 21448, ANSI/UL 4600, and SAE J3018 (all safety standards). These are the types of standards US DOT has already proposed for regulatory purposes, so they ought to be top of mind for any AV safety assessment.
  • Name the safety standards they actually conform to beyond "considering" and potentially not implementing.
  • Is there an Safety Management System (SMS)?  Didn't see it mentioned. This is safety 101, so you'd think they'd at least mention that.
  • Say who the external experts were so we can judge their reputation. Were they an accredited assessment organization? Were any of them actually qualified to opine on safety rather than security?
  • Explain how it is that a "rigorous suite of system as well as component level tests" can show safety. Because everyone would really like to know how you can do that for an AV. The safety standards are much more about engineering processes and safety engineering, with validation just being the tail of the safety dog. Certainly nothing I've seen indicates that validation-only safety assessment is possible for an AV.
Their announcement video talks about delivering against a value proposition. The only reason it gives for believing they are safe is ... saying they deliver safety and "manage risk."  That's it.  Gatik has not issued a VSSA, so no info to be had there.

Gatik -- your turn.  If you have a response I'll be happy to post it here for all to see:

... no response yet ...