Here is my talk from the October 2022 ISO 26262/SOTIF conference.
Assuming you follow the relevant standards (ISO 26262, ISO 21448, ANSI/UL 4600) in practice teams are finding the following topics difficult:
(Slides only at this time)
·
Restricted sensory information
such as potentially limited visual coverage, lack of audio information, lack of
road feel, and lack of other vehicle physical cues depending on the particular
vehicle involved. This could cause problems with reacting to emergency vehicle
sirens and reacting to physical vehicle damage that might be detected by a
physically present driver such as a tire blow-out, unusual vibration, or strange
vehicle noise. Lack of road feel might also degrade the driver’s ability to
remotely drive the vehicle to perform a fallback operation in an extreme
situation.
·
Delayed reaction time due to the
round-trip transmission lag. In some situations, tenths or even hundredths of
seconds of additional lag time in transmissions might make the difference
between a crash and a recovery from a risky situation.
·
The possibility of wireless
connectivity loss. Radio frequency interference or loss of a cell tower might
interrupt an otherwise reliable connection to the vehicle. Using two different
cell phone providers can easily have redundancy limitations due to shared
infrastructure such as cell phone towers,[1] cell tower
machine rooms (for some providers), and disruption of shared backhaul fiber bundles.[2] A
single infrastructure failure or localized interference can disrupt multiple
different connectivity providers to one or multiple AVs.
Achieving acceptable safety with remote operators
depends heavily on the duties of the remote operator. Having human operators
provide high-level guidance with soft deadlines is one thing: “Vehicle: I think
that flag holder at the construction site is telling me to go, but my
confidence is too low; did I get that right? Operator: Yes, that is a correct
interpretation.” However, depending on a person to take full control of
remotely driving a vehicle in real time with a remote steering wheel at speed
is quite another, and makes ensuring safety quite difficult.
A further challenge is the inexorable economic
pressure to have remote operators monitoring more than one vehicle. Beyond
being bad at boring automation supervision tasks, humans are also inefficient
at multitasking. Expecting a human supervisor to notice when an AV is getting
itself into a tricky situation is made harder by monitoring multiple vehicles.
Additionally, there will inevitably be a situation in which two vehicles under
control of a single supervisor will need concurrent attention when the operator
can only handle one AV in a crisis at a time.
There are additional legal issues to consider for
remote operators. For example, how does an on-scene police officer give a field
sobriety test to a remote operator after a crash if that operator is hundreds
of miles away – possibly in a different country? These issues must be addressed
to ensure that remote safety driver arrangements can be managed effectively.
Any claim of testing safety with a telepresent
operator needs to address the issues of restricted sensory information,
reaction time delays, and the inevitability of an eventual connectivity loss at
the worst possible time. There are also hard questions to be asked about the accountability
issues and law enforcement implications of such an approach.
A special remote monitoring concern is a safety
argument that amounts to the vehicle will notify a human operator when it needs
help, so there is no need for any human remote operator to continuously monitor
driving safety. Potentially the most difficult part of AV safety is ensuring
that the AV actually knows when it is in trouble and needs help. Any argument
that the AV will call for help is unpersuasive unless it squarely addresses the
issue of how it will know it is in a situation it has not been trained to
handle.
The source of this concern is that machine
learning-based systems are notorious for false confidence. In other words,
saying an ML-based system will ask for help when it needs it assumes that the
most difficult part to get right – knowing the system is encountering an
unknown unsafe condition – is working
perfectly during the testing being performed to see if, in fact, that most
difficult part is working. That type of circular dependency is a problem for
ensuring safety.
Even if such a system were completely reliable at
asking for help when needed, the ability of a remote operator to acquire
situational awareness and react to a crisis situation quickly is questionable.
It is better for the AV to have a validated capable of performing Fallback
operations entirely on its own rather than relying on a remote operator to jump
in to save the day. Before autonomous Fallback capabilities are trustworthy, a
human safety supervisor should continuously monitor and ensure safety.
Any remote operator road testing that claims the AV will inform the remote operator when attention is needed should be treated as an uncrewed road testing operation as discussed in book section 9.5.7. Any such AV should be fully capable of handling a Fallback operation completely on its own, and only ask a remote operator for help with recovery after the situation has been stabilized.
[1] For example, a cell tower
fire video shows the collapse of a tower with three antenna rows, suggesting it
was hosting three different providers.
See: https://www.youtube.com/watch?v=0cT5cXuyiYY
[2] While it is difficult to
get public admissions of the mistake of routing both a primary and backup critical
telecom service in the same fiber bundle, it does happen.
See: https://www.postindependent.com/news/local/the-goof-behind-losing-911-service-in-mays-big-outage/
Here is a Software Defined Vehicle video that covers a lot of ground. Car companies are all talking a big game about adding software to their vehicles, including big data, software updates, connectivity, and more. The possibilities are exciting, but you only have to read the news to know that the road to get there is proving bumpier than they'd like. (See this story too.)
Getting the mix of Silicon Valley software + automotive system integration + vehicle automation technology right is still a big challenge. This video talks about the possibilities. But to get there, OEMs still have a lot of work to do achieving a viable culture that addresses inherent tensions:AV safety discussions often get quite technical. But there are aspects of safety that have a lot more to do with personal safety concerns. It is important that AV technology deployments enhance rather than degrade personal safety.
An important feature of a personally owned human-driven vehicle is having more control over personal safety. A locked private vehicle provides a measure of physical protection against potential threats to personal safety. In a single occupancy conventional vehicle the driver can make personal safety choices beyond the obvious one of not sharing a vehicle with a stranger as would be the case in a taxi or ride-share vehicle.[1] The availability of a single-occupancy AV might extend this safety benefit to those who cannot drive or do not have resources to own a private vehicle.
Example safety choices beyond just riding solo
include debarking in an escort-provided portion of a parking lot, selecting
routes that seem to present lower personal risk, and deciding not to exit the
vehicle at a preselected destination location that turns out to look dangerous.
To the degree that a single-occupant AV provides similar personal risk
management features, riding solo in an AV might be safer than in a shared
vehicle, including one with a human driver.
Personal safety is especially important to more vulnerable
demographic segments, particularly when traveling alone, such as women, the elderly,
and children. Also potentially at risk are identifiable minority groups in
areas prone to abusive behavior based on race, gender, ethnicity, religion, or
other factors. Beyond that, any AV user might have personal safety concerns,
especially in areas with high crime rates.
Personal safety on shared AV mass transit vehicles will
be an obvious concern as it is with crewed transit. On crewed mass transit the
crew members can provide an additional measure of social supervision and
deterrence. A potential move to smaller AV shared transit vehicles increases
the opportunity for a passenger being isolated with a potential bad actor in a
travel module, and complicates remote surveillance by multiplying the number of
small passenger compartments being managed rather than fewer large compartments.
Supervising dozens or hundreds of people on a single fully automated passenger
train seems a more tractable problem (e.g., done with an on-train conductor) than
remotely supervising dozens or hundreds of robotaxis shared by strangers.
Beyond the ride itself, there are also safety
issues related to waiting for transport arrival and offloading. In particular,
it will be important for vulnerable passengers to be able to change their
destination at the end of the trip if local conditions at the destination seem
too dangerous. Consider a city that requires using designated drop-off points
of AV robotaxis. What should a passenger do if they do not like the looks of a
group of people, potentially armed, waiting at the stop for them to get out?
A simple argument is to say that every automated low-speed
shuttle will have an attendant. While attendants might be desirable and might
prove necessary for a variety of reasons, requiring full-time staff on an
automated vehicle that is smaller than a mass transit vehicle is largely at
odds with the argument that AVs provide economic benefits due to not having to
pay a person to be on board.
The question here is: will riding on an automated vehicle be as safe as riding in a vehicle with a human driver from a personal safety point of view?
[1] A popular meme goes
something like this: Years ago we were told not to get into cars with strangers
and not to talk to strangers on the Internet. Now we literally contact strangers
via the Internet so we can get into their cars.
While ride-share companies recognize that personal safety is a key issue, and
put effort into improving it, personal safety needs more work. See Marshall
2019: https://www.wired.com/story/criminologist-uber-crime-report-highly-alarming/
Also Saddiqui 2021:
https://www.washingtonpost.com/technology/2021/10/22/lyft-safety-report/
Gatik just announced it has completed an extensive third-party safety review of its system as part of deploying fully driverless commercial operations in Canada. But the announcement raises many questions as to how much it really assures safety.
The autonomous vehicle safety arena is full of misinformation, disinformation, safety theater -- and players earnestly trying to do the right thing. Companies routinely employ ambiguous language, half-truths, and outright propaganda to deploy safety theater. But some companies use unambiguous statements of conformity to safety standards to show they are really doing safety. Which bucket does Gatik fall into? Let's take a look at the signs from their press release.
Gatik claims that their third-party review covers safety and security. This was done with "a team of third-party experts." No mention of who these experts might have been, nor their qualifications. The gold standard is an accredited third party assessor such as TUV SUD (there are quite a few others as well).
For security they mention reasonable standards including SAE J3061, ISO/SAE 21434, and UNECE R155. It would be better to see them state "conformance" with these standards instead of just saying they were "covered" by the review. (Maybe they failed to conform as a result -- who knows?) But at least this statement shows that the experts knew enough to look at these standards. So maybe OK, but hard to say.
For safety the only standard mentioned is SAE J3016 -- which is not a safety standard. In fact, only meeting the minimum requirements for the SAE Levels is not safe in practice (e.g., driver monitoring is not required, nor is any notification to the human driver that takeover is required after some types of failures). The safety analysis, such as it is, is clearly patterned after J3016, mentioning ODD and OEDR.
There is a statement that "where they apply, the vehicle and ADS comply with safety relevant standards and best practices, such as those developed by SAE International and the International Organization for Standardization (ISO)." No mention of ANSI/UL 4600, which was included by NHTSA as a highly relevant standard. Also, what do they mean by "where they apply" exactly? Other companies have gone on the record saying none of the AV-specific safety standards apply to their AV. So maybe Gatik means they aren't following safety standards at all. Not even SAE J3018 for testing safety.
What I get out of reading the announcement is they hired some unidentified experts of unknown reputation, who likely had better credentials in security than safety. (Any bona fide safety experts would never pronounce that a system was "safe" based solely on testing results as indicated in the press release.) They took a look and say "sure, looks like it works." That's about it. (If there is more, we'd expect them to brag about it with some specificity, right?)
If there is one thing I've learned in this industry is that companies will claim the strongest thing they think they can. A weak claim means a weak result. This is a very weak safety claim.
While I appreciate that Gatik publicly messages “Safety is at the heart of everything we do," this Gatik press release fairly screams safety theater. If they want us to believe their message is compelling, they need to do better. Some examples of ways they could provide a better statement of safety:
Gatik -- your turn. If you have a response I'll be happy to post it here for all to see:
... no response yet ...
