Here is my talk from the October 2022 ISO 26262/SOTIF conference.
Assuming you follow the relevant standards (ISO 26262, ISO 21448, ANSI/UL 4600) in practice teams are finding the following topics difficult:
(Slides only at this time)
·
Restricted sensory information
such as potentially limited visual coverage, lack of audio information, lack of
road feel, and lack of other vehicle physical cues depending on the particular
vehicle involved. This could cause problems with reacting to emergency vehicle
sirens and reacting to physical vehicle damage that might be detected by a
physically present driver such as a tire blow-out, unusual vibration, or strange
vehicle noise. Lack of road feel might also degrade the driver’s ability to
remotely drive the vehicle to perform a fallback operation in an extreme
situation.
·
Delayed reaction time due to the
round-trip transmission lag. In some situations, tenths or even hundredths of
seconds of additional lag time in transmissions might make the difference
between a crash and a recovery from a risky situation.
·
The possibility of wireless
connectivity loss. Radio frequency interference or loss of a cell tower might
interrupt an otherwise reliable connection to the vehicle. Using two different
cell phone providers can easily have redundancy limitations due to shared
infrastructure such as cell phone towers,[1] cell tower
machine rooms (for some providers), and disruption of shared backhaul fiber bundles.[2] A
single infrastructure failure or localized interference can disrupt multiple
different connectivity providers to one or multiple AVs.
Achieving acceptable safety with remote operators
depends heavily on the duties of the remote operator. Having human operators
provide high-level guidance with soft deadlines is one thing: “Vehicle: I think
that flag holder at the construction site is telling me to go, but my
confidence is too low; did I get that right? Operator: Yes, that is a correct
interpretation.” However, depending on a person to take full control of
remotely driving a vehicle in real time with a remote steering wheel at speed
is quite another, and makes ensuring safety quite difficult.
A further challenge is the inexorable economic
pressure to have remote operators monitoring more than one vehicle. Beyond
being bad at boring automation supervision tasks, humans are also inefficient
at multitasking. Expecting a human supervisor to notice when an AV is getting
itself into a tricky situation is made harder by monitoring multiple vehicles.
Additionally, there will inevitably be a situation in which two vehicles under
control of a single supervisor will need concurrent attention when the operator
can only handle one AV in a crisis at a time.
There are additional legal issues to consider for
remote operators. For example, how does an on-scene police officer give a field
sobriety test to a remote operator after a crash if that operator is hundreds
of miles away – possibly in a different country? These issues must be addressed
to ensure that remote safety driver arrangements can be managed effectively.
Any claim of testing safety with a telepresent
operator needs to address the issues of restricted sensory information,
reaction time delays, and the inevitability of an eventual connectivity loss at
the worst possible time. There are also hard questions to be asked about the accountability
issues and law enforcement implications of such an approach.
A special remote monitoring concern is a safety
argument that amounts to the vehicle will notify a human operator when it needs
help, so there is no need for any human remote operator to continuously monitor
driving safety. Potentially the most difficult part of AV safety is ensuring
that the AV actually knows when it is in trouble and needs help. Any argument
that the AV will call for help is unpersuasive unless it squarely addresses the
issue of how it will know it is in a situation it has not been trained to
handle.
The source of this concern is that machine
learning-based systems are notorious for false confidence. In other words,
saying an ML-based system will ask for help when it needs it assumes that the
most difficult part to get right – knowing the system is encountering an
unknown unsafe condition – is working
perfectly during the testing being performed to see if, in fact, that most
difficult part is working. That type of circular dependency is a problem for
ensuring safety.
Even if such a system were completely reliable at
asking for help when needed, the ability of a remote operator to acquire
situational awareness and react to a crisis situation quickly is questionable.
It is better for the AV to have a validated capable of performing Fallback
operations entirely on its own rather than relying on a remote operator to jump
in to save the day. Before autonomous Fallback capabilities are trustworthy, a
human safety supervisor should continuously monitor and ensure safety.
Any remote operator road testing that claims the AV will inform the remote operator when attention is needed should be treated as an uncrewed road testing operation as discussed in book section 9.5.7. Any such AV should be fully capable of handling a Fallback operation completely on its own, and only ask a remote operator for help with recovery after the situation has been stabilized.
[1] For example, a cell tower
fire video shows the collapse of a tower with three antenna rows, suggesting it
was hosting three different providers.
See: https://www.youtube.com/watch?v=0cT5cXuyiYY
[2] While it is difficult to
get public admissions of the mistake of routing both a primary and backup critical
telecom service in the same fiber bundle, it does happen.
See: https://www.postindependent.com/news/local/the-goof-behind-losing-911-service-in-mays-big-outage/
Here is a Software Defined Vehicle video that covers a lot of ground. Car companies are all talking a big game about adding software to their vehicles, including big data, software updates, connectivity, and more. The possibilities are exciting, but you only have to read the news to know that the road to get there is proving bumpier than they'd like. (See this story too.)
Getting the mix of Silicon Valley software + automotive system integration + vehicle automation technology right is still a big challenge. This video talks about the possibilities. But to get there, OEMs still have a lot of work to do achieving a viable culture that addresses inherent tensions:AV safety discussions often get quite technical. But there are aspects of safety that have a lot more to do with personal safety concerns. It is important that AV technology deployments enhance rather than degrade personal safety.
An important feature of a personally owned human-driven vehicle is having more control over personal safety. A locked private vehicle provides a measure of physical protection against potential threats to personal safety. In a single occupancy conventional vehicle the driver can make personal safety choices beyond the obvious one of not sharing a vehicle with a stranger as would be the case in a taxi or ride-share vehicle.[1] The availability of a single-occupancy AV might extend this safety benefit to those who cannot drive or do not have resources to own a private vehicle.
Example safety choices beyond just riding solo
include debarking in an escort-provided portion of a parking lot, selecting
routes that seem to present lower personal risk, and deciding not to exit the
vehicle at a preselected destination location that turns out to look dangerous.
To the degree that a single-occupant AV provides similar personal risk
management features, riding solo in an AV might be safer than in a shared
vehicle, including one with a human driver.
Personal safety is especially important to more vulnerable
demographic segments, particularly when traveling alone, such as women, the elderly,
and children. Also potentially at risk are identifiable minority groups in
areas prone to abusive behavior based on race, gender, ethnicity, religion, or
other factors. Beyond that, any AV user might have personal safety concerns,
especially in areas with high crime rates.
Personal safety on shared AV mass transit vehicles will
be an obvious concern as it is with crewed transit. On crewed mass transit the
crew members can provide an additional measure of social supervision and
deterrence. A potential move to smaller AV shared transit vehicles increases
the opportunity for a passenger being isolated with a potential bad actor in a
travel module, and complicates remote surveillance by multiplying the number of
small passenger compartments being managed rather than fewer large compartments.
Supervising dozens or hundreds of people on a single fully automated passenger
train seems a more tractable problem (e.g., done with an on-train conductor) than
remotely supervising dozens or hundreds of robotaxis shared by strangers.
Beyond the ride itself, there are also safety
issues related to waiting for transport arrival and offloading. In particular,
it will be important for vulnerable passengers to be able to change their
destination at the end of the trip if local conditions at the destination seem
too dangerous. Consider a city that requires using designated drop-off points
of AV robotaxis. What should a passenger do if they do not like the looks of a
group of people, potentially armed, waiting at the stop for them to get out?
A simple argument is to say that every automated low-speed
shuttle will have an attendant. While attendants might be desirable and might
prove necessary for a variety of reasons, requiring full-time staff on an
automated vehicle that is smaller than a mass transit vehicle is largely at
odds with the argument that AVs provide economic benefits due to not having to
pay a person to be on board.
The question here is: will riding on an automated vehicle be as safe as riding in a vehicle with a human driver from a personal safety point of view?
[1] A popular meme goes
something like this: Years ago we were told not to get into cars with strangers
and not to talk to strangers on the Internet. Now we literally contact strangers
via the Internet so we can get into their cars.
While ride-share companies recognize that personal safety is a key issue, and
put effort into improving it, personal safety needs more work. See Marshall
2019: https://www.wired.com/story/criminologist-uber-crime-report-highly-alarming/
Also Saddiqui 2021:
https://www.washingtonpost.com/technology/2021/10/22/lyft-safety-report/
Gatik just announced it has completed an extensive third-party safety review of its system as part of deploying fully driverless commercial operations in Canada. But the announcement raises many questions as to how much it really assures safety.
The autonomous vehicle safety arena is full of misinformation, disinformation, safety theater -- and players earnestly trying to do the right thing. Companies routinely employ ambiguous language, half-truths, and outright propaganda to deploy safety theater. But some companies use unambiguous statements of conformity to safety standards to show they are really doing safety. Which bucket does Gatik fall into? Let's take a look at the signs from their press release.
Gatik claims that their third-party review covers safety and security. This was done with "a team of third-party experts." No mention of who these experts might have been, nor their qualifications. The gold standard is an accredited third party assessor such as TUV SUD (there are quite a few others as well).
For security they mention reasonable standards including SAE J3061, ISO/SAE 21434, and UNECE R155. It would be better to see them state "conformance" with these standards instead of just saying they were "covered" by the review. (Maybe they failed to conform as a result -- who knows?) But at least this statement shows that the experts knew enough to look at these standards. So maybe OK, but hard to say.
For safety the only standard mentioned is SAE J3016 -- which is not a safety standard. In fact, only meeting the minimum requirements for the SAE Levels is not safe in practice (e.g., driver monitoring is not required, nor is any notification to the human driver that takeover is required after some types of failures). The safety analysis, such as it is, is clearly patterned after J3016, mentioning ODD and OEDR.
There is a statement that "where they apply, the vehicle and ADS comply with safety relevant standards and best practices, such as those developed by SAE International and the International Organization for Standardization (ISO)." No mention of ANSI/UL 4600, which was included by NHTSA as a highly relevant standard. Also, what do they mean by "where they apply" exactly? Other companies have gone on the record saying none of the AV-specific safety standards apply to their AV. So maybe Gatik means they aren't following safety standards at all. Not even SAE J3018 for testing safety.
What I get out of reading the announcement is they hired some unidentified experts of unknown reputation, who likely had better credentials in security than safety. (Any bona fide safety experts would never pronounce that a system was "safe" based solely on testing results as indicated in the press release.) They took a look and say "sure, looks like it works." That's about it. (If there is more, we'd expect them to brag about it with some specificity, right?)
If there is one thing I've learned in this industry is that companies will claim the strongest thing they think they can. A weak claim means a weak result. This is a very weak safety claim.
While I appreciate that Gatik publicly messages “Safety is at the heart of everything we do," this Gatik press release fairly screams safety theater. If they want us to believe their message is compelling, they need to do better. Some examples of ways they could provide a better statement of safety:
Gatik -- your turn. If you have a response I'll be happy to post it here for all to see:
... no response yet ...
Waymo just published a study showing that they can outperform an unimpaired (NIEON) driver on a set of real-world crashes. That's promising, but it's only half the story. To be a safe driver, an Autonomous Vehicle (AV) must both not only be good -- but also not be bad. Those are two different things. Let me explain...
A Sept. 29, 2022 blog posting by Waymo explains their next step in showing that their automated driver can perform better than people at avoiding crashes. The approach is to recreate crashes that happened in the real world and show that their automated driver would have avoided them.
For this newest work they compare their driver to not just any human, but a Non-Impaired, with Eyes always ON the conflict (NIEON) driver. They correctly point out that no human is likely to be this good (you gotta blink, right?), but that's OK. The point is to have an ideal upper bound on human performance, which is a good idea.
Setting a goal that to be acceptably safe an AV should be at least as good as a NIEON driver makes sense. It cuts out all the debate about which human driver is being used as a reference (e.g., a tired 16 year old vs. a well rested professional limo driver 50 year old will have very different driving risk exposure).
Unsurprisingly Waymo does better than a NIEON on scenarios they know they will be running, because computers can react more quickly than people when they correctly sense, detect, and model what is going on in the real world. Doing so is not trivial, to be sure. As Waymo describes, this involves not just good twitch reflexes, but also realizing when things are getting risky and slowing down to give other road users more space, etc.
That is the "be a good driver" part, and kudos to Waymo and other industry players who are making progress on this. This is the promise of AV safety in action.
But to be safe in the real world, that is not enough. You also have to not be a bad driver. Doing better at things humans mess up is half the picture. The other half is not making "stupid" mistakes that humans would likely avoid.
AVs will surely have crashes that would be unlikely for a human driver to experience. Or will sometimes fail to be cautious when they should, and so on. While this did not lead to crashes, the infamous Waymo ride video showing construction zone cone confusion shows there is more work to be done in getting AVs to handle unusual situations. To its credit, in that scenario the Waymo vehicle did not experience a crash. But other uncrewed AVs are in fact having crashes due to misjudging traffic situations. And an automated test truck crashed into a safety barrier despite having a human safety driver due to an ill-considered initialization strategy that at least some would consider a rookie mistake (e.g., repeats a type of mistake made in Grand Challenge events -- should have known better).
It is good to see Waymo showing their automated driver can do well when it correctly interprets the situation it is in. That shows it is a potentially capable driver. We still need to see it is additionally not making mistakes in novel situations that aren't part of the human driver crash dataset. If Waymo can show NIEON level safety in both the knowns and the unknowns, that will be an impressive achievement.
The #1 ethical issue in autonomous vehicles is not the infamous Trolley Problem. It is the question of who gets to decide when it is OK to deploy a vehicle without a safety driver on public roads.
Consider a thought experiment which, if you follow AV industry news, you might recognize as not entirely hypothetical. You, the reader, are in charge of a company that needs to do a public road demonstration with no driver in an AV. You know that safety is not where you would like it to be. In fact, you have no safety case at all. You might not even have any real safety engineers on staff. But you have a smart, super-capable team. You have done a lot of test driving and it is going pretty well.
You intuitively figure it is more likely than not that you can pull off a one-time demo without a crash, and even less likely that a crash will kill someone. You figure you have something like 5 chances out of 6 of pulling off the demo with nobody getting hurt, and it is, in your mind, near certainty due to low-speed urban driving that any crash would avoid a fatality. For good measure maybe you plan to station employees near the demo site to shoo away any pedestrians and light mobility users who would be at increased risk of harm, and do the demo very late at night when roads are usually empty of other road users. Regulators are not in a position to influence your decision.
Your investors have told you they will pull the plug on your entire company if you do not demo by December 31st. Right now, it is the first week of December, and it is time to decide what to do. If the investors pull the plug at the end of the month, you lose perhaps $1B in personal equity you hope to net in next year’s public offering. And all your employees will lose their equity as well as their jobs. This will also end a journey you have spent your life on to build and deploy a truly self-driving car.
Further negotiations with the investors are not possible. It is time to decide. That leaves you three main options:
• Case 1: The AV company does not do the demo because it cannot assure a PRB level of safety. The company runs out of money and folds. This option kills the company.
• Case 2: The AV company does the demo and harms a road user. This might or might not result in termination of funding depending on the optics of the crash (perhaps a pedestrian victim can be blamed for jaywalking, being impaired, or having low societal status; maybe all three). You think minor harm is more likely than a fatality, and you will have lots of money available to pay off a potential victim to keep quiet. You will not pre-announce the demo, so you feel able to control the narrative if something goes wrong. The company and the mission go on unless there is a truly unlucky break during that one demo/test session that cannot be cleaned up. Even Uber ATG kept going for a while after a really bad crash, and you know in your heart that your team is better.
• Case 3: The AV company does the demo and gets lucky, not harming any other road users. The company meets its milestone and gets more funding. This the most likely case, and it would be a perfect victory.
Given this setup doing the demo is clearly the best financial bet for the company. Probably you will get lucky with a positive outcome. No harm will be done, and the demo can be said to be safe under the culturally dominant no harm/no foul principle. But if the demo is skipped due to safety, the company is sure to die, and the decision maker is out a billion dollars.
Even if you get unlucky, the cost of a few million dollar settlement pales in comparison with the billions of dollars on the table. Really – you might think – a payout is just the cost of doing business. And even if the crash optics get out of control and the startup company folds, the investors have hedged their bets and the team can simply move to another company and try again. Pretty much everyone will do fine. Except for the victim, if there is one.
This is how demo milestones incentivize deploying systems when the calendar and funding flow says it is time for a demo rather than when the demo is known to be acceptably safe. After all, it is someone else who is injured or dies, not the decision maker. And a billion dollars is a ton of money. And probably it will be fine. After all, you think, only other companies kill pedestrians.
Summary: The human role in FMVSS should not be removed for exemption request because there is no driver. Rather, what should be stated is how the driver is being replaced in that safety role as a matter of system design.
These comments were filed regarding the General Motors Petition for Temporary Exemption from FMVSS -- Federal Motor Vehicle Safety Standards (https://www.regulations.gov/document/NHTSA-2022-0067-0002) However, they are likely to apply to any autonomous vehicle that does not have conventional driver controls or operates in a mode which does not have an officially designated driver.
It is good to see companies working to advance the potential benefits of autonomous vehicle technology. However, it is also important for NHTSA to ensure public safety when such technology is deployed on public roads.
There will
almost certainly need to be an auxiliary controller to command motion of
vehicles without normally accessible driver controls. To the extent that these
are in-vehicle wired controls the question arises as to their suitability for
safe vehicle control (e.g., using a "Playstation" type video game
controller) and whether they can be improperly activated during autonomous
operation. To the extent that these are wirelessly connected controls there are
serious security issues that must be considered. In the absence of normal
driver controls any petitioner should be required to justify the safety and
security of its maintenance and supplemental human-operated vehicle control
strategy beyond more general claims of working on cybersecurity.
FMVSS
101: Petitioner
should demonstrate that all telltales that are intended to prompt human driver
reaction similarly prompt a relevant ADS reaction to an indicated exceptional
condition. If no such demonstration is provided, NHTSA
would simply be taking Petitioner’s word that the telltale functions are
implemented in a way that provides equivalent safety. (This also applies to
other telltales such as FMVSS No. 126, 138, etc.)
FMVSS
102: While the ADS
is said to control the transmission, there will be times in which humans need
to know the state of the transmission for safety, especially whether the
vehicle is in park or not. This includes emergency responders and maintenance
personnel to ensure that the vehicle will not move unexpectedly when it is
operating in a degraded or post-crash state. NHTSA should evaluate the
suitability of a passenger video display for reliably displaying vehicle park
state to non-passenger stakeholders.
FMVSS
104: From the
NTHSA summary: "For GM's petition
for exemption from portions of FMVSS No. 104, GM argues that the purpose and
intent of the safety standard is obviated by the Origin's sensor system design." GM's argument is ridiculous from a technical
point of view. (Their technical
disclosure is more substantive, but this type of lawyer argument in their
summary serves to undermine their filing's credibility.) In the context of an ADS, the intent of the
safety standard is not to permit a human driver's eyes to see out the
windshield. Rather, it is to ensure that all visual sensors can see through the
windshield (if interior mounted), or through their lenses and weather covers as
appropriate. It is completely foreseeable that camera and lidar operation will
be impaired by adverse operational conditions. Moreover, the problem is likely
to be worse than for human drivers because even a relatively small bug splat on
a camera lens will form a visual obstruction covering a much larger fraction of
the field of view compared to the same bug impacting the windshield of a
human-driven vehicle, because human drivers can and do simply move their heads
a bit to see past windshield obstructions.
GM states that it keeps its sensors clean, but provides no objective way
for NHTSA to validate the claim.
FMVSS
111: From the
NHTSA summary: "GM points out that the purpose and intent of FMVSS No. 111
is based on human perception and visibility so there is no operational safety
need for these requirements when applied to a vehicle driven by an ADS." This
argument amounts to the common fallacy of: "humans have limitations and
therefore computers will be perfect." (Anyone paying attention knows that
computers fail, often spectacularly – just in different ways than humans.) There
are safety requirements behind FMVSS 111 that are not about "human
perception and visibility" but rather, for example, not running over
children that are not visible to the driver sensors. Again, GM's lawyer-phrased argument
undermines the credibility of their filing. In addressing this request for
exemption the petitioner should explain why it is the case that small children
will be protected from being run over at least as well as by a human driver using
a rear view camera as one example. Again, the technical substance is better,
but leaves a big open question. GM provides pictures indicating that cameras
can potentially see areas required. However, there is no data supporting that
those cameras will correctly sense and classify children close to the vehicle
with high accuracy, especially at night when the vehicle automation might rely
more heavily on roof-top mounted lidar sensors. The safety requirement is not
camera visibility, but rather that the driver+camera will avoid hitting
children. Given apparent issues with camera-based AEB systems having trouble
sensing and avoiding hitting children this is an issue that needs to be taken
seriously.
FMVSS
201: From the
NHTSA summary: "GM argues that sun visors are not necessary because the
Origin is not operated by a human driver, and the ADS does not use the
windshield for visibility." Again, limitations of human drivers are not
the real point for ADS safety. It is well known that cameras struggle when looking
into the sun. GM should explain how they handle that problem (sun visors are a
way human drivers can handle it) and now NHTSA can validate that sun glare is
handled safely. GM states it is not a
problem for this system, hinting that perhaps they rely on lidar+radar instead
of a camera in such situations, but that raises additional concerns. GM should
provide a way for NHTSA to validate that sun glare does not present undue risk.
Validation: For all requested exemptions it is
not enough to simply say more or less "trust us, we have figured this
out" for an ADS equipped vehicle than it is for a conventional vehicle. If
"trust us" were enough, there would be no need for the carefully
design tests in FMVSS for any vehicle. Rather, petitioners should explain (a)
why the broader safety objective of the FMVSS element being waived is still
being met (e.g., not hitting children vs. field of view), and (b) a way that
NHTSA can validate for itself that the safety objective is in fact being met. See: Koopman, “How to keep self-driving cars safe
when no one is watching for dashboard warning lights,” The Hill, June
30, 2018. https://thehill.com/opinion/technology/394945-how-to-keep-self-driving-cars-safe-when-no-one-is-watching-for-dashboard/
Regarding
Public Interest Considerations:
Safety Benefit: It is essential to keep in mind
when evaluating ADS petitions that potential safety benefits are purely
aspirational. There is not yet any proof that ADS-equipped vehicles will be
even as safe as a comparable active-safety-equipped human-driven vehicle using
any currently available or near term deployable technology. While supporting
the development of technology that has the potential to reduce harm is
laudable, this should not be done at the cost of increased near-term harm.
Putting road users at increased risk today because someday, maybe, perhaps,
autonomous vehicles might be safer is unacceptable. NHTSA should be in the
safety business, not the hopium business. That means exemptions should require
a concrete justification of deployed safety that is not weakened by promises of
ADS perhaps increasing safety in the future.
Environmental
Impact: Requiring
environmental impact data reporting is an excellent idea. It is important to be
mindful of indirect environmental and safety consequences of ADS-equipped
vehicle adoption. One example is that (presumably) decreased cost per mile is
prone to increasing demand, resulting in a potential increase in overall
emissions if small ADS-equipped vehicles are used for more trips. Both human harm and environmental damage
could be increased if ADS-equipped vehicles draw passenger loads away from mass
transit even if the total number of user-miles across all modes remains the
same. Moreover, use of uncrewed ADS-equipped vehicles can increase both total
pedestrian harm and emissions as well as road congestion if that leads to a
large increase in on-demand delivery services.
Equity: As NHTSA notes in their summary,
petitioners for exemptions extoll the potential benefits to expand
transportation options to under-served areas and customers with disabilities.
It is important to point out that these populations can be served with
traditional human-operated vehicles as well, so this is not a fundamentally new
capability, but rather an argument that cheaper costs will increase vehicle
availability (perhaps at the cost of increased road congestion). Note that
vehicle sharing does not require autonomy, nor does use of electric vehicles. To
the extent that the exemptions are for "robo-taxi" and shuttle type
applications, petitioners should not only say that their technology might
possibly help disadvantaged populations, but also how they plan to serve such
populations directly as a result of the exemption. Promising to serve such
populations in an exemption petition but then fielding a single token
wheelchair-accessible vehicle or only deploying in a rich urban center would
result in such statements of providing equitable transportation being only
aspirational rather than directly related to the exemption decision. Credit
should not be given for empty promises in evaluating an application.
Safety: While considering economic impacts
might be relevant as NHTSA notes in their summary, it is essential to remind
all stakeholders that the "S" in NHTSA stands for "Safety."
It is imperative that economic motivations not over-ride the NHTSA mission to
ensure safety. Safety must come first, and not be overwhelmed by economic
pressure.
Standards: Petitioners who want exemptions
from FMVSS provisions should have an alternate method of ensuring that a
rigorous engineering approach has been used to ensure safety. Such an approach
was proposed by NHTSA itself in an ANRPM (Docket No. NHTSA-2020-0106 / RIN
2127-AM15 Framework for Automated Driving System Safety https://www.regulations.gov/docket/NHTSA-2020-0106 ). NHTSA should respond to
comments from that ANPRM and work further on setting up such a framework that
could provide and an alternate basis for ADS-equipped vehicle regulation.
SMS: NHTSA should require all
petitioners to implement an effective Safety Management System (SMS) as a
condition of receiving an exemption, with SMS data included in periodic reports
for the life of any vehicle.
Crash
Reporting: NHTSA
should continue to require crash reporting as proposed. The definition of a
crash should be scoped to include: (a) any contact between the vehicle and
another road user regardless of severity; (b) any contact between the vehicle
and other obstacles or infrastructure regardless of severity; (c) any
"near hit" that required evasive action from another road users
(e.g., pedestrian jumping back on sidewalk to avoid being hit while in a
crosswalk); (d) any substantive moving violation (e.g., running a red
light). Any responsible tester/operator
of an ADS will have data already compiled on all these events as part of their
SMS to ensure that they are operating safely on public roads, so the reporting
burden should be minimal. These reports
should be required for the life of any exempt vehicle.
Commenter
Qualifications: Prof.
Philip Koopman is an internationally recognized expert on Autonomous Vehicle
(AV) safety whose work in that area spans over 25 years. He is also actively
involved with AV policy and standards as well as more general embedded system
design and software quality. His pioneering research work includes software
robustness testing and run time monitoring of autonomous systems to identify
how they break and how to fix them. He has extensive experience in software
safety and software quality across numerous transportation, industrial, and
defense application domains including conventional automotive software and
hardware systems. He was the principal technical contributor to the UL 4600
standard for autonomous system safety issued in 2020. He is a faculty member of
the Carnegie Mellon University ECE department where he teaches software skills
for mission-critical systems. In 2018 he was awarded the highly selective
IEEE-SSIT Carl Barus Award for outstanding service in the public interest for
his work in promoting automotive computer-based system safety. In 2022 he was
named to the National Safety Council's Mobility Safety Advisory Group. He is
the author of the book How Safe is Safe Enough: measuring and predicting
autonomous vehicle safety (2022). https://users.ece.cmu.edu/~koopman/
Continuous Learning Approach to Safety Engineering
Rolf Johansson & Philip Koopman / CARS @EDCC 2022
Abstract:
A phase change moment is upon us as the automotive industry moves from conventional to highly automated vehicle operation, with questions about how to assure safety. Those struggles underscore larger issues with current functional safety standards in terms of a need to strengthen the traceability between required practices and safety outcomes. There are significant open questions regarding both the efficiency and effectiveness of standards-based safety approaches, including whether some engineering practices might be dropped, or whether others must be added to achieve acceptable safety outcomes. We believe that rather than an incremental approach, it is time to rethink how safety standards work. We propose that real-world field feedback for an initially safe deployment should support a DevOps-style continuous learning approach to lifecycle safety. Safety engineering should trace from a safety case to engineering practices to safety outcomes. Such an approach should be incorporated into future safety standards s (including ISO 26262) to improve safety engineering efficiency and effectiveness.
A WSJ scoop confirms the TuSimple April 6th crash video and FMCSA investigation. See: https://www.wsj.com/articles/self-driving-truck-accident-draws-attention-to-safety-at-tusimple-11659346202?st
This article validates what we saw in this video showing the crash: https://lnkd.in/ebGF4Quv
TuSimple blames human error, but it sounds more like a #moralcrumplezone situation, with serious safety culture concerns if the story reflects what is actually going on there.A left turn command was pending from a previous disengagement minutes before. When re-engaged the system started executing a sharp left turn while at 65 mph on a multi-lane highway. That resulted in crossing another traffic lane, a shoulder, and then hitting a barrier.
A proposal: Any mention of blame in an autonomous vehicle incident report should immediately discredit the reporting entity's claims of caring about safety and tar them with the brush of "safety theater"
PA HB 2398 would legalize autonomous vehicles (AVs) without human drivers in Pennsylvania. Having passed the PA House, it is pending in the PA Senate Transportation Committee. While the bill has improved, my 25 years of experience working on AV safety at Carnegie Mellon University leave me with significant remaining concerns:
- A municipal preemption clause would prevent Pittsburgh from restricting the testing of immature self-driving vehicle technology in active school zones and other high risk locations.
- A loophole regarding vehicles “approved for noncommercial use” apparently exempts most AVs from certification when using conventional vehicle retrofits, potentially rendering the bill toothless.
- Test drivers are not required to conform to an established industry standard for testing safety as done elsewhere in the US. Argo AI is the sole company conforming to the relevant safety standard: SAE J3018.
- The recent National Highway Traffic Safety Administration release of AV crash data makes it clear that crashes will happen. The bill should require AV testers to attest that their technology is acceptably safe. Perfect safety might be unrealistic, but AV companies should at least promise on the record that their testing will be no more dangerous than human driven vehicles.
The current bill leaves Commonwealth constituents unnecessarily vulnerable on public roads. In exchange for placing road users at risk from this work-in-progress technology, we at least deserve to have these issues fixed before the bill is made into law.
Philip Koopman, Ph.D.
Squirrel Hill neighborhood, Pittsburgh PA
----
Pittsburgh Post-Gazette article was published here: https://www.post-gazette.com/news/transportation/2022/06/22/self-driving-vehicles-expanded-testing-safety-concerns-emergency-drivers-pennsylvania-house/stories/202206220121
Update 6/30/22: the PA Senate Transportation Committee updated SB 965 to largely include language from the PA House bill. It does seem to have addressed two of my points to a degree:
This past week the PA House Transportation Committee significantly revised and then passed a bill granting sweeping authority to operate Highly Automated Vehicles (HAVs) in the Commonwealth of Pennsylvania. That includes light vehicles, heavy trucks, and platoons of heavy trucks. This bill has evolved over time and seems a better candidate for a final law than the older, much more problematic Senate bill.
It has some good points compared to what we've seen in other states, such as an insurance minimum of $1M, and placing PennDOT in regulatory control instead of Public Safety. By way of contrast, in other states the State Police are in charge of regulating (they have no real ability to do so, and realize this, but the HAV industry pushed to have it this way), and insurance minimums are as low as $25K or $50K. So we're doing better than some other states.
The PA bill establishes an advisory committee, but it is unclear whether it will have much power, and its current mandate is to report benefits of HAVs without being tasked to report on any public safety concerns (or benefits).
However, a great number of issues identified in earlier versions have not been addressed. A very significant concern is a municipal preemption clause. For example, cities are prevented from curtailing testing of experimental, immature HAVs in school zones, even with no safety driver in the vehicle.
There are a number of other serious concerns unaddressed by this bill especially in the area of safety, but also with regard to compensation, transparency, inclusion, and non-discrimination: see Five Principles for Regulation of Highly Automated Vehicles.
A particular problematic issue boils down to who goes to jail if an HAV has a software defect that results in driving behavior that would, for a human driver, result in criminal penalties. This bill is at least clear about the "certificate holder" being on the hook, whereas other states are silent on this topic. However, it is unclear if a certificate holder who might have no understanding of HAV software and no ability to influence HAV operational safety is the right person to be sending to jail for reckless driving by an HAV that results in deaths. (Yes, this is a difficult problem. But the HAV industry has had years and years to address concerns such of this. Apparently their plan is to deflect blame away from the tech companies and onto whoever ends up holding the bag as a certificate holder.)
The manner of how HAV bills are being pushed through the legislature is also extremely disappointing. The Senate rammed through a bill that was not disclosed until the last minute with no public hearing. To its credit the House did have a public hearing on its initial bill. However, this very significant modification was kept secret until the Transportation Committee meeting voted it through along party lines. The industry certainly knows what is in the bills and amendments well in advance, because we have had public events in which they were thanked for helping author them. If they really believed that public safety was #1 and stakeholder engagement mattered, the industry would not be resorting to releasing legislation in the dead of night to ram it through votes.
I fully expect this will be pushed through both House and Senate in the most industry-friendly way that can be managed. The PA Governor has already promised to sign HAV legislation. We're going to be stuck with regulations that disproportionately favor the industry so that they can attempt to reap the IPO and SPAC compensation rewards of chasing a trillion dollar market while exporting risks of public road testing to other road users. (Some companies are doing better than others on safety, but the industry as a whole as, for example, represented by AVIA is quite clearly all about the $$$ and not really about public safety.)
It is sad to see legislators seduced by the "jobs and economic opportunity" mantra of the HAV industry while most companies are merely paying lip service to safety. But I guess this is how it will be until we have a sufficiently high number of crashes and other adverse newsworthy events to put on public pressure to do better.
Note: there is one clause that is a potentially HUGE issue. Page 29 lines 16-18 appear to exempt any vehicle that is not strictly commercial (in practice anything except heavy trucks) from the requirement for a PennDOT certificate. It is unclear whether this is an intentional loophole or just a drafting mistake. Either way it should be fixed.
