What Happens After The Industry's JohnnyCab Adventure?

Recent news has people questioning whether autonomous vehicles are viable. Promises that victory is right around the corner are too optimistic. But it's far too early to declare defeat. There is a lot to process. But a pressing technology roadmap question is: if robotaxis aren't really the answer, what happens next for passenger vehicles?

A robotoxi ride that did not go as expected. (Total Recall 1990)

We can expect OEMs to double down on their Level 2è2+è3 strategies. But there is a very real risk of a race to the bottom as companies scramble to deploy shiny automation technology while skipping over reasonable, industry-created safety practices. A subtle, yet crucial, point is that asking a driver to supervise a driver assistance system is a completely different situation than putting a civilian car owner in the position of being an untrained autonomous vehicle test driver.

Where are we now?

The recent demise of Argo AI has made it crystal clear that Big Auto is pivoting away from robotaxis. Big Auto has kept plugging away at driver assistance systems while hedging its bets by taking stakes in Level 4 robotaxi companies. Those Level 4 companies were carefully kept at arms length against the day it was time to close out those hedges. And here we are.

Other companies continue to work on robotaxis -- most notably Waymo, Cruise, and some players in China that are carrying passengers on public roads. Waymo seems unconstrained by runway length. Presumably Cruise has a year-end milestone as they did in 2020 and 2021, and we'll see how that goes. Meanwhile Tesla continues to celebrate anniversaries of its promise to put a million robotaxis on the road within a year. Heavy trucks, parcel delivery, and low speed shuttles are also part of the mix, but face their own challenges that are beyond the scope of this discussion.

For now let's entertain the possibility that robotaxis are not going to happen anytime soon. It seems likely to take years of hammering away at the heavy tail of edge cases to get there. Sure, some of us can get a demo ride in a Taxi of Tomorrow in a couple cities. But let's assume that robotaxis are more of a Disney-esque experience than a practical tool for profitable mobility at scale anytime soon.

What happens next?

We can expect to see OEMs double down on evolutionary strategies. Ford has pretty much said as much, promising a Level 3 system is on the way. It will probably be talked about as climbing up the SAE Levels from 2 to 3 to 4, which has been their narrative all along. That narrative has significant issues because it makes the common mistake of using SAE levels as a progressive ladder to climb -- which it is definitely not. But it will be the messaging framework nonetheless.

In the mix are several options:

• Level 2 highway cruising systems that are already on the road
• Level 2+ add-ons, but with the driver still responsible for safety
• Traffic jam pilot as the first step to drivers taking attention off the road
• Level 3/4 capabilities beyond traffic jams (harder than it might seem)
• Abuse of Level 2/2+ designations to evade regulation

We'll see all of these in play. Each option comes with its own challenges.

Level 2 systems: highway cruising

The general idea is that first you build an SAE Level 2 system that has a speed+steering cruise control capability (automated speed control and automated lane keeping). This is intended for use on highways and other situations involving well behaved, in-lane travel. This functionality is already widely available on new high-end vehicles. The driver is responsible for continuously monitoring the car's performance and the road conditions, and intervening instantly if necessary -- whether the car asks for intervention or not.

A significant challenge for these systems is ensuring that drivers remain attentive in spite of inevitable automation complacency. This requires a highly capable driver monitoring system (DMS) that, for example, tracks where the driver is looking to ensure constant monitoring of the situation on the roadway.

Performance standards for monitoring driver attentiveness are in their infancy, and capabilities of the DMS on different vehicles varies dramatically. It is pretty clear that a camera-based system is much more effective than steering wheel touch sensors. And a camera that can see the driver's eye gaze direction, including having infrared emitters for night operation, is likely to be more effective than one that can't.

Another crucial safety feature that should be implemented is restricting operation of Level 2 features to the conditions they are designed to be safe in. The SAE J3016 taxonomy standard makes this totally optional -- which is one of the many reasons J3016 is not a safety standard, and why no government should write the SAE Levels into their safety regulations. But if the automation is supposed to only be used on divided highways with no cross-streets, then it should refuse to engage unless that is the road type it is being used on. Some companies restrict their vehicle to operation on specific roads, but others do not.

NTSB has made a number of recommendations to improve DMS capability and related Level 2 functionality, but the industry is still working on getting there.

Vehicle automation proponents have been beating the drum of potential safety benefit for years -- to the point that many take safety benefits as an article of faith. However, the reality is that there is no credible evidence that Level 2 capabilities make systems safer. There is plenty of propaganda to be sure, but much of that is based on questionable comparisons of very capable high-end vehicles with AEB and five-star crash test results vs. fleet-average 12 year old cars without all that fancy safety technology. Comparisons also tend to involve apples-meets-oranges different operational scenarios. The results are essentially meaningless, and simply provide grist for the hype machine.

At best, from available information, Level 2 systems are a safety-neutral convenience feature. At worst, AEB is compensating for a moderate overall safety loss from Level 2 features, contributing to an ever-increasing number of fatalities that might be associated with disturbing patterns such as crashing into emergency responders. However, nobody really knows the extent of any problems because of extreme resistance to sharing data by car companies that would reveal the true safety situation. (NHTSA has mandated crash reporting for some crashes, but there is no denominator of miles driven, so crash rates cannot be determined in a reliable way using publicly available data.) 

If the safety benefits were really there, one imagines that these companies full of PhDs who know how to write research papers would be falling all over themselves to publish rigorously stated safety analyses. But all we're seeing is marketing puffery, so we should assume the safety benefit is not yet there (or at least potential safety benefits are not yet supported by the actual data).

In the face of these questions about automation safety and pervasive industry safety opacity, the industry's plan is to ... add even more automation.

Level 2+ systems

"Level 2+" is a totally made up marketing term. The SAE J3016 standard says this term is invalid. But it gets used anyway so let's go with it and try to see what it might mean in practice

Once you have a vehicle that stays in its lane and avoids most other vehicles, companies feel competitive pressure to add more capability. Features such as lane changing, automatically merging into traffic, and automatically detecting traffic lights might help drivers with workload. And sure, that sounds nice if it can be done safely.

The question is what happens to driver attention as automation features get added and automation performance improves. There is plenty of reason to believe that as the driver perceives automation quality is improving, they will struggle to remain engaged with vehicle operation. They will succumb to driver complacency. The better the automation, the worse the problem, as illustrated by this conceptual graph:

A significant complication is that the driver has to monitor not only the road conditions, but also the car's behavior to determine if the car will do the right thing or not. This can be pretty tricky, especially if it is difficult to know if the car "sees" something happening on the road or not. What if there is a firetruck stopped on the highway in front of you? Will the car detect and stop, or run right into it? How long do you wait to find out? What if you wait just a little too long and suffer a crash? Monitoring automation behavior can be tricky, and is a much different task than simply driving. Assuming that a competent driver is also a competent monitor is a bit of a leap of faith. As you add more complex automation behavior, the driver's ability to track what is and is not supposed to be handled and compare that to what the vehicle is doing can easily be overwhelming. 

Monitoring cross-traffic can be especially difficult. How can the supervising driver know their car is about to accelerate out across traffic to make a left turn when there is an oncoming car?  Pressing the brake quickly after the car has already started lunging into traffic as a safety supervising driver is an entirely different safety proposition than waiting for a driver to command speed when the road is clear. The human/computer interface implications here are tricky.

Car companies will continue to pile on more automation features. At least some of them will continue to improve DMS capability. There are many human/machine interface issues here to resolve, and the outcome will in large part depend on non-linear effects related to often the human driver needs to intervene quickly and correctly to ensure safety. How that will work out in real world conditions remains to be seen. 

The moral hazard of blaming the driver

A significant issue with Level 2 and 2+ systems is that the driver is responsible for safety, but might not be put into a position to succeed. There are natural limits to human performance when supervising automation, and we know that it is difficult for humans to do well at this task. We should not ask human drivers to display superhuman capabilities, then blame them for being mere mortals. Driver monitoring might help, but we should respect the limits to how much it can help.

There is a temptation to blame the driver for any crash involving automation technology. However, doing so is counterproductive. A crash is a crash. Blame does not change the fact that a crash happened. Blame is a fairly ineffective deterrent to other drivers slacking off in general, and is wholly ineffective at converting normal humans into super-humans. If using a Level 2 or 2+ system results in a net increase in crashes, blaming the drivers won't bring the increased fatalities back to life -- even if we bring criminal charges.

If real-world crashes increase with the use of driver automation (compared to a comparably equipped vehicle in the same conditions without driver automation), they should be considered  unreasonably dangerous. Something would need to be done about such systems to change the system, human behavior, or both. Changing human behavior via "education" is usually what is attempted, and almost never works. The human/computer interface and the feature set are much more likely what need to change.

As a simple example of how this might play out, let's say a car company runs TV advertisements showing drivers singing and clapping hands while engaging a Level 2/2+ system. (Dialing back the #autonowashing, later ads show only the passengers doing this.) The company should have data showing that even if a worst case driving automation error for their system takes place mid-clap, the driver will be attentive enough to the situation (despite being caught up in the song) and have a quick enough reaction time to get their hands back onto the steering wheel and intervene for safety. Blaming the driver for clapping hands after airing a TV commercial amounts should not be a permissible tactic. 

Note that we are not saying hands-off is inherently unsafe, but rather that permitting (even encouraging) hands-off operation significantly increases the challenge of ensuring practical safety. The data needs to be there to justify safety for whatever operational concept is being deployed and encouraged.

ALKS: a baby Step Toward Level 3

The way the term "Level 3" is being used by almost everyone seldom matches what the SAE J3016 standard actually says. (See Myths #6, #7, #8. #14, #15, etc in this J3016 analysis.) But this is not the place to hash out that incredibly messy situation other than to note that SAE J3016 terminology is not how we should be describing these safety critical driving features at all.

For our purposes, let's assume when a car maker says "Level 3" they mean that the driver can take their eyes off the road, at least for a little bit, so long as they are ready to jump back into the role of driving if the car sounds an alarm for them to do so. 

The industry's baby step toward this vision of Level 3 is Automated Lane Keeping Systems (ALKS) as described by the European standard UNECE Reg. 157. (To its credit, this standard does not mention SAE Levels at all.) The short version is that drivers can take their eyes off the road and let the car do the driving in slow speed situations. In general, this is envisioned as a traffic jam assistant (slow speed, stop-and-go, traffic jam situations). We can expect this to be the first Level 3 step in an envisioned transition from Level 2 to 2+ to 3 strategy that is already said to be deployed at small scale in Europe and Japan.

ALKS might work out well.  Traffic jams on freeways are pretty straightforward compared to a lot of other operational scenarios. Superficially there are a lot of slow moving cars and not much else. Going just a bit deeper, if the jam is due to a crash there will be road debris, emergency responders, and potentially dazed victims walking through traffic. So saying "no pedestrians" or even "pedestrians will be well behaved" is unrealistic near a crash scene. But one can envision this can be managed, so it seems a reasonable first step so long as safety is considered carefully. 

Broader Level 3 safety requirements

Going beyond the strict workings of SAE J3016 (which, if followed to the letter and not exceeded, will almost certainly result in an unsafe vehicle), Level 3 driving safety only works if:

• The Automated Driving System (ADS) is 100% responsible for safety the entire time it is engaged. Period. Any weasel wording "but the driver should..." will lead to Moral Crumple Zone designs (blaming people for automation failing to operate as advertised). Put another way, if the driver is ever blamed for a crash when a Level 3 automation feature is engaged, it wasn't really Level 3.
• The ADS needs to determine with essentially 100% reliability when it is time for the driver to intervene. This is one aspect of the most difficult problem in autonomous vehicle design: having the automation know when it doesn't know what to do. The magnitude of this challenge should not be underestimated. Safe Level 3 is not just a little harder than Level 2. The difference is night and day in any but the most benign operational conditions. Machine learning is terrible at handling unknowns (things it hasn't trained on), but recognizing something is an unknown is required to make sure the driver intervenes when the ADS cannot handle the situation.
• The ADS ensures reasonable safety until the driver has time to respond, despite the fact that something has gone wrong (or is about to go wrong) to prompt the takeover request. This means the ADS needs to keep the car safe at least for a while. If the driver takes a long time to respond, the ADS needs to do something reasonable. In some cases perhaps an in-lane stop is good enough; in others not. (In practice this pushes the ADS arguably to be a very low-end Level 4 system, but we're back to J3016 standards gritty details so let's not even go there. The point is that the driver might take a long time to respond, and the ADS can't simply dump blame on the driver if a crash happens when the going gets tough.)

For ALKS, the main safety plan is to go slowly enough that the car can stop before it is likely to hit anything. That "anything" is predominantly other cars, and sometimes people at an emergency scene. High speed animal encounters, cross traffic, and other situations are ruled out by the narrowly limited operational design domain (ODD), which should be enforced by prohibiting activation in inappropriate situations. The ALKS standard presumes drivers will respond to takeover notifications in 10 seconds, but the vehicle has to do something reasonable even if that does not happen.

More generally, it is OK to take credit for most drivers being able to respond relatively quickly in most situations. It is likely that a more advanced Level 3 system will progressively degrade its operation over a period of many seconds, such as first slowing down, then coming to a stop in the safest place it can reach given the situation.  If a driver falls asleep from boredom, it might take a while for honking horns from other drivers wake them up. Or it might take longer than 10 seconds to regain situational awareness if overly engrossed in a movie they were told was OK to watch while in this operational mode. Human subject studies can be used to claim credit for most drivers intervening relatively quickly  (to the degree that is true) even if all drivers cannot respond that quickly. Credit could be taken for being able to pull over to the side of the road in most situations to reduce the risk of being hit by other vehicles, even if that is not possible in all situations. Safety for human driver takeover is not the result of a single ten second threshold, but rather a stack-up of various levels of degraded operation and probabilities of mishaps in a variety of real-world scenarios.

One might be able to argue net acceptable safety as long as the worst cases are infrequent. If the worst cases happen too often -- including inevitable misuse and abuse -- some sort of redesign will be needed. Again, blaming drivers or "educating" them won't fix a fundamentally unsafe operational concept. Safety must be built into the system, not dodged by blaming drivers for having an unacceptably high crash rate.

Advanced features and test platforms

Once ALKS is in place, the inevitable story is going to be to slowly expand the ODD. For example, if things seem safe, increase the speed at which ALKS can operate. Then let it do on/off ramps. And so on.

The same thinking will be in place for Level 2+ systems. If in-lane cruise control works, add lane changing. Add traffic merging. Let it operate in suburbs. Add unprotected left turns. Next thing you know, we'll be back to saying robotaxis will be here next year -- this time for sure!

While this seductive story sounds promising, it's not going to be easy. Automation complacency combined with limits of drivers to supervise potentially unpredictable autonomy functions will be a first issue to overcome. The second will be confusing a Level 2 driver with an autonomous vehicle testbed driver.

We've already discussed automation complacency. However, it is going to get dramatically worse with more advanced functionality. Humans learn to trust automation far too quickly. If a car correctly makes an unprotected left turn 100 times in a row, any safety driver might well be mentally unprepared to intervene in time when it pulls out in front of oncoming traffic on the 101st turn. Even something as simple as a automation being able to see moving cars but not detecting an overturned truck on the road has led to crashes, presumably due to automation complacency.

While car companies can argue that trained, professional safety drivers might keep testing acceptably safe (e.g., by conforming to the SAE J3018 safety standard -- which mostly is not being done), that does not apply to retail customers who are driving vehicles.

Tesla has led the industry in some good ways by promoting the adoption of electric vehicles. But it has set a terrible precedent by also adopting a policy of having untrained civilian vehicle owners operate as testers for immature automation technology, leading to reckless driving on public roads.

There is already significant market pressure for other companies to follow the Tesla playbook by pushing out immature automation features to their vehicles and telling drivers that they are responsible for ensuring safety. After all, if Tesla can get away with it, only suffering the occasional wrist slap for egregious rule violations, is it not the case that other companies have a duty to their stockholders to do the same so as to compete on a level playing field? As long as we (the public) allow companies to get away with blaming drivers for crashes instead of pinning responsibility on auto makers for deploying poorly executed automation features, reckless road testing (for example, running red lights) will continue to proliferate.

While it is difficult to draw a line in the sand, we can propose one. Any vehicle that can make turns at intersections with no driver control input is not a Level 2 system -- it is an autonomous vehicle prototype. There is an argument supporting this based on the evolution of J3016. (See page 242 of this paper.) But more importantly, supervising turns in traffic is clearly of a different nature than supervising cruise control on a highway. It takes much more attention and awareness of what the vehicle is about to do to prevent a tragic crash. Especially if you are busy clapping hands to a rock song.

Abusing Level 2/2+ to evade regulation

Another trail blazed by Tesla is letting regulators think they are deploying unregulated Level 2/2+ systems when they are really putting Level 3/4 testbeds on the roads in the hands of ordinary vehicle owners.

Companies should not call something Level 2+ when it is really a testbed for Level 3 features.  Deployed Level 2+ features must be production ready instead of experimental, and should never put human drivers in the role of being an unqualified "tester" of safety critical functionality. Among other things, the autonowashing label of "beta test" that implies drivers are responsible for crashes due to automation malfunctions should be banned. 

The details are subtle here: if a vehicle does something dangerous that is not reasonably expected by an ordinary driver, that should be considered a software malfunction. Any dangerous behavior that cannot be compensated for by an ordinary driver should not happen -- regardless of whether the driver has been told to expect it or not. Products sold to retail customers should not malfunction, nor should they exhibit unreasonably dangerous behaviors. Telling the driver they are responsible for safety should not change the situation.

Software should either work properly or be limited to trained testers operating under a Safety Management System. Rollout of Level 2+ and Level 3 features can happen in parallel, and that's fine. But a prototype Level 3 feature that can malfunction is not in any way the same as a mature Level 2+ feature -- even if they superficially behave the same way. It's not about the intended behavior, it's about having potentially very ill-behaved malfunctions. A Level 2+ system might not "see" a weird object in the road and perhaps the driver should be able to handle such a situation if they have been told that is an expected behavior. But any system that suddenly swerves the vehicle left into oncoming traffic is not fit for use by everyday drivers. A suddenly left-swerving vehicle is not Level 2/2+ -- it is a malfunctioning prototype vehicle that should only be operated by trained test drivers regardless of the claimed automation level.

Over time we'll see car companies aggressively try to get features to market. We can expect repeated iterations of arguments that the next incremental feature is no big deal for a driver to supervise. But if that feature comes with cautions that drivers must pay special attention beyond what a driver monitoring system supports, or incur extra responsibility, that should be a red flag that we're looking at asking them to be testers rather than drivers. Warnings that the vehicle "may do the wrong thing at the worst time" are unambiguously a problem. Such testing should be strictly forbidden for deployment to ordinary drivers. 

Telling someone that a product is likely to have defects (that's what "beta" means these days, right?) means that they are being sold a likely-defective product. As such, they should not be distributed as retail products to everyday customers, regardless of click-through disclaimers. After all, any unqualified "tester" is not just placing themselves at risk, but other road users as well.

For those who say that this will impede progress, see my discussion of the myths propagated by the AV industry in support of avoiding safety regulation. Following basic road testing safety principles should be a part of any development process. Anything else is irresponsible and, in the long run, will hurt the industry more than it helps by generating bad press and accompanying ill will.

Civilian drivers should never be held responsible for malfunctioning technology by being used as Moral Crumple Zones. That especially includes mischaracterizing Level 3 (and above) prototype features as Level 2+ systems.

The bottom line 

Perhaps NHTSA will wake up and put a regulatory framework in place to help the industry succeed at autonomous vehicle deployment safety. (They already have proposed a plan, but it is in suspended animation.) Even if the new NHTSA framework proceeds, regulations and proposals from both NHTSA and the states address levels 3-5, while leaving a situation ripe for Level 2+ abuse. Until that changes, expect to see companies pushing the envelope on Level 2+ as they compete for dominance in the vehicle automation market.

Video: How Safe Is Safe Enough for Autonomous Vehicles

Video lecture summarizing main topics covered in my book:

• YouTube video (55 minutes)
• YouTube playlist (in case you want to find a particular slide)
• Archive.org video  (55 minutes)
• Lecture Slides
• More information about the book

Abstract:

The most pressing question regarding autonomous vehicles is: will they be safe enough? The usual metric of "at least as safe as a human driver" is more complex than it might seem. Which human driver, under what conditions? And are fewer total fatalities OK even if it means more pedestrians die? Who gets to decide what safe enough really means when billions of dollars are on the line? And how will anyone really know the outcome will be as safe as it needs to be when the technology initially deploys without a safety driver?

In this talk I outline some key factors involved in measuring and predicting autonomous vehicle (AV) safety. This includes what people mean by "safe," setting an acceptable safety goal, measuring & predicting safety, deciding when to deploy, and ethical AV deployment. A framework for making a responsible deployment decision needs to include not just risk, but also deal with inevitable uncertainty, stakeholder inclusion, and an ethical governance model. The talk is a high level overview of my recently published book: How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety.

Update on 2022 PA AV bill

 PA AV legislation update: The PA Senate Transportation Committee passed the PA House bill on AVs to the PA Senate for a vote. This is an amended version of PA HB 2398 bill that has already passed the PA House.   https://www.legis.state.pa.us/cfdocs/billinfo/billinfo.cfm?syear=2021&sInd=0&body=H&type=B&bn=2398

Key aspects of this new version (PN3563) fixes many of the issues I've noted in previous bills, which is good news. But still some remaining issues. A summary:

- Permits operation of an AV without a driver.
- A responsible Certificate Holder must be a company (it being "a person" is struck out).
- Human safety driver, if any, must be an employee or contractor.
- Permits platooning, but seems to require a driver in each vehicle.
- Requires reports of crashes involving harm or damage to property to PennDOT
- Public posting of contact info for crash claims
- Registration requirement with PennDOT includes safety management plan
- $1M insurance requirement (not as high as it might be, but better than many other states)

Some not-so-great parts
- Municipal preemption clause (but at least now it allows local authorities to enforce existing laws)
- PennDOT appears to have very limited ability to reject registrations
- Any computer driver automatically gets a driver license with no testing and no independent assessment of driving skill required
- No requirement to follow industry safety standards (J3016 is mentioned, but is NOT a safety standard)
- An advisory committee that reports on economic benefits (good) -- but no apparent charter for safety concerns
- Looks really difficult to suspend or revoke a certificate in practice. It is unclear that a severe crash is enough to do that, at least immediately (it seems only after a criminal conviction of killing someone -- which might take years). I guess we'll have to see how soft law works in this area over time.
- The Certificate Holder (remember that is a company, not a person) is considered the driver, and is specifically called out to be cited by police for violations. So if there is a criminal driving offense committed by an automated driver (something a human driver would go to jail for) there is quite literally nobody (no natural person) held responsible. Will be interesting to see how PennDOT handles driver license points for moving violations, if at all.

Hearing video here starting at time 2:12:

https://transportation.pasenategop.com/trans-101822/?wvideo=lggitdbqy5

More about various state bills including this one here:

https://safeautonomy.blogspot.com/2022/02/kansas-av-regulation-bill-hearings.html

Talk: Autonomous Vehicles Standards & Open Challenges

 Here is my talk from the October 2022 ISO 26262/SOTIF conference.

Assuming you follow the relevant standards (ISO 26262, ISO 21448, ANSI/UL 4600) in practice teams are finding the following topics difficult:

• Fail operational architecture
• Building an accurate, predictive world model
• Safety beyond the driving task (system safety, traffic system interactions)
• Determining how safe is safe enough in an equitable way

Link to slides

(Slides only at this time)

AV Safety with a Telepresent Driver or Remote Safety Operator

Some teams propose to test or even operate autonomous vehicles (AVs) with a telepresent driver or remote safety operator.  Making this safe is no easy thing.

Typically the remote human driver/supervisor located at a remote operating base, although sometimes they will operate by closely following the AV test platform in a chase vehicle for cargo-only AV configurations.

Beyond the considerations for an in-vehicle safety driver, telepresent safety operators have to additionally contend with at least:

·        Restricted sensory information such as potentially limited visual coverage, lack of audio information, lack of road feel, and lack of other vehicle physical cues depending on the particular vehicle involved. This could cause problems with reacting to emergency vehicle sirens and reacting to physical vehicle damage that might be detected by a physically present driver such as a tire blow-out, unusual vibration, or strange vehicle noise. Lack of road feel might also degrade the driver’s ability to remotely drive the vehicle to perform a fallback operation in an extreme situation.

·        Delayed reaction time due to the round-trip transmission lag. In some situations, tenths or even hundredths of seconds of additional lag time in transmissions might make the difference between a crash and a recovery from a risky situation.

·        The possibility of wireless connectivity loss. Radio frequency interference or loss of a cell tower might interrupt an otherwise reliable connection to the vehicle. Using two different cell phone providers can easily have redundancy limitations due to shared infrastructure such as cell phone towers,[1] cell tower machine rooms (for some providers), and disruption of shared backhaul fiber bundles.[2] A single infrastructure failure or localized interference can disrupt multiple different connectivity providers to one or multiple AVs.

Role of remote safety operator

Achieving acceptable safety with remote operators depends heavily on the duties of the remote operator. Having human operators provide high-level guidance with soft deadlines is one thing: “Vehicle: I think that flag holder at the construction site is telling me to go, but my confidence is too low; did I get that right? Operator: Yes, that is a correct interpretation.” However, depending on a person to take full control of remotely driving a vehicle in real time with a remote steering wheel at speed is quite another, and makes ensuring safety quite difficult.

A further challenge is the inexorable economic pressure to have remote operators monitoring more than one vehicle. Beyond being bad at boring automation supervision tasks, humans are also inefficient at multitasking. Expecting a human supervisor to notice when an AV is getting itself into a tricky situation is made harder by monitoring multiple vehicles. Additionally, there will inevitably be a situation in which two vehicles under control of a single supervisor will need concurrent attention when the operator can only handle one AV in a crisis at a time.

There are additional legal issues to consider for remote operators. For example, how does an on-scene police officer give a field sobriety test to a remote operator after a crash if that operator is hundreds of miles away – possibly in a different country? These issues must be addressed to ensure that remote safety driver arrangements can be managed effectively.

Any claim of testing safety with a telepresent operator needs to address the issues of restricted sensory information, reaction time delays, and the inevitability of an eventual connectivity loss at the worst possible time. There are also hard questions to be asked about the accountability issues and law enforcement implications of such an approach.

Active vs. passive remote monitoring

A special remote monitoring concern is a safety argument that amounts to the vehicle will notify a human operator when it needs help, so there is no need for any human remote operator to continuously monitor driving safety. Potentially the most difficult part of AV safety is ensuring that the AV actually knows when it is in trouble and needs help. Any argument that the AV will call for help is unpersuasive unless it squarely addresses the issue of how it will know it is in a situation it has not been trained to handle.

The source of this concern is that machine learning-based systems are notorious for false confidence. In other words, saying an ML-based system will ask for help when it needs it assumes that the most difficult part to get right – knowing the system is encountering an unknown unsafe condition –  is working perfectly during the testing being performed to see if, in fact, that most difficult part is working. That type of circular dependency is a problem for ensuring safety.

Even if such a system were completely reliable at asking for help when needed, the ability of a remote operator to acquire situational awareness and react to a crisis situation quickly is questionable. It is better for the AV to have a validated capable of performing Fallback operations entirely on its own rather than relying on a remote operator to jump in to save the day. Before autonomous Fallback capabilities are trustworthy, a human safety supervisor should continuously monitor and ensure safety.

Any remote operator road testing that claims the AV will inform the remote operator when attention is needed should be treated as an uncrewed road testing operation as discussed in book section 9.5.7. Any such AV should be fully capable of handling a Fallback operation completely on its own, and only ask a remote operator for help with recovery after the situation has been stabilized.

---

[1] For example, a cell tower fire video shows the collapse of a tower with three antenna rows, suggesting it was hosting three different providers. 
See: https://www.youtube.com/watch?v=0cT5cXuyiYY

[2] While it is difficult to get public admissions of the mistake of routing both a primary and backup critical telecom service in the same fiber bundle, it does happen.
See: https://www.postindependent.com/news/local/the-goof-behind-losing-911-service-in-mays-big-outage/

The Software Defined Vehicle Is Still More Wish Than Reality

Here is a Software Defined Vehicle video that covers a lot of ground. Car companies are all talking a big game about adding software to their vehicles, including big data, software updates, connectivity, and more. The possibilities are exciting, but you only have to read the news to know that the road to get there is proving bumpier than they'd like. (See this story too.)

Getting the mix of Silicon Valley software + automotive system integration + vehicle automation technology right is still a big challenge. This video talks about the possibilities. But to get there, OEMs still have a lot of work to do achieving a viable culture that addresses inherent tensions:

• Cutting edge cloud software vs. life critical embedded systems
• Role of automation vs. realistic expectations of human drivers
• A shift from "recall" mentality to continuous improvement processes
• Fast updates vs. assured safety integrity
• Role of suppliers vs. OEM, especially for autonomous vehicle functions
• Monetizing data vs. consumer rights
• OEMs stepping up to the system integration challenges
• Getting a regulatory approach that balances risks and benefits across all stakeholders

(Sadly, the video includes an incorrect statement that "95% to 96% of the accidents happen because of distracted driving" in the context of fatalities. Drivers are not perfect, but distracted driving only contributes to about 9% of fatalities per US DOT, about one-tenth of what was stated.)

YouTube: https://youtu.be/T4wEe2bNSSk

Enhanced personal safety for autonomous vehicles

AV safety discussions often get quite technical. But there are aspects of safety that have a lot more to do with personal safety concerns. It is important that AV technology deployments enhance rather than degrade personal safety.

Do autonomous vehicles improve personal safety compared to alternatives?

An important feature of a personally owned human-driven vehicle is having more control over personal safety. A locked private vehicle provides a measure of physical protection against potential threats to personal safety. In a single occupancy conventional vehicle the driver can make personal safety choices beyond the obvious one of not sharing a vehicle with a stranger as would be the case in a taxi or ride-share vehicle.[1] The availability of a single-occupancy AV might extend this safety benefit to those who cannot drive or do not have resources to own a private vehicle.

Example safety choices beyond just riding solo include debarking in an escort-provided portion of a parking lot, selecting routes that seem to present lower personal risk, and deciding not to exit the vehicle at a preselected destination location that turns out to look dangerous. To the degree that a single-occupant AV provides similar personal risk management features, riding solo in an AV might be safer than in a shared vehicle, including one with a human driver.

Personal safety is especially important to more vulnerable demographic segments, particularly when traveling alone, such as women, the elderly, and children. Also potentially at risk are identifiable minority groups in areas prone to abusive behavior based on race, gender, ethnicity, religion, or other factors. Beyond that, any AV user might have personal safety concerns, especially in areas with high crime rates.

Personal safety on shared AV mass transit vehicles will be an obvious concern as it is with crewed transit. On crewed mass transit the crew members can provide an additional measure of social supervision and deterrence. A potential move to smaller AV shared transit vehicles increases the opportunity for a passenger being isolated with a potential bad actor in a travel module, and complicates remote surveillance by multiplying the number of small passenger compartments being managed rather than fewer large compartments. Supervising dozens or hundreds of people on a single fully automated passenger train seems a more tractable problem (e.g., done with an on-train conductor) than remotely supervising dozens or hundreds of robotaxis shared by strangers.

Beyond the ride itself, there are also safety issues related to waiting for transport arrival and offloading. In particular, it will be important for vulnerable passengers to be able to change their destination at the end of the trip if local conditions at the destination seem too dangerous. Consider a city that requires using designated drop-off points of AV robotaxis. What should a passenger do if they do not like the looks of a group of people, potentially armed, waiting at the stop for them to get out?

A simple argument is to say that every automated low-speed shuttle will have an attendant. While attendants might be desirable and might prove necessary for a variety of reasons, requiring full-time staff on an automated vehicle that is smaller than a mass transit vehicle is largely at odds with the argument that AVs provide economic benefits due to not having to pay a person to be on board.

The question here is: will riding on an automated vehicle be as safe as riding in a vehicle with a human driver from a personal safety point of view?

---

[1] A popular meme goes something like this: Years ago we were told not to get into cars with strangers and not to talk to strangers on the Internet. Now we literally contact strangers via the Internet so we can get into their cars.            
While ride-share companies recognize that personal safety is a key issue, and put effort into improving it, personal safety needs more work. See Marshall 2019: https://www.wired.com/story/criminologist-uber-crime-report-highly-alarming/
Also Saddiqui 2021:         
https://www.washingtonpost.com/technology/2021/10/22/lyft-safety-report/

Gatik Announcement -- Is it real safety? Or just AV safety theater?

Gatik just announced it has completed an extensive third-party safety review of its system as part of deploying fully driverless commercial operations in Canada. But the announcement raises many questions as to how much it really assures safety.

The autonomous vehicle safety arena is full of misinformation, disinformation, safety theater -- and players earnestly trying to do the right thing. Companies routinely employ ambiguous language, half-truths, and outright propaganda to deploy safety theater. But some companies use unambiguous statements of conformity to safety standards to show they are really doing safety. Which bucket does Gatik fall into?  Let's take a look at the signs from their press release.

Gatik claims that their third-party review covers safety and security. This was done with "a team of third-party experts." No mention of who these experts might have been, nor their qualifications. The gold standard is an accredited third party assessor such as TUV SUD (there are quite a few others as well).

For security they mention reasonable standards including SAE J3061, ISO/SAE 21434, and UNECE R155. It would be better to see them state "conformance" with these standards instead of just saying they were "covered" by the review. (Maybe they failed to conform as a result -- who knows?)  But at least this statement shows that the experts knew enough to look at these standards. So maybe OK, but hard to say.

For safety the only standard mentioned is SAE J3016 -- which is not a safety standard. In fact, only meeting the minimum requirements for the SAE Levels is not safe in practice (e.g., driver monitoring is not required, nor is any notification to the human driver that takeover is required after some types of failures). The safety analysis, such as it is, is clearly patterned after J3016, mentioning ODD and OEDR. 

There is a statement that "where they apply, the vehicle and ADS comply with safety relevant standards and best practices, such as those developed by SAE International and the International Organization for Standardization (ISO)."  No mention of ANSI/UL 4600, which was included by NHTSA as a highly relevant standard. Also, what do they mean by "where they apply" exactly? Other companies have gone on the record saying none of the AV-specific safety standards apply to their AV. So maybe Gatik means they aren't following safety standards at all.  Not even SAE J3018 for testing safety.

What I get out of reading the announcement is they hired some unidentified experts of unknown reputation, who likely had better credentials in security than safety. (Any bona fide safety experts would never pronounce that a system was "safe" based solely on testing results as indicated in the press release.) They took a look and say "sure, looks like it works." That's about it. (If there is more, we'd expect them to brag about it with some specificity, right?)  

If there is one thing I've learned in this industry is that companies will claim the strongest thing they think they can. A weak claim means a weak result. This is a very weak safety claim.

While I appreciate that Gatik publicly messages “Safety is at the heart of everything we do," this Gatik press release fairly screams safety theater. If they want us to believe their message is compelling, they need to do better. Some examples of ways they could provide a better statement of safety:

• What exactly do you mean by "acceptable" safety?
• Name the safety standards they "considered."   Were they ISO 26262, ISO 21448, ANSI/UL 4600, and SAE J3018 (all safety standards). These are the types of standards US DOT has already proposed for regulatory purposes, so they ought to be top of mind for any AV safety assessment.
• Name the safety standards they actually conform to beyond "considering" and potentially not implementing.
• Is there an Safety Management System (SMS)?  Didn't see it mentioned. This is safety 101, so you'd think they'd at least mention that.
• Say who the external experts were so we can judge their reputation. Were they an accredited assessment organization? Were any of them actually qualified to opine on safety rather than security?
• Explain how it is that a "rigorous suite of system as well as component level tests" can show safety. Because everyone would really like to know how you can do that for an AV. The safety standards are much more about engineering processes and safety engineering, with validation just being the tail of the safety dog. Certainly nothing I've seen indicates that validation-only safety assessment is possible for an AV.

Their announcement video talks about delivering against a value proposition. The only reason it gives for believing they are safe is ... saying they deliver safety and "manage risk."  That's it.  Gatik has not issued a VSSA, so no info to be had there.

Gatik -- your turn.  If you have a response I'll be happy to post it here for all to see:

... no response yet ...

The NIEON Driver Benchmark and the Two-Sided Coin for AV safety

Waymo just published a study showing that they can outperform an unimpaired (NIEON) driver on a set of real-world crashes. That's promising, but it's only half the story. To be a safe driver, an Autonomous Vehicle (AV) must both not only be good -- but also not be bad. Those are two different things. Let me explain...

Learning to drive includes learning how to drive well.

It also includes learning to avoid making avoidable mistakes. They're not the same thing.

A Sept. 29, 2022 blog posting by Waymo explains their next step in showing that their automated driver can perform better than people at avoiding crashes. The approach is to recreate crashes that happened in the real world and show that their automated driver would have avoided them. 

For this newest work they compare their driver to not just any human, but a Non-Impaired, with Eyes always ON the conflict (NIEON) driver. They correctly point out that no human is likely to be this good (you gotta blink, right?), but that's OK. The point is to have an ideal upper bound on human performance, which is a good idea.

Setting a goal that to be acceptably safe an AV should be at least as good as a NIEON driver makes sense. It cuts out all the debate about which human driver is being used as a reference (e.g., a tired 16 year old vs. a well rested professional limo driver 50 year old will have very different driving risk exposure).

Unsurprisingly Waymo does better than a NIEON on scenarios they know they will be running, because computers can react more quickly than people when they correctly sense, detect, and model what is going on in the real world. Doing so is not trivial, to be sure. As Waymo describes, this involves not just good twitch reflexes, but also realizing when things are getting risky and slowing down to give other road users more space, etc.

That is the "be a good driver" part, and kudos to Waymo and other industry players who are making progress on this. This is the promise of AV safety in action.

But to be safe in the real world, that is not enough. You also have to not be a bad driver. Doing better at things humans mess up is half the picture. The other half is not making "stupid" mistakes that humans would likely avoid. 

AVs will surely have crashes that would be unlikely for a human driver to experience. Or will sometimes fail to be cautious when they should, and so on. While this did not lead to crashes, the infamous Waymo ride video showing construction zone cone confusion shows there is more work to be done in getting AVs to handle unusual situations. To its credit, in that scenario the Waymo vehicle did not experience a crash. But other uncrewed AVs are in fact having crashes due to misjudging traffic situations. And an automated test truck crashed into a safety barrier despite having a human safety driver due to an ill-considered initialization strategy that at least some would consider a rookie mistake (e.g., repeats a type of mistake made in Grand Challenge events -- should have known better).

It is good to see Waymo showing their automated driver can do well when it correctly interprets the situation it is in. That shows it is a potentially capable driver. We still need to see it is additionally not making mistakes in novel situations that aren't part of the human driver crash dataset. If Waymo can show NIEON level safety in both the knowns and the unknowns, that will be an impressive achievement.

The Autonomous Vehicle Deployment Governance Problem

The #1 ethical issue in autonomous vehicles is not the infamous Trolley Problem. It is the question of who gets to decide when it is OK to deploy a vehicle without a safety driver on public roads.

Consider a thought experiment which, if you follow AV industry news, you might recognize as not entirely hypothetical. You, the reader, are in charge of a company that needs to do a public road demonstration with no driver in an AV. You know that safety is not where you would like it to be. In fact, you have no safety case at all. You might not even have any real safety engineers on staff. But you have a smart, super-capable team. You have done a lot of test driving and it is going pretty well.

You intuitively figure it is more likely than not that you can pull off a one-time demo without a crash, and even less likely that a crash will kill someone. You figure you have something like 5 chances out of 6 of pulling off the demo with nobody getting hurt, and it is, in your mind, near certainty due to low-speed urban driving that any crash would avoid a fatality. For good measure maybe you plan to station employees near the demo site to shoo away any pedestrians and light mobility users who would be at increased risk of harm, and do the demo very late at night when roads are usually empty of other road users. Regulators are not in a position to influence your decision.

Your investors have told you they will pull the plug on your entire company if you do not demo by December 31st. Right now, it is the first week of December, and it is time to decide what to do. If the investors pull the plug at the end of the month, you lose perhaps $1B in personal equity you hope to net in next year’s public offering. And all your employees will lose their equity as well as their jobs. This will also end a journey you have spent your life on to build and deploy a truly self-driving car.

Further negotiations with the investors are not possible. It is time to decide. That leaves you three main options:

• Case 1: The AV company does not do the demo because it cannot assure a PRB level of safety. The company runs out of money and folds. This option kills the company.

• Case 2: The AV company does the demo and harms a road user. This might or might not result in termination of funding depending on the optics of the crash (perhaps a pedestrian victim can be blamed for jaywalking, being impaired, or having low societal status; maybe all three). You think minor harm is more likely than a fatality, and you will have lots of money available to pay off a potential victim to keep quiet. You will not pre-announce the demo, so you feel able to control the narrative if something goes wrong. The company and the mission go on unless there is a truly unlucky break during that one demo/test session that cannot be cleaned up. Even Uber ATG kept going for a while after a really bad crash, and you know in your heart that your team is better.

• Case 3: The AV company does the demo and gets lucky, not harming any other road users. The company meets its milestone and gets more funding. This the most likely case, and it would be a perfect victory.

Given this setup doing the demo is clearly the best financial bet for the company. Probably you will get lucky with a positive outcome. No harm will be done, and the demo can be said to be safe under the culturally dominant no harm/no foul principle. But if the demo is skipped due to safety, the company is sure to die, and the decision maker is out a billion dollars.

Even if you get unlucky, the cost of a few million dollar settlement pales in comparison with the billions of dollars on the table. Really – you might think – a payout is just the cost of doing business. And even if the crash optics get out of control and the startup company folds, the investors have hedged their bets and the team can simply move to another company and try again. Pretty much everyone will do fine. Except for the victim, if there is one.

This is how demo milestones incentivize deploying systems when the calendar and funding flow says it is time for a demo rather than when the demo is known to be acceptably safe. After all, it is someone else who is injured or dies, not the decision maker. And a billion dollars is a ton of money. And probably it will be fine. After all, you think, only other companies kill pedestrians.

FMVSS Exemption Considerations for Fully Autonomous Vehicles

Summary: The human role in FMVSS should not be removed for exemption request because there is no driver. Rather, what should be stated is how the driver is being replaced in that safety role as a matter of system design.

These comments were filed regarding the General Motors Petition for Temporary Exemption from FMVSS -- Federal Motor Vehicle Safety Standards (https://www.regulations.gov/document/NHTSA-2022-0067-0002)  However, they are likely to apply to any autonomous vehicle that does not have conventional driver controls or operates in a mode which does not have an officially designated driver.

It is good to see companies working to advance the potential benefits of autonomous vehicle technology. However, it is also important for NHTSA to ensure public safety when such technology is deployed on public roads.

There will almost certainly need to be an auxiliary controller to command motion of vehicles without normally accessible driver controls. To the extent that these are in-vehicle wired controls the question arises as to their suitability for safe vehicle control (e.g., using a "Playstation" type video game controller) and whether they can be improperly activated during autonomous operation. To the extent that these are wirelessly connected controls there are serious security issues that must be considered. In the absence of normal driver controls any petitioner should be required to justify the safety and security of its maintenance and supplemental human-operated vehicle control strategy beyond more general claims of working on cybersecurity.

FMVSS 101: Petitioner should demonstrate that all telltales that are intended to prompt human driver reaction similarly prompt a relevant ADS reaction to an indicated exceptional condition. If no such demonstration is provided, NHTSA would simply be taking Petitioner’s word that the telltale functions are implemented in a way that provides equivalent safety. (This also applies to other telltales such as FMVSS No. 126, 138, etc.)

FMVSS 102: While the ADS is said to control the transmission, there will be times in which humans need to know the state of the transmission for safety, especially whether the vehicle is in park or not. This includes emergency responders and maintenance personnel to ensure that the vehicle will not move unexpectedly when it is operating in a degraded or post-crash state. NHTSA should evaluate the suitability of a passenger video display for reliably displaying vehicle park state to non-passenger stakeholders.

FMVSS 104: From the NTHSA summary:  "For GM's petition for exemption from portions of FMVSS No. 104, GM argues that the purpose and intent of the safety standard is obviated by the Origin's sensor system design."  GM's argument is ridiculous from a technical point of view.  (Their technical disclosure is more substantive, but this type of lawyer argument in their summary serves to undermine their filing's credibility.)  In the context of an ADS, the intent of the safety standard is not to permit a human driver's eyes to see out the windshield. Rather, it is to ensure that all visual sensors can see through the windshield (if interior mounted), or through their lenses and weather covers as appropriate. It is completely foreseeable that camera and lidar operation will be impaired by adverse operational conditions. Moreover, the problem is likely to be worse than for human drivers because even a relatively small bug splat on a camera lens will form a visual obstruction covering a much larger fraction of the field of view compared to the same bug impacting the windshield of a human-driven vehicle, because human drivers can and do simply move their heads a bit to see past windshield obstructions.  GM states that it keeps its sensors clean, but provides no objective way for NHTSA to validate the claim.

FMVSS 111: From the NHTSA summary: "GM points out that the purpose and intent of FMVSS No. 111 is based on human perception and visibility so there is no operational safety need for these requirements when applied to a vehicle driven by an ADS." This argument amounts to the common fallacy of: "humans have limitations and therefore computers will be perfect." (Anyone paying attention knows that computers fail, often spectacularly – just in different ways than humans.) There are safety requirements behind FMVSS 111 that are not about "human perception and visibility" but rather, for example, not running over children that are not visible to the driver sensors.  Again, GM's lawyer-phrased argument undermines the credibility of their filing. In addressing this request for exemption the petitioner should explain why it is the case that small children will be protected from being run over at least as well as by a human driver using a rear view camera as one example. Again, the technical substance is better, but leaves a big open question. GM provides pictures indicating that cameras can potentially see areas required. However, there is no data supporting that those cameras will correctly sense and classify children close to the vehicle with high accuracy, especially at night when the vehicle automation might rely more heavily on roof-top mounted lidar sensors. The safety requirement is not camera visibility, but rather that the driver+camera will avoid hitting children. Given apparent issues with camera-based AEB systems having trouble sensing and avoiding hitting children this is an issue that needs to be taken seriously.

FMVSS 201: From the NHTSA summary: "GM argues that sun visors are not necessary because the Origin is not operated by a human driver, and the ADS does not use the windshield for visibility." Again, limitations of human drivers are not the real point for ADS safety. It is well known that cameras struggle when looking into the sun. GM should explain how they handle that problem (sun visors are a way human drivers can handle it) and now NHTSA can validate that sun glare is handled safely.  GM states it is not a problem for this system, hinting that perhaps they rely on lidar+radar instead of a camera in such situations, but that raises additional concerns. GM should provide a way for NHTSA to validate that sun glare does not present undue risk.

Validation: For all requested exemptions it is not enough to simply say more or less "trust us, we have figured this out" for an ADS equipped vehicle than it is for a conventional vehicle. If "trust us" were enough, there would be no need for the carefully design tests in FMVSS for any vehicle. Rather, petitioners should explain (a) why the broader safety objective of the FMVSS element being waived is still being met (e.g., not hitting children vs. field of view), and (b) a way that NHTSA can validate for itself that the safety objective is in fact being met. See:  Koopman, “How to keep self-driving cars safe when no one is watching for dashboard warning lights,” The Hill, June 30, 2018. https://thehill.com/opinion/technology/394945-how-to-keep-self-driving-cars-safe-when-no-one-is-watching-for-dashboard/

 

Regarding Public Interest Considerations:

Safety Benefit: It is essential to keep in mind when evaluating ADS petitions that potential safety benefits are purely aspirational. There is not yet any proof that ADS-equipped vehicles will be even as safe as a comparable active-safety-equipped human-driven vehicle using any currently available or near term deployable technology. While supporting the development of technology that has the potential to reduce harm is laudable, this should not be done at the cost of increased near-term harm. Putting road users at increased risk today because someday, maybe, perhaps, autonomous vehicles might be safer is unacceptable. NHTSA should be in the safety business, not the hopium business. That means exemptions should require a concrete justification of deployed safety that is not weakened by promises of ADS perhaps increasing safety in the future.

Environmental Impact: Requiring environmental impact data reporting is an excellent idea. It is important to be mindful of indirect environmental and safety consequences of ADS-equipped vehicle adoption. One example is that (presumably) decreased cost per mile is prone to increasing demand, resulting in a potential increase in overall emissions if small ADS-equipped vehicles are used for more trips.  Both human harm and environmental damage could be increased if ADS-equipped vehicles draw passenger loads away from mass transit even if the total number of user-miles across all modes remains the same. Moreover, use of uncrewed ADS-equipped vehicles can increase both total pedestrian harm and emissions as well as road congestion if that leads to a large increase in on-demand delivery services.

Equity: As NHTSA notes in their summary, petitioners for exemptions extoll the potential benefits to expand transportation options to under-served areas and customers with disabilities. It is important to point out that these populations can be served with traditional human-operated vehicles as well, so this is not a fundamentally new capability, but rather an argument that cheaper costs will increase vehicle availability (perhaps at the cost of increased road congestion). Note that vehicle sharing does not require autonomy, nor does use of electric vehicles. To the extent that the exemptions are for "robo-taxi" and shuttle type applications, petitioners should not only say that their technology might possibly help disadvantaged populations, but also how they plan to serve such populations directly as a result of the exemption. Promising to serve such populations in an exemption petition but then fielding a single token wheelchair-accessible vehicle or only deploying in a rich urban center would result in such statements of providing equitable transportation being only aspirational rather than directly related to the exemption decision. Credit should not be given for empty promises in evaluating an application.

Safety: While considering economic impacts might be relevant as NHTSA notes in their summary, it is essential to remind all stakeholders that the "S" in NHTSA stands for "Safety." It is imperative that economic motivations not over-ride the NHTSA mission to ensure safety. Safety must come first, and not be overwhelmed by economic pressure.

Standards: Petitioners who want exemptions from FMVSS provisions should have an alternate method of ensuring that a rigorous engineering approach has been used to ensure safety. Such an approach was proposed by NHTSA itself in an ANRPM (Docket No. NHTSA-2020-0106 / RIN 2127-AM15 Framework for Automated Driving System Safety https://www.regulations.gov/docket/NHTSA-2020-0106 ). NHTSA should respond to comments from that ANPRM and work further on setting up such a framework that could provide and an alternate basis for ADS-equipped vehicle regulation.

SMS: NHTSA should require all petitioners to implement an effective Safety Management System (SMS) as a condition of receiving an exemption, with SMS data included in periodic reports for the life of any vehicle.

Crash Reporting: NHTSA should continue to require crash reporting as proposed. The definition of a crash should be scoped to include: (a) any contact between the vehicle and another road user regardless of severity; (b) any contact between the vehicle and other obstacles or infrastructure regardless of severity; (c) any "near hit" that required evasive action from another road users (e.g., pedestrian jumping back on sidewalk to avoid being hit while in a crosswalk); (d) any substantive moving violation (e.g., running a red light).  Any responsible tester/operator of an ADS will have data already compiled on all these events as part of their SMS to ensure that they are operating safely on public roads, so the reporting burden should be minimal.  These reports should be required for the life of any exempt vehicle.

Commenter Qualifications: Prof. Philip Koopman is an internationally recognized expert on Autonomous Vehicle (AV) safety whose work in that area spans over 25 years. He is also actively involved with AV policy and standards as well as more general embedded system design and software quality. His pioneering research work includes software robustness testing and run time monitoring of autonomous systems to identify how they break and how to fix them. He has extensive experience in software safety and software quality across numerous transportation, industrial, and defense application domains including conventional automotive software and hardware systems. He was the principal technical contributor to the UL 4600 standard for autonomous system safety issued in 2020. He is a faculty member of the Carnegie Mellon University ECE department where he teaches software skills for mission-critical systems. In 2018 he was awarded the highly selective IEEE-SSIT Carl Barus Award for outstanding service in the public interest for his work in promoting automotive computer-based system safety. In 2022 he was named to the National Safety Council's Mobility Safety Advisory Group. He is the author of the book How Safe is Safe Enough: measuring and predicting autonomous vehicle safety (2022). https://users.ece.cmu.edu/~koopman/

Continuous Learning Approach to Safety Engineering

Continuous Learning Approach to Safety Engineering

Rolf Johansson & Philip Koopman / CARS @EDCC 2022

Abstract:

A phase change moment is upon us as the automotive industry moves from conventional to highly automated vehicle operation, with questions about how to assure safety. Those struggles underscore larger issues with current functional safety standards in terms of a need to strengthen the traceability between required practices and safety outcomes. There are significant open questions regarding both the efficiency and effectiveness of standards-based safety approaches, including whether some engineering practices might be dropped, or whether others must be added to achieve acceptable safety outcomes. We believe that rather than an incremental approach, it is time to rethink how safety standards work. We propose that real-world field feedback for an initially safe deployment should support a DevOps-style continuous learning approach to lifecycle safety. Safety engineering should trace from a safety case to engineering practices to safety outcomes. Such an approach should be incorporated into future safety standards s (including ISO 26262) to improve safety engineering efficiency and effectiveness.

Full paper here: link