Tuesday, February 14, 2023

Insurance Does Not (and will not) Make AVs Acceptably Safe

I frequently hear arguments that insurance will make autonomous vehicles (AVs) safe. For example, : "the insurance company issued a policy, so the AV must be safe," and "economic pressure from insurance premiums will ensure safety." While it is true that insurance premium pressure (and companies) will mitigate egregiously dangerous AVs, they have nowhere near enough power to enforce safety acceptable to many stakeholders in an industry of risk-takers chasing a trillion-dollar market.

Dall-E picture of baby carriage, car and a dollar sign

Insurance policies do not make you safe

Getting an insurance policy does not mean you are objectively “safe.” You can insure plenty of things that might be considered risky by everyday standards: skydiving injury insurance, commercial rocket launch payload insurance, marine piracy insurance,[1] and life insurance for front-line military personnel are all routinely issued.

An insurance company issuing a policy does not mean any particular activity in general or AV in particular is objectively safe. Rather, it means that the insurance company thinks it understands the risks well enough to set a policy rate that is, on average, economically attractive (profitable to the insurance company) across the members of the risk pool.

Taking a lot of risk? Expect a higher premium. But you will still get an insurance policy for high-risk activities so long as the insurance company feels comfortable it can estimate likely future losses and charge accordingly.

To be sure, the insurance industry does support improving safety. Historically, the insurance industry has spawned activities to create safety standards and help their customers manage risks.[2] Organizations such as the Insurance Institute for Highway Safety (IIHS)[3] are both active and vocal in support of vehicle automation safety. Insurance companies typically have a loss prevention activity to support their clients as well. So these remarks should not be interpreted as indicating industry disregard for safety.

Nonetheless, the economic reality of the situation is that you can get insurance for activities that are objectively dangerous so long as you are willing to pay the required premium. With a multi-billion dollar war chest for AV development and aggressive timelines to deploy, reducing insurance costs is nowhere near the top budget item of concern until well after deployment. Higher insurance premiums are unlikely to drive safety improvements very hard until the industry is operating at significant scale. And even then, other issues discussed regarding risk mitigation incentives will still apply.

Low premiums do not necessarily mean low harm

A crucial aspect of insurance for vehicle crashes is that total cost of insurance bundles together harm to people as well as property damage. For personal policies the contributing costs are broken out into several categories. But if you are tracking business profitability what matters is the total insurance cost, which includes both harm to people and property damage. If the property damage risk far outweighs the risk of harm, there is reduced economic pressure to mitigate harm to people.

An everyday example of divergence between insurance premiums and fatality rates can be seen in motorcycles.

Overall, motorcyclists account for 14% of all traffic fatalities, with those fatalities occurring nearly 27 times more frequently than for passenger cars on a per-mile basis.[4] Even though motorcycles are 27 times more dangerous in terms of risk of death per mile, motorcycle insurance costs about half that of car insurance. This difference is attributed to motorcycle crashes causing much lower equipment damage bills (both to the motorcycle and any other vehicle hit) that more than offset the cost of increased fatalities.[5]

Based on motorcycle insurance alone, it is clear that insurance cost can be a poor prediction of occupant harm because of the influence of property damage on insurance rates. Cheaper insurance does not necessarily mean a vehicle is safer. By the same token, more expensive insurance on an expensive-to-repair vehicle chock-full of crash safety technology does not mean such a high-end vehicle is less safe.

In short, insurance rates are not necessarily predictive of safety.

Insurance premiums will not force acceptable safety

You can perform risk management exclusively by purchasing insurance and doing no risk mitigation whatsoever, so long as you can afford the premiums. People and businesses do precisely that on a regular basis.[6] However, there is economic pressure for sophisticated companies to perform risk mitigation to lower insurance premiums – to a point.

In principle, risk mitigation will lower your insurance premiums, but that might or might not be worth your while. If hypothetically you are spending $1 per mile to run a vehicle and insurance costs $0.05 per mile, the theoretical limit to the benefit of risk mitigation is only 5% of your costs. You might be better off from a purely economic point of view spending management attention on optimizing the other $0.95 per mile of cost. Note that safety does not enter into such an insurance-driven risk management calculation – it is purely about optimizing profits.

Yes, if insurance is 95% of your cost, you have strong incentive to reduce risk. But as insurance cost becomes small compared to other factors, there is less and less pressure to do risk mitigation to further reduce costs. This is especially true if you are in a fast-moving business where things like time to market and ability to scale the fleet quickly are an existential threat to your business vs. a few cents per mile of insurance cost. (If you only have a handful of cars on the road because they don't really work yet, a few cents per mile of insurance cost are simply not on your list of worries.)

In other words, the ability to buy insurance does not mean that an AV is safe, but rather that the insurance company has decided they are willing to get paid a certain amount to cover any potential losses, and the AV maker has decided they can afford to pay that amount while achieving their goals. If you have a company spending more than a million dollars a day on engineering costs, a few dollars extra of insurance cost for test fleets is inconsequential. Perhaps insurance costs will be optimized after deployment, but even then it is economically pressing to do so only when all the much greater business costs have been optimized on a large deployed fleet.

It is unreasonable to expect an AV developer to delay market introduction to improve safety simply to shave a few pennies per mile off their insurance costs. Rather, they will be incentivized to deploy as soon as they can to capture market share – even if they lose money on every mile driven to do so – and worry about incrementally reducing insurance costs later.

Still another consideration is that insurance companies might low-ball quotes to obtain market share. The theory is that if AVs become a big insurance market, it is advantageous for insurance companies to use early policies as loss leaders to in effect “buy” part of the market by establishing early relationships with AV developers. That might mean, for example, that a company could write a policy for an AV tester that is the same rate as for an ordinary vehicle even if the risk might be higher or even largely unknown. Given a fixed payout limit set by a policy cap, the worst-case downside for a crash is limited to that policy cap. The upside is preferential access to a potentially huge future insurance market.

Insurance premiums are further reduced by artificially low insurance policy cap requirements compared to the risk that is likely being taken during testing and early deployment. State laws require insurance maximums to be much lower than reasonably expected jury compensation awards, sometimes no larger than the state insurance minimum requirement for human drivers.[7]

Insurance provides a comparatively weak economic incentive to be “safer.” But as with risk management, that economic incentive runs out of steam when the expected insurance cost becomes small compared to other financial considerations and management imperatives.



[1] Yes, this is a thing even for ships without sails and cannons. 
See:
https://en.wikipedia.org/wiki/Captain_Phillips_(film)

[2] Underwriters Laboratories was founded in the 1890s to help improve fire safety in partnership with the insurance industry. See: https://www.ul.com/about/history

[5]Source: https://policyscout.com/auto-insurance/learn/motorcycle-insurance-vs-car-insurance A possible confounder is number of miles per year driven by motorcycles being less than for cars. But that is unlikely to explain the entire difference here.

[6] If you doubt this, the next time you are at a rental car counter during a quiet shift, take a few minutes to ask the attendant for their worst horror story of damage carelessly or perhaps even intentionally done to a vehicle for which someone purchased the expensive zero deduction insurance waivers.

[7] State insurance requirements for AVs range from state minimums of $25K to $50K up to a high of $5M. Even if a court verdict is higher, the insurance company is only on the hook for the maximum, with the rest of any potential liability falling back on responsible parties – if the plaintiff can manage to collect. For a list of state insurance requirements see:       
https://www.iihs.org/topics/advanced-driver-assistance/autonomous-vehicle-laws

Monday, January 23, 2023

Correct use of terms: regression, bug, glitch, testing, and beta

I just saw another misuse of critical system terminology today by someone who is working on scholarly publications. Because they picked up that habit by working with an autonomous vehicle company. The misuse and abuse of terminology to desensitize people to life critical system defects gets worse all the time. Perhaps it is time to refresh vocabulary for those who have only heard the misuses and don't realize they sound dumb as a box of rocks when they simply repeat what they hear others saying at work. (Caution -- pet peeve meets yelling at clouds here, because the tech industry has invested a decade degrading the meaning of these terms for PR value. If that bothers you just move to the next posting...). 

pexels-thisisengineering-3862610.jpg


Here are some key terms in play. (And it's not just me -- I point to wikipedia entries for each.)

Regression:

This is not the general term for "bug". It is a very specific defect in which a previously operational feature stops working now. Even further back, it was specifically a previous bug fix that stopped working in a new version. If it's a "defect" call it a "defect." Or maybe a "requirements defect" if it is a discovery of behavior you did not previously realize needed to be in the requirements. (https://en.wikipedia.org/wiki/Software_regression)


Bug:

While originally humorous slang, it now does more harm than good to use this term. Use the word "defect" instead. (https://en.wikipedia.org/wiki/Bug_(engineering))


Glitch:

This is a defect too, and even more dangerous to use because it minimizes the issue in a safety critical system. ("It was just a glitch -- let's see if it kills anyone else before we fix it.") The correct use is a transient defect, typically one that is difficult to reproduce. (https://en.wikipedia.org/wiki/Glitch) Whether it is less serious than a permanent defect depends on the body count.


Testing:

Executing a system according to a plan intended to validate engineered behavior by comparing it against expected behavior. Driving a car around to see what happens is not really testing -- it is just messing around. (Messing around can have value to discover requirements, but it is not properly called "testing"). (https://en.wikipedia.org/wiki/System_testing)


Beta:

A specific type of testing carried out by sophisticated early adopters for a product that is believed to be fully functional (if any significant defects, the Beta tester is explicitly warned about each and every one in detail). It is not a legal CYA word for "it doesn't really work, but go ahead and have a go anyway" which is more properly called an engineering prototype, and which should not be sold to the general public via retail channels as if it were a real product. (https://en.wikipedia.org/wiki/Software_testing#Beta_testing)


Sunday, January 8, 2023

The case for AVs being 10 to 100 times safer than human drivers

There is a case to be made that at-scale AV deployments should be at least ten times safer than human drivers, and perhaps even safer than that. The rationale for this large margin is leaving room for the effects of uncertainty via incorporating a safety factor of some sort.[1]

Busy intersection at night

Consider all the variables and uncertainty discussed in this chapter. We have seen significant variability in fatality and injury rates for baseline human drivers depending on geographic area, road type, vehicle type, road user types, driver experience, and even passenger age. All those statistics can change year by year as well.

Additionally, even if one were to create a precise model for acceptable risk for a particular AV’s operational profile within its ODD, there are additional factors that might require an increase:

·        Human biases to both want an AV safer than their own driving and to over-estimate their own driving ability as discussed in a previous section. In short, drivers want an AV driving their vehicle to better than they think they are rather than better than they actually are.

·        Risk of brand tarnish from AV crashes which are treated as more newsworthy than human-driven vehicle crashes of comparable severity. Like it or not, AV crashes are going to be covered by news outlets as a consequence of the same media exposure that created interest in and funding for AV developers. Even if AVs are exactly as safe as human drivers in every respect, each highly publicized crash will call AV safety into question and degrade public trust in the technology.

·        Risk of liability exposure to the degree that AV crashes are treated as being caused by product defects rather than human driver error. For better or worse (mostly for worse), “driver error” is attributed to a great many traffic fatalities rather than equipment failure or unsafe infrastructure design. Insurance tends to cover the costs. Even if a judicial system is invoked for drunk driving or the like, the consequences tend to be limited to the participants of a single mishap, and the limits of personal insurance coverage limit the practical size of monetary awards in many cases. However, the stakes might be much higher for an AV if it is determined that the AV is systematically prone to crashes in certain conditions or is overall less safe than a human driver. A product defect legal action could affect an entire fleet of AVs and expose a deep-pockets operator or manufacturer to having to pay a large sum. Being seen to be dramatically safer than human drivers could help both mitigate this risk and provide a better argument for responsible AV developer behavior.

·         The risk of not knowing how safe the vehicle is. The reality is that it will be challenging to predict how safe an AV is when it is deployed. What if the safety expectation is too optimistic? Human-driven vehicle fatalities in particular are so rare that it is not practicable to get enough road experience to validate fatality rates before deployment. Simulation and other measures can be used to estimate safety but will not provide certainty. The next chapter talks about this in more detail.

Taken together, there is an argument to be made that AVs should be safer than human drivers by about a factor of 10 (being a nice round order of magnitude number) to leave engineering margin for the above considerations. A similar argument could be made for this margin to be an even higher factor of 100, especially due to the likelihood of a high degree of uncertainty regarding safety prediction accuracy while the technology is still maturing.

The factor of 100 is not to say that the AV must be guaranteed to be 100 times safer. Rather, it means that the AV design team should do their best to build an AV that is expected to be 100 times safer plus or minus some significant uncertainty. The cumulative effect of uncertainties in safety prediction, inevitable fluctuations in operational exposure to risky driving conditions, and so on might easily cost a factor of 10 in safety.[2] That will in turn reduce achieved safety to “only” a factor of 10 better than a baseline human driver. That second factor of 10[3] is intended to help deal with the human aspect of expectations being not just a little better than the safety of human drivers, but a lot better, the risk of getting unlucky with an early first crash, and so on.

Waiting to deploy until vehicles are thought to be 100 times safer than humans is not a message investors and design teams are likely to want to hear. But it is, however, a conservative way to think about safety that leaves room for the messiness of real-world engineering to deploy AVs. Any AV deployed will have a safety factor over (or under) Positive Risk Balance (PRB).

The question is whether the design team will manage their PRB safety factor proactively. Or not.


[1] Safety factors and derating are ubiquitous in non-software engineering. It is common to see safety factors of 2 for well understood areas of engineering, but values can vary. A long-term challenge for software safety is understanding how to make software twice as “strong” for some useful meaning of the word “strong.” Over-simplifying, with mechanical structures, doubling the amount of steel should make it support twice the load. But with software, adding twice the number of lines of code just doubles the number of defects, potentially making the system less reliable instead of more reliable unless special techniques are applied very carefully. And even then, predicted improvement can be controversial.
See:
https://en.wikipedia.org/wiki/Factor_of_safety    
https://en.wikipedia.org/wiki/Derating          
and
https://en.wikipedia.org/wiki/N-version_programming

[2] For better or worse – but given the optimism ingrained in most engineers, probably not for better.

[3] Some good news here – by the time you have a safety factor of 10 or more, nuances such as driver age and geofence zip codes start being small compared to the safety factor. If someone says they have a safety factor of 10, it is OK not to sweat the small stuff.

The Tesla Autopilot Crashes Just Keep Coming

Picture from a Tesla AP-related crash just before impact into a disabled vehicle:


(Video here on twitter) Tesla autopilot crashes are still happening when drivers (apparently) succumb to automation complacency. It seems they've just stopped being news.

The above picture is from a Tesla camera a fraction of a second before impact. (Somehow it seems there was no injury.) The Tesla was said to have initiated AEB (and disabled AP) about two seconds before impact. The video shows clear sightline to the disabled vehicle for at least 5 seconds, but the driver apparently did not react.

Tesla fans can blame the driver all they want -- but that won't stop the next similar crash from happening. Pontificating about personal responsibility and that the driver should have known better won't change things either. And we're far, far past the point where "education" is going to move the needle on this issue.

It's time to get serious about:
- Requiring effective driver monitoring
- Addressing the very real #autonowashing problem that has so many users of these features thinking their cars really drive themselves.
- Requiring vehicle automation features to account for reasonably foreseeable misuse (you might fix the vehicle, or you might fix the driver, or more likely fix both, but casting blame accomplishes nothing)

The deeper issue here is pretending that autopilot-type systems involve humans who think they are driving. The car is driving and the humans are along for the ride, no matter what disclaimers are in the driver manual and/or warnings -- unless the vehicle designers can show they have a driver monitoring system and engagement model that provide real-world results.

The reality is that these are not "driver assistance" systems. They are automated vehicles with a highly problematic approach to safety. This goes for all companies. Tesla is simply the most egregious due to poor driver monitoring quality and scale of deployed fleet. As human-supervised automated driving gets more functionality the safety problem will just keep getting worse.

Source on twitter: https://twitter.com/greentheonly/status/1607475055713214464?ref_src=twsrc%5Etfw from Dec 26th: contains video of impact. No injury apparent to the person in the video, but it was a very close thing. Also a screenshot of the vehicle log showing AEB engaged 2 seconds before impact. https://twitter.com/greentheonly/status/1609271955383029763?ref_src=twsrc%5Etfw
For those saying "but Teslas are safer overall" that statement does not stem from any credible data I've ever seen: https://safeautonomy.blogspot.com/2022/12/take-tesla-safety-claims-with-about.html

Wednesday, December 21, 2022

Holiday AV Safety Video Viewing

Daily video & reading suggestions for the holiday season for those into autonomous vehicle safety. Primarily based on new materials from 2022 that you might have missed.

Friday, December 16, 2022

I have to get out NOW from my autonomous vehicle: urgent egress and passenger overrides

What if you need to get out of the vehicle RIGHT NOW in a robotaxi -- is that allowed? What are the implications?

Woman looking out a car window.


In any automated system there will be times when an occupant wants to override the automation, and especially when they want to exit a moving automated vehicle. Reasons might include: wanting to re-open transit vehicle doors if a passenger was unable to exit in time at their stop; an attack of claustrophobia; wanting to get away from another passenger due to personal safety concerns; or even needing to escape a cabin fire. Some egress requests might constitute misuse or abuse, such as stopping a vehicle to intentionally block traffic, or intentionally accessing an off-limits area such as a bridge with no pedestrian infrastructure.

Creating a complete list of all possible motivations is difficult, and weighing the merits of all such egress attempts in advance seems intractable. Nonetheless, there are times when a passenger desire to exit a moving vehicle should be honored, although the vehicle should likely at least stop before permitting an exit.

In still other situations passengers might want to force an otherwise stopped vehicle to move. One reason might be fear for personal safety if threatened by malicious actors while stopped at a traffic light. Another reason might be overriding a police stop if the vehicle occupant suspects a stopping officer is instead a criminal imposter, at least until legitimacy of the police stop can be confirmed via contact with an emergency dispatcher.[1]

Another special situation is one in which a passenger has a compelling reason to order an AV to operate outside its ODD or with degraded equipment in an emergency, even if doing so will result in a reduced safety margin. For example, an AV might be programmed not to drive through heavy smoke, but doing so might be required to escape a burning town in a wildfire situation. The AV occupant might want to take the chance of driving rather than remaining in the burning town.[2]

Human drivers have the authority to deal with these situations so long as they are willing to accept the responsibility. Do you start driving when someone is trying to forcibly break into your car at a traffic light even if you might injure that malicious actor by doing so? The choice – and responsibility for consequences – falls upon the person driving a manually driven car.

The question is: to what degree should an AV support operator overrides of safety-relevant behaviors? A complication is that there might not be a responsible individual in a vehicle to exercise control. What if a passenger is allowed to override some behaviors of the vehicle, but that passenger is impaired, or not capable of exercising mature judgment? Should an 8 year old riding solo be able to command vehicle safety overrides?[3]

Answers as to how much control a passenger should have over AV operation will depend on how stringent qualifications are for a passenger to be capable of mature decision making. It is easy to say there must be one qualified driver if there are any passengers in an AV and that manual controls must be available if needed. However, requiring a qualified driver undermines the potential benefits that AVs might provide for those who are not capable of driving or should not be driving at a particular time.

If other than unimpaired licensed drivers are permitted to override AV behaviors, there will be difficult tradeoffs as to what overrides might be permitted. Likely an 8 year old child should not be permitted to exit a school vehicle in the middle of a highway to avoid going to school. On the other hand, a 14 year old[4] might be considered mature enough to demand an emergency stop if the vehicle tries to drive into flood waters, or initiate an emergency exit with their younger sibling if the cabin fills with smoke from a vehicle battery fire.

Even if a passenger is an adult licensed to drive, should that adult be permitted to override vehicle behavior if drunk or otherwise impaired? If not, should the vehicle disable override capability if the passenger is drunk? Or should it be illegal to enter an AV with override capability when drunk?

While it can be an interesting exercise to conjure extreme situations, the issue of passenger overrides and egress can also be as simple as a passenger saying “I want to get out now” when the vehicle is stopped at a red traffic light but not at the end of the scheduled trip. Should the passenger be able to unlock doors and exit? Or should the passenger be kept locked inside the vehicle until the end of the trip? Should there be a workaround available such as changing the destination? If so, should the passenger have permission to do this if some authority figure such as a parent input the original destination? Where should the threshold be drawn at which such a passenger request is denied both in context (speeding down a highway vs. stopped) or passenger maturity (a passenger one day before turning 18 years old but with no driver’s license vs. grade school age child)?

There is the possibility that remote operators will need to mediate requests for overriding AV behavior either routinely or if there is doubt as to the competence of passengers to make reasonable decisions. However, any such remote operators can be expensive, will have problems scaling, and might result in wait times long enough to impair safety by delaying decisions in urgent situations.[5]

For AVs to be deployed at scale, designers will need to decide how much authority passengers have to override vehicle behavior, and whether emergency manual vehicle controls will be required even in vehicles that are intended to be completely automated. There will be no perfect policy choice, but not setting a consistent policy is also a policy choice.


[1] Yes, this is a thing. Report of an accused police imposter pulling over a van full of legitimate police detectives: https://www.youtube.com/watch?v=ogGBwrrkKY4

[2] This consideration has become especially relevant for residents of California. See: https://www.insideedition.com/how-drive-through-fire-48422

[3] One might say that no 8 year old should ride in an AV solo. But if that is the case, what exactly is the cut-off age? Some public high school systems rely on public mass transit instead of dedicated school buses, so any AV public transit vehicle will have under-age ridership. Is training or perhaps even a “rider license” required to ride in an AV and use the override controls? This topic gets complex quickly.

[4] Some states issue driver licenses to 14 year-olds in special cases. Would such a driver license be required in this case? See: https://www.thedrive.com/news/39184/americas-rarest-drivers-license-lets-14-year-olds-hit-the-road-legally

[5] The usual solution proposed is remote customer service operators that intervene when needed. Those proposing that passengers need have no control because remote operators can solve all safety problems need to spend more time waiting in customer service phone waiting queues. An additional consideration is the likely disruption to emergency response services during a natural disaster that will also require simultaneous attention to numerous AV passenger distress situations. New Year’s Eve screening of requests from potentially drunk passengers will also be challenging.

Wednesday, December 7, 2022

SCSC Talk: Bootstrapping Safety Assurance

Bootstrapping Safety Assurance

Abstract:
The expense and general impracticability of doing enough real-world testing to demonstrate safety for autonomous systems motivates finding some sort of shortcut. A bootstrapped testing approach is often proposed, using evidence from initial mishap-free testing to argue that continued testing is safe enough. In this talk I'll explain why pure bootstrapping based on testing exposure as well as arguments involving "probably perfect" bootstrapping expose public road users to undue risk. Moreover, phased deployments often used to argue safe update release have the same problem. An approach that bootstraps on the safety case rather than on vehicle testing is proposed as a potentially better alternative. While the examples given involve autonomous ground vehicles, the principles involved apply to any argument that safety will be demonstrated via a bootstrap testing process.

This talk was recorded as part of the SCSC Future of Testing for Safety-Critical Systems seminar on Dec. 1, 2022.
Talks and videos are available here (access with paid annual club membership):  https://scsc.uk/e966prog

Free public-access copy of slides here: 




Friday, December 2, 2022

Blaming the autonomous vehicle computer as a regulatory strategy

The AV industry has been successfully pursuing state regulations to blame the computer for any crashes by saying that the Automated Driving System (the computer) is considered to be the driver of any AV operating on public roads. That way there is no person at fault for any harm to road users. Yes, really, that is what is going on.[1]

Person pointing a finger at a computer

The general AV industry tactic when lobbying for such rules is to argue that when fully automated driving is engaged the “driver” is the driving computer (the ADS). Any remote safety supervisor is just there to lend a hand. In some states a remote human support team member need not have an appropriate driver license, because it is said that the ADS that is the driver. Superficially this seems to make sense. After all, if you are a passenger who has paid for a retail robotaxi ride and the AV breaks a traffic law due to some flaw in the design, you as the passenger should not be the one to receive a ticket or go to jail.

But the tricky bit is that ADS computers are not afforded the legal status of being a “person” – nor should they be.[2] Corporations are held to be fictitious people in some legal circumstances, but a piece of equipment itself is not even a fictitious person.[3]

If a software defect or improper machine learning training procedures result in AV behavior that would count as criminally reckless driving if a human were driving, what happens for an AV? Perhaps nothing. If the ADS is the “driver” then there is nobody to put on trial or throw into jail. If you take away the driver’s license for the ADS, does it get its license back with the next software update?[4] Where are the repercussions for an ADS being a bad actor? Where are the consequences?

Blaming the ADS computer for a bad outcome removes a substantial amount of deterrence due to negative consequences because the ADS does not fear being harmed, destroyed, locked up in jail, fined, or having its driver’s license revoked. It does not feel anything at all.

A related tactic is to blame the “operator” or “owner” for any crash. In the early days of AV technology these roles tended to be either the technology developer or a support contractor, but that will change over time. Contractors perform testing operations for AV developers. Individual vehicle owners are operators for some AV technology road tests. Other AV operators might work through a transportation network service. Someone might buy an AV in the manner of a rental condo and let it run as a robotaxi while they sleep.

Imagine an arrangement in which an investor buys a share in a group of robotaxis as might be done for a timeshare condo. A coordinator lines up independent contractors to manage investment money, negotiate vehicle purchases, arrange maintenance contracts, and participate in a ride-hailing network. Each AV is the sole asset of a series LLC to act as a liability firewall between vehicles. The initial investor later sells their partial ownership shares to an investment bank. The investment bank puts those shares into a basket of AV ownership shares. Various municipal retirement funds buy shares of the basket. At this point, who owns the AV has gotten pretty complicated, and there is no substantive accountability link between the AV “owner” and its operation beyond the value of the shares.

Then a change to the underlying vehicle (which was not sold as an AV platform originally, but rather was adapted by an upfitter contractor) impairs functionality of the aftermarket add-on ADS manufactured by a company that is no longer in business. If there is a crash who is the “operator?” Who is the “owner?” Who should pay compensation for any harm done by the AV? If the resultant ADS behavior qualifies as criminally negligent reckless driving, who should go to jail? If the answer is that nobody goes to jail and that only the state minimum insurance of, say, $25K pays out, what is the incentive to ensure that such an arrangement is acceptably safe so long as the insurance is affordable compared to the profits being made?

While the usual reply to concerns about accountability is that insurance will take care of things, recall that we have taken some passes at discussing insurance and risk management can be insufficient incentive to ensure acceptable safety, especially when it only meets a low state minimum insurance requirement[5] originally set for human drivers that have skin in the game for any crashes.


[1] For a compilation of US state laws and legislative hearing materials see:        https://safeautonomy.blogspot.com/2022/02/kansas-av-regulation-bill-hearings.html

[2] Despite occasional hype to the contrary, machine learning-based systems are nowhere near achieving sentience, let alone being reasonably qualified to be a “person.”

[3] I am not a lawyer (IANAL/TINLA), so this is a lay understanding of the rules that apply and nothing in this should be considered as legal advice.

[4] In several states an ADS is automatically granted a driver’s license even though it is not a person. It might not even be possible to take that license away.

[5] IIHS/HLDI keeps a list of autonomous vehicle laws including required insurance minimums. The $1M to $5M numbers fall short of the $12M statistical value of human life, and are typically per incident (so multiple victims split that maximum). In other states the normal state insurance requirement can apply, which can be something like a maximum of $50,000 per incident and might permit self-insurance by the AV company, such as is the case in Kansas: https://insurance.kansas.gov/auto-insurance/ This insurance maximum payout requirement is less than the cost of a typical AV. In practice it might be the case that victims are limited to recovering insurance plus the scrap value of whatever is left of the AV after a crash, with everyone else being judgement-proof.