The business model and exit plan for an AV company can powerfully incentivize behavior that is at odds with public safety and transparency. This is probably not news regarding any private company, but it is especially a problem for AV safety.
An AV developer with a plan to develop, deploy, and long-term sustain their technology should be incentivized to reach at least some level of safety subject to all the ethical issues discussed already in this chapter. If they do not, they will probably not have a viable long-term business. Arguments for a light regulatory touch often make this argument that companies will act in their own long-term best interest. But what if the business incentive model is optimized for something shorter than the “long-term” outcomes?
Short-term aspects of the business objectives and the
business structure itself can add pressure that might tend to erode any
commitment to acceptable safety. Factors include at least the following,
several of which can interact with each other:
·
Accepting money from traditional
venture capital sources can commit a company to a five-year timeline to produce
products. Thus far we have seen that five-year timelines are far too aggressive
to develop and deploy an AV at scale. Re-planning and raising more funding can
lengthen the timeline, but there remains risk that funding incentivizes
aggressive milestones to show increased functionality and, in particular,
remove safety drivers rather than core efforts on safety. Some companies will
likely be better at resisting this pressure than others.
·
A business exit plan of an Initial
Public Offering (IPO), going public via a Special Purpose Acquisition Company
(SPAC), or being bought out by a competitor historically emphasize perceived
progress on functionality rather than safety. If the exit plan is to make
safety someone else’s problem post-exit, it is more difficult to justify
spending resources on safety rather than functionality until the company goes
public.[1]
·
The AV industry as a whole takes
an aggressively non-regulatory posture, with that policy approach historically
enabled by US DOT.[2]
This situation forces little, if any, accountability for safety until crashes
happen on public roads. There is a tendency for at least some companies seem to
treat safety more as a public relations and risk management function than a
substantive safety engineering function. Short-term incentives can align with a
dysfunctional approach.
·
Founders of AV companies with a
primarily research, consumer software, or other non-automotive background might
not appreciate what is involved in safety at scale for such systems. They might
earnestly – but incorrectly – believe that when bugs are removed that
automatically bestows safety, or otherwise have a limited view of the different
factors of safety discussed in chapter 4. They might also earnestly believe
some of the incorrect talking points discussed in section 4.10 regarding safety
myths promoted by the AV industry.
·
The mind-boggling amount of money
at stake and potential winnings for participants in this industry would make it
difficult for anyone to stay the course in ensuring safety in the face of rich
rewards for expediency and ethical compromise. No matter how pure of spirit and
well intentioned.
It is impossible to know the motivations, ethical
framework, and sincerity of every important player in the AV industry. Many
participants, especially rank and file engineers, are sincere in their desire
to build AVs and believe they are helping to build a better, safer world.
Regardless of that sincerity, it is important to have checks and balances in
place to ensure that those good intentions translate into good outcomes for
society.
One has to assume that outcomes will align with incentives. Without checks and balances, dangerous incentives can be expected to lead to dangerous outcomes. Checks and balances need to be a combination of internal corporate controls and government regulatory oversight. A profit incentive is insufficient to ensure acceptable safety, especially if it is associated with a relatively short-term business plan.
[1] Safety theater money
spent to impart an aura of safety is a different matter, and spending on this
area can bring good return on investment. But we are talking about real safety
here. In the absence of a commitment to conform to industry safety standards it
can be difficult to tell the difference without a deep dive into company
practices and culture.
[2] The official US
Department of Transportation policy still in effect at the time of this writing
states: “In this document, NHTSA offers a nonregulatory approach to automated
vehicle technology safety.” See page ii of:
https://www.nhtsa.gov/document/automated-driving-systems-20-voluntary-guidance
