Monday, January 29, 2024

The Exponent Report on the Cruise Pedestrian Dragging Mishap

On October 2, 2023, a Cruise robotaxi Autonomous Vehicle (AV) in San Francis-co struck and then later – as part of different maneuver – dragged a pedestrian under the vehicle as part of a more complex road mishap. The circumstances included a different human-driven vehicle striking the pedestrian first and what amounts to an attempted cover-up by Cruise senior management of the final portion (the dragging part) of the mishap. High level Cruise leadership was largely sacked. A significant fraction of staff were let go as well. A 3rd party external review was commissioned. Now we have the report..

Quinn Emanuel Investigation report:  Original Link / Archive.org link

High level takeaways based on the Exponent analysis (just the technical portion of the report):

  • A computer-speed response could have avoided the initial crash between the Cruise AV and the pedestrian entirely. A slightly less quick response could have significantly reduced harm. This would have required the AV to recognize a person suddenly appearing in the travel lane in front of it, which it was not able to do. "Unavoidable" is not strictly true.
  • The Cruise AV accelerated toward a pedestrian it knew was directly in front of the AV in a crosswalk, apparently because its end-to-end neural network predicted the pedestrian would be out of the way by the time it got there. That additional speed contributed to having a tight reaction time.
  • The Cruise AV tracked the pedestrian through the collision with the adjacent Nissan. It continued accelerating even though a pedestrian had just been hit in an adjacent lane, additionally reducing available reaction time.
  • The Cruise vehicle had the legs of the trapped pedestrian in camera view during the entirety of the dragging event. In other words it had a sensor that saw the entrapment well enough that post-crash analysis confirmed that view. But that available sensor data did not result in a detection. Rather, the vehicle eventually stopped because of the resistance and excessive wheel spin caused by a rear wheel running up and over the pedestrian's legs and potentially other impediments to movement that might have been caused by the pedestrian trapped under the vehicle.
  • The Cruise vehicle definitely knew it had hit something, but decided to initiate a pull-over maneuver and restart motion for <reasons>. Exponent states: "an alert and attentive human driver would be aware that an impact of some sort had occurred and would not have continued driving without further investigating the situation"

Details follow:

Graph excerpt (page 83):

I'll start by saying it is good to see that Cruise released the 3rd party reports instead of keeping them secret. Sadly, the technical report is heavily redacted, but here is what I can see from what I have to work with.

Both the main report and the attachment are masterpieces of saying the most favorable thing possible given a situation that reflects poorly on the client sponsoring the project. I leave analysis of the lawyer-written portion to others. Here I focus on the technical portion written by Exponent. (Page numbers are from the Exponent report, which starts at page 1 of the appendix.) 

This is not an exercise in blaming Cruise technical workers for what happened. They were clearly pressured by those at the top to put technology into service before it was fully ripe. Rather, this is an exercise in showing how an apparently objective (superficially) technical report can severely skew the narrative. Perhaps some will say this analysis pushes the other way. But what the reader should not do is swallow the Cruise-sponsored narrative whole. Doing that will not serve the much-needed imperative to get Cruise on a track to safe deployment of this technology.

The Exponent report is written in what looks like an expert witness style for use in any potential court case. Exponent does a lot of work for car companies like that, and the "tell" is the phrase: "The findings presented herein are made to a reasonable degree of engineering and scientific certainty." (pg. 13)  This blog post is an informal look based on limited time (nobody paid me to do this post -- but I can assure you Exponent gets paid plenty for their work). Since the report is heavily redacted I might well change my thoughts based on further information or further consideration of the information already available. Nonetheless, here is what I got out of what I could see. I welcome any factual corrections.

  • Despite a previous statement by Cruise that Exponent's scope would be expanded to recommending safety engineering and process improvements, Exponent explicitly stated that these were out of scope for this report. Perhaps that is a separate effort, but that topic is explicitly out of scope for both this report and the Quinn Emmanuel main report.
    • Page 13: "A review of Cruise's overall safety systems, culture, and technology is beyond the scope of this investigation."
  • As documented by Exponent it seems the pedestrian was hurrying across an intersection right after the light had changed.
    • Page 44, table 6: 
      • Light changes at -10.0 seconds
      • Pedestrian visually appears to enter crosswalk at -7.9 seconds
      • 2.1 seconds have elapsed + dwell between opposing light changes, if any
      • Both Cruise and adjacent other vehicle have started moving at this point from the far side of the intersection.
  • The Cruise AV accelerated directly toward the pedestrian while that pedestrian was in a crosswalk in its own travel lane.
    • Page 83 figure 62: AV acceleration is positive between the times the pedestrian enters and exits the Cruise AV's lane. Speed increases from about 5 mph to about 13 mph during that time (visual approximate estimate from graph).
    • California Rules of the Road have an explicit that a vehicle in this situation should reduce speed:
      • "The driver of a vehicle approaching a pedestrian within any marked or unmarked crosswalk shall exercise all due care and shall reduce the speed of the vehicle or take any other action relating to the operation of the vehicle as necessary to safeguard the safety of the pedestrian." (21950(c))
  • Exponent both says that the pedestrian crash "may not have been avoidable" and that the vehicle could have avoided the crash. The word "may" is doing some really heavy lifting here. In fact, Exponent admits that the vehicle could have avoided the crash with a computer-speed fast response to a pedestrian in its path (the kind we have been promised every time an AV advocate says that computers can react faster than people to problems):
    • Page 16: "Calculations of potential AV stopping distance indicate that a collision of the AV with the pedestrian may not have been avoidable, even if the ADS had reacted to the collision between the Nissan and the pedestrian."
    • Page 66: "Accounting for brake system latency, the system would have needed to initiate a brake request no later than 0.78 seconds prior to AV-pedestrian contact in order to completely avoid contact. At this relative time, the pedestrian had just fallen into the AV’s travel lane, the AV was traveling at approximately 18.4 mph and the AV was approximately 6.55 m (21.5 ft) from the point at which AV-pedestrian contact occurred. It is noteworthy that a hypothetical brake activation occurring after this time (and prior to when the AV initiated braking at 0.25 s) would have potentially mitigated the severity of the initial collision between the AV and the pedestrian."
  • The Cruise vehicle maintained tracking with the pedestrian until almost a second after the impact with the adjacent vehicle. So it had the opportunity to detect that something very bad was happening in terms of a pedestrian hit by a vehicle a few feet from its lane, but it did not react to that available information.
    • Page 15: "As evidenced by the video and sensor data, the classification and tracking of the pedestrian became intermittent within 1.0 s after the initial contact between the pedestrian and the Nissan until the last correct object classification occurred at approximately 0.3 s prior to the collision between the AV and the pedestrian. This intermittent classification and tracking of the pedestrian led to an unknown object being detected but not accurately tracked by the automated driving system (ADS) of the AV and the AV detected occupied space in front of the AV."
    • Also see Table 6 on page 44:  0.9 seconds between Nissan contact and pedestrian track ID being dropped. Potentially much of that 0.9 seconds was waiting for tracking to recapture the pedestrian during an interval of missing detections.
  • The AV was accelerating during the entire event, including speeding up from 17.9 mph to 19.1 mph during the time the pedestrian was being struck by the adjacent Nissan.
    • Page 15: "At this separation time, the AV was traveling at a speed of approximately 17.9 mph and was approximately one car length behind the Nissan in the adjacent right lane."
    • Page 15: "This deceleration resulted in a vehicle speed reduction from approximately 19.1 mph, prior to the onset of braking, to approximately 18.6 mph at the time of impact with the pedestrian."
  • Exponent admits that lack of anticipation of a problem contributed to the mishap. It then attempts to excuse this by saying a human driver could not react to the pedestrian strike -- but we were promised robots would be better than humans.
    • Page 17: "The AV’s lack of anticipation of a potential future incursion of the pedestrian into its travel lane was a contributing factor to this incident. Reasonable human drivers would face challenges reacting to the pedestrian being projected into their lane of travel and would likely not have been able to avoid the collision under similar circumstances. This difficulty could be due to violations of expectancy, glare, or A-pillar obstruction, or a combination of these, as well as to a failure to predict the collision of the Nissan with the pedestrian in the adjacent lane and/or the resulting redirection of the pedestrian into their lane of travel. Moreover, reasonable human drivers would not likely have had adequate time to avoid the collision once the pedestrian was struck by the Nissan."
  • In fact, the AV did not realize it was hitting the pedestrian. Rather it noticed something ("occupied space") was in front of/beside it, which apparently it does treat as a Vulnerable Road User. It did not brake until 1/4 second before impact. The vehicle decelerated from 19.1 mph to 18.6 mph before the strike. It did in fact have a lidar observation of the pedestrian's leg, but did not classify this as a pedestrian. (A pedestrian prone in the travel lane after having fallen, tripped, been shoved, etc. is an eminently foreseeable road hazard.)  Any statement about initiating aggressive braking before impact is a bit over-stated. To be sure, it takes a while to put on the brakes due to physical constraints. It is more like a case of better late than never, with aggressive braking more properly characterized as initiating before impact but actually taking place after impact.
    • Page 15: "The ADS started sending steering and braking commands to the vehicle at approximately 0.25 s prior to the collision between the AV and the pedestrian due to the detection of occupied space in front of the AV. Consequently, just prior to the collision with the pedestrian, the AV’s heading momentarily changed rightward, and the vehicle began decelerating. This deceleration resulted in a vehicle speed reduction from approximately 19.1 mph, prior to the onset of braking, to approximately 18.6 mph at the time of impact with the pedestrian."
    • Page 16: "Only the pedestrian’s raised leg, which was bent up and out toward the adjacent lane, was in view of these lidar sensors immediately prior to collision."
    • Page 83 figure 62: braking force at best -0.4g or -0.5g at time of impact, spiking down to perhaps -1.2g right after impact.
  • Exponent gives <reasons> why the dragging occurred, but they admit that a human driver would not have made the mistake of dragging a pedestrian under the vehicle:
    • page 17: "After the AV contacted the pedestrian, an alert and attentive human driver would be aware that an impact of some sort had occurred and would not have continued driving without further investigating the situation."
  • Exponent confirms that the pedestrian was dragged approximately 20 feet at speeds of up to 7.7 mph. This dragging occurred entirely after the vehicle initially stopped post-impact. This does not include whatever dragging might have occurred during the initial impact and initial stop.
    • Page 16: "During this maneuver, the AV reached a speed of 7.7 mph and traveled approximately 20 feet while dragging the pedestrian before reaching its final rest position."
  • The AV had camera data that it had a pedestrian trapped under it the whole time, but failed to recognize the situation. So any claim that this was all due to lack of a "trapped pedestrian sensor" would not be the whole story.   
    • Page 16: "The pedestrian's feet and lower legs were visible in the wide-angle left side camera view from the time of the collision between the pedestrian and the AV through to the final rest position of the AV."
  • The vehicle did a degraded mode shutdown after the dragging started not because it explicitly recognized it hit a pedestrian, but rather because it noticed something was wrong with the spinning of the wheel that was over the pedestrian's legs. Note that they say the degraded state initiated an "immediate stop" which took 3 seconds to complete even though the speed was slow.
    • Page 16: " A traction control system event was recorded at approximately 3.8 s after the initial contact between the pedestrian and the AV due to the pedestrian physically resisting the motion of the vehicle. An accumulated offset between the wheel rotation of the left-rear wheel relative to the others from the wheel speed sensors led to the AV entering a degraded state approximately 5.8 s after the initial contact between the pedestrian and the AV. This degraded state caused the vehicle to initiate an immediate stop, and the vehicle reached its final point of rest approximately 8.8 s after the initial contact between the pedestrian and the AV."
  • Exponent claims they know of no failures or faults that contributed to the incident. This is an odd statement -- does that mean dragging a pedestrian is intended functionality? More likely if pressed their expert would say it was due to a "functional insufficiency." But if that is the case, it means they had deployed a vehicle into public use as a commercial service that was not fully capable of operating safely within its intended Operational Design Domain. In other words they seem to be saying it was not "broken" -- just unsafe.
    • Page 14: "Exponent did not identify any evidence of reported vehicle, sensor, actuator, or computer hardware failures or software faults that could have contributed to the incident."
    • Note that they did not opine there were no failures/faults, but rather that they "did not identify any evidence" -- which is a somewhat different statement since there is no sign they did a source code review, hardware diagnostics themselves etc. This limits the scope of their statement more than might be obvious at a quick read.

We also learned (page 29): "The Cruise ADS employs an end-to-end deep learning-based prediction model in order to interpret the motion of tracked objects and contextual scene information in order to generate predicted object trajectories."   I have no idea how they might validate an end-to-end model for life critical applications used in a system which, apparently, is happy to accelerate toward a pedestrian in a crosswalk because the black box model says it will work out fine.  (Maybe there is a deterministic safety checker, but it obviously did not work well enough in this mishap.)

Thursday, January 25, 2024

Cruise Pedestrian Dragging Incident Report

Cruise has released materials from an investigation into the pedestrian dragging incident and related external communication transparency failures of last October. There is a lot to unpack, but remarkably little of it truly has to do with safety. Most of it seems to be an exercise in blaming senior leadership (who have largely been sacked) and explaining why the problems were more due to misguided individuals and poor leadership rather than malefaction. 

Cruise Blog post: Original Link / Archive.org link
Quinn Emanuel Investigation report:  Original Link / Archive.org link

The explanations in the report include some pretty remarkable events. Repeated internet disruptions across multiple meetings only for the regulatory audience when showing the videos that affected the pedestrian dragging part.  Combined with intentionally not showing the disrupted portions to others, including especially the press. And failing to correct mistaken impressions that were favorable to Cruise (while aggressively trying to fix ones that were unfavorable). And leaving the bad part out of public statements. And a paralegal who just didn't realize they should be putting the pedestrian dragging part into a report to NHTSA. Twice. And not talking to external parties who attended meetings to hear their side of the story in preparing this very investigative report when some key points had mixed evidentiary support.

Regardless of what you might think of the report veracity, my safety takeaways are:

  1. These reports do not actually address safety and safety processes. Initially Cruise said those topics would be addressed. However the Quinn Emanuel report specifically states those are out of scope, and instead limits inquiry to regulatory compliance and media relations issues. The Exponent report is clearly limited to root cause for that specific crash. Perhaps someone is working on an independent look at safety and safety processes, but it is not mentioned anywhere I could find.
  2. A pivotal moment (perhaps this will be the pivotal moment in retrospect) was in the timeline (page 12) Oct 3, 3:21 AM where the Director of Systems Integrity for Cruise created a video that, at the request of Cruise government affairs, left out the strike of the pedestrian and subsequent dragging. My recollection from listening to at least one journalist is that this video was later shown to journalists. An intentional decision not to tell the whole truth to the public gave regulators ammunition to blame Cruise for holding back regardless of the ultimate facts of those discussions.
  3. The significantly redacted Exponent report has some new information about the crash dynamics, including the vehicle speed at impact. Those are of interest to understanding the root cause, but have little to do with much bigger safety concerns. The very final sentence of the report is the most telling, and accurately summarizes the big safety concern for this particular mishap: "After the AV contacted the pedestrian, an alert and attentive human driver would be aware that an impact of some sort had occurred and would not have continued driving without further investigating the situation."
Cruise's narrow technical problem boils down to their vehicle continuing driving even though it knew that some sort of impact had occurred. Their regulatory/governance problem of the moment is the scope and intent of the cover-up.

Cruise's bigger safety problems are out of scope for this report, and from everything I've seen remain unaddressed. That topic will have to be resolved for Cruise to remain viable.

Saturday, January 20, 2024

Vay deployment of tele-driven cars

Vay is deploying tele-driven cars in Las Vegas. At a first pass they have done some safety homework, but open questions remain that they should address to instill public confidence. I spent some time doing background research before posting (especially listening to this All About Autonomy podcast with their founder https://lnkd.in/eYQH72EE ).



Initial summary of their system:
- The operational concept is a rental car ferry service with a tele-driving kit on top of an ordinary vehicle. 100% remotely driven between rentals, and a regular human driver during a rental period.
- Telecom disruptions result in the vehicle doing a safing response of some sort, details unspecified.
- They claim conformance to ISO 26262 (functional safety), ISO 21448 (SOTIF), and ISO 21434 (security) with TUV certification. This is some good news for teleoperation. Would like to know if ISO 26262 includes software (companies often play a game and just do hardware; don't know the case for Vay). Does not address full scope of autonomous safing response when needed.
- They mention, but do not claim conformance to, UL 4600. That standard applies both when driving and during an autonomous safing response.
- This is an SAE Level 4 vehicle because it must completely control itself during a safing response. (This is another reason SAE J3016 levels are unsuitable for regulation, but that is what states are using nonetheless.)
- Limited to slow speeds. They claim they are characterizing remote connectivity lag and modulating maximum speed accordingly, etc.

Initial Thoughts:
- There is no safety report and no VSSA evident on their company web page. They would be well advised to be more public about safety before the first crash.
- The timing, dynamics, and human/machine interface issues of remote driving are potentially very problematic. They say they have a handle on it. Because of opacity on safety (typical of the industry) we either have to take their word for it -- or not. But at least they acknowledge it is a concern, and that is better than the obvious cluelessness of some previous companies on this topic. Driver training will be huge.
- I'll bet at this point the failure response to communications loss is an in-lanes stop. They say they have redundancy but we'll just have to see how this works out for both safety and stranded cars. We won't really know until they try to scale up.
- I'd like to know what will happen if there is a catastrophic large-scale communication disruption -- are all the cars stranded in travel lanes impeding emergency response vehicles? For example deploying in cities prone to earthquakes this is an important question.

Takeaways:
- My take is that safe remote driving is a strong claim, requiring strong evidence.
- There is a risk any crash will be blamed on the remote driver regardless of any contributing technology failure. We'll have to see how that turns out.