AMCI Testing releases its 2nd set of Tesla “Full Self Driving” (Supervised) evaluation videos – once again demonstrating unreliable performance and potentially dangerous inconsistencies
Far from perfect results underscore a growing need for regulatory standards that define failures and “interventions”
Los Angeles, CA. October 2, 2024
AMCI Testing continues its extensive, 1000-mile evaluation of the Tesla Full Self Driving (Supervised) system in advance of the company’s October 10th Robotaxi reveal event. Six newly filmed driving scenarios are available at the link below. As with previous videos, these continue to show that, although FSD can often drive the car competently for limited distances across a wide range of scenarios, the mistakes it does make continue to put occupants and the public at significant risk.
Beyond FSD’s actual performance in the instances shown, these potentially dangerous driving errors demonstrate the incontrovertible need for regulation and agreed upon metrics for comparable, system-to-system evaluation. One of the key metrics to define across all types of autonomous and semi-autonomous systems is what constitutes an “intervention” – which system actions warrant one, and how they should be scored by regulators whenever one occurs.
When AMCI Testing Intervenes
AMCI Testing’s protocol requires an intervention (the driver taking control and forcing disengagement of the system) whenever FSD’s actions put the occupants, the public or other motorists at risk. As stated in our previous release, FSD’s behavior required 75 interventions in 1000 miles of real-world testing, for an average forced system disengagement rate of 1 every 13 miles.
“The key consideration is, currently can any hands-free system be ethically operated by consumers on public roads. To advance the safety of the industry, AMCI Testing has now articulated a standard. But really that is the question – should we or the public be generating a standard or is this the responsibility of a Federal or State Regulator to arrive at an ‘intervention’ standard?” asked David Stokols, CEO of parent company, AMCI Global. “Further, if there are too many incidents, as we have seen in AMCI Testing’s results then the public will lose confidence in all FSD and Robotaxi-type software solutions from any OEM.”
We have found FSD’s evolving programming and unexpected changes between software versions as proof of the critical need for more specific regulation and oversight. The obvious example is when “Autoland” was being developed for airliners in the mid-1970s, designed to allow zero-visibility operation with many fewer variables than occur on a public road. Certification required a failure probability per occurrence of less than 1 in 150,000.
Arguably, the 1 in 150,000 goal is what we should be aiming for in road-based autonomous systems. “Extrapolate the failure rate AMCI testing experienced in only 1000 miles of driving with FSD (Supervised) and you can see the Tesla system is nowhere near that mark. Additionally, FSD does not appear to be on a progressive, problem-solving track. There are inexplicable performance regressions that sometimes occur as the software updates,” said Guy Mangiamele, Director of AMCI Testing.
“For example, during our testing on version 12.5.1, FSD would only command the car out of the left lane on the freeway and toward the exit ramp 0.2 miles before the ramp itself. This distance is already too short. Yet in the later 12.5.3 update, the distance was even further reduced to just 0.1 mile, regardless of the surrounding traffic. Not only does this unpredictable sort of behavior keep the user guessing about the system’s intentions, but it would make the system’s performance nearly impossible to assess within a Certification-program architecture.”
AMCI Testing has dropped the next 3 in the series of test-videos intended to demonstrate the complex issues of trust and performance that FSD continues to pose to drivers and the public. Please follow the link for all the test-videos released to date at https://amcitesting.com/tesla-fsd/ We will continue to evaluate subsequent iterations of Tesla’s FSD as they become available. Go to www.amcitesting.com to sign up to receive updates as they occur.
For further information please contact:
Ian Beavis, Chief Strategy officer at AMCI Global; Ph 714 813 6300 or Email ibeavis@amciglobal.com
About AMCI Testing
AMCI Testing is an independent automotive research firm, specializing in unbiased, exclusive, comparative evaluations of automotive products since 1984. The breadth of our testing includes ICE, HEV, PHEV, BEV, FCEV powertrains and every facet of measurement and product category. AMCI Testing Certification is recognized globally as an industry gold standard.
