The Mystery of AI Gunshot-Detection Accuracy Is Finally Unraveling

Liz González’s vicinity successful East San Jose tin beryllium loud. Some of her neighbors seemingly privation the full artifact to perceive their cars, others similar to airy fireworks for each occasion, and occasionally determination are gunshots.

In February 2023, San Jose began piloting AI-powered gunshot detection exertion from the institution Flock Safety successful respective sections of the city, including Gonzalez’s neighborhood. During the archetypal 4 months of the pilot, Flock’s gunshot detection strategy alerted constabulary to 123 shooting incidents. But new data released by San Jose’s Digital Privacy Office shows that lone 50 percent of those alerts were really confirmed to beryllium gunfire, portion 34 percent of them were confirmed mendacious positives, meaning the Flock Safety strategy incorrectly identified different sounds—such arsenic fireworks, construction, oregon cars backfiring—as shooting incidents. After Flock recalibrated its sensors successful July 2023, 81 percent of alerts were confirmed gunshots, 7 percent were mendacious alarms, and 12 percent could not beryllium determined 1 mode oregon the other.

For 2 decades, cities astir the state person utilized automated gunshot detection systems to rapidly dispatch constabulary to the scenes of suspected shootings. But reliable information astir the accuracy of the systems and however often they rise mendacious alarms has been difficult, if not impossible, for the nationalist to find. San Jose, which has taken a starring role successful defining liable authorities usage of AI systems, appears to beryllium the lone metropolis that requires its constabulary section to disclose accuracy information for its gunshot detection system. The study it released connected May 31 marks the archetypal clip it has published that information.

The false-positive complaint is of peculiar interest to communities of color, immoderate of whom fearfulness that gunshot detection systems are unnecessarily sending constabulary into neighborhoods expecting gunfire. Nonwhite Americans are more often subjected to surveillance by the systems and are disproportionately killed successful interactions with police. “For us, immoderate enactment with constabulary is simply a perchance unsafe one,” says Gonzalez, an organizer with Silicon Valley De-Bug, a assemblage advocacy radical based successful San Jose.

San Jose did not effort to quantify however galore shooting incidents successful the covered country the Flock System failed to detect, besides known arsenic the false-negative rate. However, the study says that “it is wide the strategy is not detecting each gunshots the section would expect.”

Flock Safety says its Raven gunshot detection strategy is 90 percent accurate. SoundThinking, which sells the ShotSpotter system, is the astir fashionable gunshot detection exertion connected the market. It claims a 97 percent accuracy rate. But the information from San Jose and a fistful of different communities that utilized the technologies suggest the systems—which usage machine algorithms, and successful SoundThinking’s case, quality reviewers, to find whether the sounds captured by their sensors are gunshots—may beryllium little reliable than advertised.

Last year, journalists with CU-CitizensAccess obtained data from Champaign, Illinois, showing that lone 8 percent of the 64 alerts generated by the city’s Raven strategy implicit a six-month play could beryllium confirmed arsenic gunfire. In 2021, the Chicago Office of Inspector General reported that implicit a 17-month play lone 9 percent of the 41,830 alerts with dispositions that were generated by the city’s ShotSpotter strategy could beryllium connected to grounds of a gun-related crime. SoundThinking has criticized the Chicago OIG report, saying it relied connected “incomplete and irreconcilable data.”