Last week, Mayor Eric Adams announced that he was kicking off a 90-day waiting period to test weapons scanners on New York City's subway system.
The scanners that he demonstrated, provided by Evolv Technology, purport to mix physical detection technology with artificial intelligence to indicate when someone is carrying a gun or a knife, and have been used in City hospitals before, something Adams alluded to at the press conference at Fulton Street station.
"We are going to make sure, based on our numbers, hit ratio, false hit, we're going to do our data," Adams told reporters. "What I'm hearing from my corporations, from my hospitals, from others, they're saying this is living up to our expectation."
What Adams didn't mention was that the City already has some data, and it's not good. Evolv's scanners were installed at Jacobi Medical Center in the Bronx in 2022, and triggered huge numbers of false positives, according to the results of the pilot obtained by Hell Gate through a public records request filed with NYC Health + Hospitals.
Over the seven months that the Jacobi pilot was active, 194,000 people passed through Jacobi's scanners, and in just over 50,000 of those cases, the scanners threw up an alarm—an incidence rate of around 26 percent, or over one of every four times someone passed through the scanner. Of those 50,000 alarms, around 43,800, or a little more than 85 percent, were false positives; 7,027 of the alarms, or 14 percent, were law enforcement officers who were presumably carrying their service weapons; and just 295 alarms, or 0.57 percent, were determined to be a non-law enforcement person carrying either a knife, a gun, or a threat type labeled only "other," which likely entails other weapons like bats.
Notably, Evolv's scanners did not get any more accurate as the pilot progressed—there was not a single month where the alarm to visitor ratio fell below 25 percent. In September of 2022, the final month of the seven-month pilot, 27,900 visitors passed through the scanners, and nearly 7,000 threw alarms; out of those 7,000 alarms, just 345 potential threats were identified—a false positive rate of 95 percent, with only 0.45 percent of alerts being non-law enforcement threats. Throughout the entire pilot, the alerts led to the finding of 24 guns, 139 knives, and 132 other potential threats, out of the 50,000 alarms sounded.
"It's essentially a coincidence," said Daniel Schwarz, senior privacy and technology strategist for the New York Civil Liberties Union, who reviewed the pilot data at Hell Gate's request, and was referring to the small number of weapons that were actually found. "They're searching all these people and then they accidentally find something on them."
NYCLU has been vocal in opposing not just these scanners but various other NYPD surveillance technologies not just on the basis that they violate New Yorkers' civil liberties, but also because they don’t work particularly well, and their use can create unintended consequences. On the subways, the concern is that the tech would not only potentially cost millions—each scanner is a service the company offers for around $2,500 a month, and police officers would have to oversee them—but clog up a system that relies on speed and ease of use and put cops in a confrontational posture over the constant alarms.
"At the scale of the NYC subway system, if we envision the four million daily riders, there will be one million false flags or alerts, bringing the whole system to a standstill," Schwarz said. Evolv CEO Peter George himself said in an investor call earlier this month that "subways in particular are not a place that we think is a good use-case for us." (Evolv did not respond to a request for comment.)
Broadly, the system works similarly to a metal detector but aims to specifically detect weapons as opposed to metal objects more generally. It has sensitivity settings set by a user; at higher sensitivities, it will go off more often, detecting additional guns and knives but also generating more false positives. At lower sensitivity settings, it will stop sounding as many bogus alarms, but also miss more actual weapons. In one infamous case, a school district in Illinois found that the tech was consistently labeling Chromebook laptops as guns. It’s unclear at what exact sensitivity settings the Jacobi system was set.
Though it's been two years since the Evolv pilot at Jacobi, there are good indications that the technology hasn’t progressed to the point that it’s reliable. Evolv’s own shareholders filed a class-action suit against the company this month arguing that the company’s marketing materials and public pronouncements overstated the effectiveness of the technology. This follows the start of investigations by both the Federal Trade and the Securities and Exchange Commissions. There are individual accounts of the technology failing to detect weapons at schools around the country, while throwing false alarms for innocuous objects.
The mayor gave mixed messages as to whether Evolv itself will ultimately be the vendor for the latest scanner effort. During his March 28 announcement, Adams said that "the NYPD will work to identify all vendors with effective technology and expertise," and later said in response to a question that the bidding process was "going to be extremely competitive, the best product is going to be used in our city." Nonetheless, he used an Evolv scanner to demonstrate the technology, and said "the company that we're partnering with and announcing today is Evolv."
In response to our questions about whether the Jacobi data raised any concerns and why an additional pilot might be necessary, a City Hall spokesperson wrote that "last week's announcement was never about one specific company, it was about our intention to use every tool available…Over the course of this 90-day waiting period, we will engage with as many tech companies as possible that work in this space to see what our best option for a pilot program is. Our pilot program will then test to see what works, what doesn't."
Evolv has a handful of competitors, including companies like ZeroEyes and Patriot One, which enjoyed a flurry of attention and funding in the aftermath of mass shootings around 2018 and 2019. Still, no weapons scanner technology has conclusively proven that it is reliably accurate.
In addition to Jacobi Hospital, Evolv scanners were also placed briefly at City Hall. We filed a freedom of information request for the results of that 2022 pilot, but the NYPD denied it on the dubious claim that the data "would reveal non-routine techniques and procedures."
"We have no proof that this technology is effective in finding guns," Schwarz said. "The only proof we have is that it’s not."
