The National Security Agency is harvesting huge numbers of images of people from communications it intercepts through its global surveillance operations for use in sophisticated facial-recognition programs, according to top-secret documents.
The spy agency's reliance on facial-recognition technology has grown significantly in the past four years as the agency has turned to new software to exploit the flood of images included in emails, text messages, social media, videoconferences and other communications, the documents show.
Agency officials believe that technological advances could revolutionize the way the agency finds intelligence targets around the world, the documents show. The agency's ambitions for this highly sensitive ability and the scale of its effort have not previously been disclosed.
The agency intercepts "millions of images per day" -- including about 55,000 "facial recognition quality images" -- which translate into "tremendous untapped potential," according to 2011 documents obtained from former agency contractor Edward Snowden. Although once focused on written and oral communications, the National Security Agency now considers facial images, fingerprints and other identifiers just as important to its mission of tracking suspected terrorists and other intelligence targets, the documents show.
"It's not just the traditional communications we're after: It's taking a full-arsenal approach that digitally exploits the clues a target leaves behind in their regular activities on the net to compile biographic and biometric information" that can help "implement precision targeting," noted a 2010 document.
One agency PowerPoint presentation from 2011, for example, displays several photographs of an unidentified man -- sometimes bearded, other times cleanshaven -- in different settings, along with more than two dozen data points about him. These include whether he was on the Transportation Security Administration's no-fly list, his passport and visa status, known associates or suspected terrorist ties, and comments made about him by informants to U.S. intelligence agencies.
It is not clear how many people around the world, and how many Americans, might have been caught up in the effort. Neither federal privacy laws nor the nation's surveillance laws provide specific protections for facial images. Given the agency's foreign-intelligence mission, much of the imagery would involve people overseas whose data were scooped up through cable taps, Internet hubs and satellite transmissions.
Because the agency considers images a form of communications content, it would be required to get court approval for imagery of Americans collected through its surveillance programs, just as it must to read emails or eavesdrop on phone conversations, an agency spokesman said. Cross-border communications in which an American might be emailing or texting an image to someone targeted by the agency overseas could be excepted.
Civil-liberties advocates and other critics are concerned that the power of the improving technology, used by government and industry, could erode privacy. "Facial recognition can be very invasive," said Alessandro Acquisti, a researcher on facial-recognition technology at Carnegie Mellon University. "There are still technical limitations on it, but the computational power keeps growing, and the databases keep growing, and the algorithms keep improving."
State and local law enforcement agencies rely on a wide range of databases of facial imagery, including driver's licenses and Facebook, to identify suspects. The FBI is developing what it calls its "next-generation identification" project to combine its automated fingerprint-identification system with facial imagery and other biometric data.
The State Department has what several outside experts say could be the largest facial-imagery database in the federal government, storing hundreds of millions of photographs of American passport holders and foreign visa applicants. And the Department of Homeland Security is funding pilot projects at police departments around the country to match suspects against faces in a crowd.
The National Security Agency, though, is unique in its ability to match images with huge troves of private communications.
"We would not be doing our job if we didn't seek ways to continuously improve the precision of signals-intelligence activities -- aiming to counteract the efforts of valid foreign intelligence targets to disguise themselves or conceal plans to harm the United States and its allies," said Vanee Vines, the agency spokesman.
She added that the agency did not have access to photographs in state databases of driver's licenses or to passport photos of Americans, while declining to say whether the agency had access to the State Department database of photos of foreign visa applicants. She also declined to say whether the spy agency collected facial imagery of Americans from Facebook and other social media through means other than communications intercepts.
"The government and the private sector are both investing billions of dollars into face-recognition" research and development, said Jennifer Lynch, a lawyer and expert on facial recognition and privacy at the Electronic Frontier Foundation in San Francisco. "The government leads the way in developing huge face-recognition databases, while the private sector leads in accurately identifying people under challenging conditions."
Lynch said a handful of recent court decisions could lead to new constitutional protections for the privacy of sensitive facial recognition data. But she said the law was still unclear and that Washington was operating largely in a legal vacuum. Laura Donohue, director of the Center on National Security and the Law at Georgetown Law School, agreed. "There are very few limits on this," she said.
Facial-recognition technology can still be a clumsy tool. It has difficulty matching low-resolution images, and photographs of faces taken from the side or angles can be impossible to match against mug shots or other head-on photographs.
Dalila Megherbi, an expert on facial-recognition technology at the University of Massachusetts at Lowell, explained that "when pictures come in different angles, different resolutions, that all affects the facial-recognition algorithms in the software."
That can lead to errors, the documents show. A 2011 PowerPoint showed one example when Tundra Freeze, the spy agency's main in-house facial-recognition program, was asked to identify photos matching the image of a bearded young man with dark hair. The document says the program returned 42 results, and displays several that were obviously false hits, including one of a middle-age man.
But the technology is powerful. One 2011 PowerPoint showed how the software matched a bald young man, shown posing with another man in front of a water park, with another photo where he has a full head of hair, wears different clothes and is at a different location.
It is not clear how many images the agency has acquired. The agency does not collect facial imagery through its bulk metadata collection programs, including that involving Americans' domestic phone records, authorized under Section 215 of the USAPATRIOT Act, Vines said.
The agency has accelerated its use of facial-recognition technology under President Barack Obama's administration, the documents show, intensifying its efforts after two intended attacks on Americans. The first was the case of the so-called underwear bomber, in which Umar Farouk Abdulmutallab, a Nigerian, tried to trigger a bomb hidden in his underwear in 2009 while flying to Detroit on Christmas Day. Just a few months later, in May 2010, Faisal Shahzad, a Pakistani-American, attempted a car bombing in Times Square.
One of the agency's broadest efforts to obtain facial images is a program called Wellspring, which strips out images from emails and other communications and displays those that might contain passport images. In addition to in-house programs, the spy agency relies in part on commercially available facial-recognition technology, including from PittPatt, a small company owned by Google, the documents show.
The agency can now compare spy satellite photographs with intercepted personal photographs taken outdoors to determine the location. One document shows what appear to be vacation photographs of several men standing near a small waterfront dock in 2011. It matches their surroundings to a spy satellite image of the same dock taken about the same time, located at what the document describes as a militant training facility in Pakistan.
A Section on 06/01/2014