Gen. Paul Nakasone, the nation's top cyberwarrior, and other U.S. officials responsible for cybersecurity are now consumed by what they missed for at least nine months: a hacking, now believed to have affected upward of 250 federal agencies and businesses.
Three weeks after the intrusion came to light, U.S. officials are still trying to understand whether what the Russians pulled off was simply an espionage operation inside the systems of the American bureaucracy or something more sinister, inserting "backdoor" access into government agencies, major corporations, the electric grid and laboratories developing and transporting new generations of nuclear weapons.
At a minimum, the attack has set off alarms about the vulnerability of government and private-sector networks in the United States and raised questions about how and why the nation's cyberdefenses failed.
Those questions have taken on particular urgency given that the breach was not detected by any of the government agencies that share responsibility for cyberdefense -- the military's Cyber Command and the National Security Agency, both of which are run by Nakasone, and the Department of Homeland Security -- but by a private cybersecurity company, FireEye.
"This is looking much, much worse than I first feared," said Sen. Mark Warner, D-Va., the ranking member of the Senate Intelligence Committee. "The size of it keeps expanding. It's clear the United States government missed it."
"And if FireEye had not come forward," he added, "I'm not sure we would be fully aware of it to this day."
The intentions behind the attack remain shrouded. But with a new administration taking office in three weeks, some analysts say the Russians may be trying to shake Washington's confidence in the security of its communications and to demonstrate their cyberarsenal to gain leverage against President-elect Joe Biden before nuclear arms talks.
"We still don't know what Russia's strategic objectives were," said Suzanne Spaulding, who was the senior cyberofficial at the Homeland Security Department during the Obama administration. "But we should be concerned that part of this may go beyond reconnaissance. Their goal may be to put themselves in a position to have leverage over the new administration, like holding a gun to our head to deter us from acting to counter [Russian President Vladimir] Putin."
Interviews with key players investigating what intelligence agencies believe to be an operation by Russia's SVR intelligence service revealed these points:
• The breach is far broader than first believed. Initial estimates were that Russia sent its probes only into a few dozen of the 18,000 government and private networks it gained access to when it inserted code into network management software made by the Texas company SolarWinds. But as businesses like Amazon and Microsoft that provide cloud services dig deeper for evidence, it now appears Russia exploited multiple layers of the supply chain to gain access to as many as 250 networks.
• The hackers managed their intrusion from servers inside the United States, exploiting legal prohibitions on the National Security Agency from engaging in domestic surveillance and eluding cyberdefenses deployed by the Department of Homeland Security.
• "Early warning" sensors placed by Cyber Command and the National Security Agency deep inside foreign networks to detect brewing attacks failed. There is also no indication yet that any human intelligence alerted the United States to the hacking.
• The government's emphasis on election defense, while critical in 2020, may have diverted resources and attention from long-brewing problems such as protecting the supply chain of software. In the private sector, too, companies that were focused on election security, like FireEye and Microsoft, are now revealing that they were breached as part of the larger supply chain attack.
• SolarWinds, the company that the hackers used as a conduit for their attacks, had a history of lackluster security for its products, making it an easy target, according to current and former employees and government investigators. Its chief executive, Kevin Thompson, who is leaving his job after 11 years, has sidestepped the question of whether his company should have detected the intrusion.
• Some of the compromised SolarWinds software was engineered in Eastern Europe, and American investigators are now examining whether the incursion originated there, where Russian intelligence operatives are deeply rooted.
GROWING HIT LIST
The U.S. government was clearly the main focus of the attack, with the Treasury Department, the State Department, the Commerce Department, the Energy Department and parts of the Defense Department among the agencies confirmed to have been infiltrated. The Pentagon insists the attacks on its systems were unsuccessful, though it has offered no evidence.
But the hacking also breached large numbers of corporations, many of which have yet to step forward. SolarWinds is believed to be one of several supply chain vendors Russia used in the hacking. Microsoft, which had tallied 40 victims as of Dec. 17, initially said that it had not been breached, only to discover last week that it had been -- and that resellers of its software had been, too. A previously unreported assessment by Amazon's intelligence team found the number of victims may have been five times greater, though officials warn some of those may be double-counted.
Publicly, officials have said they do not believe the hackers from Russia's SVR pierced classified systems containing sensitive communications and plans. But privately, officials say they still do not have a clear picture of what might have been stolen.
They said they worried about delicate but unclassified data the hackers might have taken from victims such as the Federal Energy Regulatory Commission, including Black Start, the detailed technical blueprints for how the United States plans to restore power in the event of a cataclysmic blackout.
The plans would give Russia a hit list of systems to target to keep power from being restored in an attack like the one it pulled off in Ukraine in 2015, shutting off power for six hours in the dead of winter. Moscow long ago implanted malware in the U.S. electric grid, and the United States has done the same to Russia as a deterrent.
SUPPLY CHAIN WEAKNESS
One main focus of the investigation so far has been SolarWinds, the company based in Austin, Texas, whose software updates the hackers compromised.
But the cybersecurity arm of the Department of Homeland Security concluded the hackers worked through other channels, too. And last month, CrowdStrike, another security company, revealed that it was also targeted, unsuccessfully, by the same hackers, but through a company that resells Microsoft software.
Because resellers are often entrusted to set up clients' software, they -- like SolarWinds -- have broad access to Microsoft customers' networks. As a result, they can be an ideal Trojan horse for Russia's hackers. Intelligence officials have expressed anger that Microsoft did not detect the attack earlier; the company, which said Thursday that the hackers viewed its source code, has not disclosed which of its products were affected or for how long hackers were inside its network.
"They targeted the weakest points in the supply chain and through our most trusted relationships," said Glenn Chisholm, a founder of Obsidian Security.
Interviews with current and former employees of SolarWinds suggest it was slow to make security a priority, even as its software was adopted by America's premier cybersecurity company and federal agencies.
Employees say that under Thompson, an accountant by training and a former chief financial officer, every part of the business was examined for cost savings, and common security practices were eschewed because of their expense. His approach helped almost triple SolarWinds' annual profit margins to more than $453 million in 2019 from $152 million in 2010.
But some of those measures may have put the company and its customers at greater risk for attack. SolarWinds moved much of its engineering to satellite offices in the Czech Republic, Poland and Belarus, where engineers had broad access to the Orion network management software that Russia's agents compromised.
The company has said only that the manipulation of its software was the work of human hackers rather than of a computer program. It has not publicly addressed the possibility of an insider being involved in the breach.
SolarWinds declined to address questions about the adequacy of its security. In a statement, it said it was a "victim of a highly sophisticated, complex and targeted cyberattack" and was collaborating closely with law enforcement authorities, intelligence agencies and security experts to investigate.
But security experts note that it took days after the Russian attack was discovered before SolarWinds' websites stopped offering clients compromised code.