WASHINGTON -- U.S. government agencies and private companies rushed Monday to secure their computer networks after the disclosure of a sophisticated and long-running cyberespionage intrusion that experts said almost certainly was carried out by a foreign state.
It is not yet clear who was responsible for the intrusion, though experts suspect it was conducted by Russia, and the extent of the damage is not yet known. The threat was significant enough that the Department of Homeland Security's cybersecurity unit directed all federal agencies to remove compromised network management software, and thousands of companies were expected to do the same.
What was striking about the operation was its potential scope as well as the manner in which the perpetrators managed to pierce cyberdefenses and gain access to email and internal files at the Treasury and Commerce departments and potentially elsewhere. It provided stark evidence of the vulnerability of supposedly secure government networks, even after well-known attacks in the past.
"It's a reminder that offense is easier than defense, and we still have a lot of work to do," said Suzanne Spaulding, a former U.S. cybersecurity official who is now a senior adviser to the Center for Strategic and International Studies.
The campaign came to light when a prominent cybersecurity firm, FireEye, learned it had been breached and alerted that foreign governments and major corporations were also compromised. FireEye would not say who it suspected, though many experts quickly suspected Russia given the level of skill involved.
U.S. authorities acknowledged Sunday that federal agencies were part of the breach, providing few details. The Cybersecurity and Infrastructure Security Agency said in an unusual directive that the widely used network software SolarWinds had been compromised and should be removed from any system using it.
The cybersecurity agencies of Britain and Ireland issued similar alerts.
The U.K. National Cyber Security Center "is working closely with FireEye and international partners on this incident," said a spokesperson in an emailed statement. "Investigations are ongoing, and we are working extensively with partners and stakeholders to assess any U.K. impact."
SolarWinds is used by hundreds of thousands of organizations around the world, including most Fortune 500 companies and multiple U.S. federal agencies. The perpetrators were able to embed malware in a security update issued by the company, based in Austin, Texas. Once inside, they could impersonate system administrators and have total access to the infected networks, experts said.
"Quite honestly, my heart sank when I saw some of the details, just the amount of information they could potentially have if they are reading everyone's emails and they are accessing sensitive files within places like Treasury or Commerce," said Ben Johnson, a former National Security Agency cyberengineer who is now chief technology officer of software security firm Obsidian.
The Washington Post, citing unnamed sources, said the attack was carried out by Russian government hackers who go by the nicknames APT29 and Cozy Bear and are part of that nation's foreign intelligence service.
U.S. officials have declined to say who they believe is responsible. National Security Council spokesman John Ullyot on Monday said only that the Trump administration was working with the cybersecurity agency, U.S. intelligence agencies, the FBI and government departments that were affected to coordinate a response to whoever was behind it.
The FBI said it is "appropriately engaged," declining to comment further.
Microsoft cybersecurity researchers Monday tied the hacks to "nation-state activity at significant scale," aimed at both government and the private sector.
"It's obviously incredibly significant and widespread," said Chris Painter, who coordinated cyberpolicy at the State Department during the Obama administration. "How much was compromised? How much was exfiltrated? There are lots of open questions now."
Kremlin spokesman Dmitry Peskov said Monday that Russia had "nothing to do with" the attacks.
"Once again, I can reject these accusations," Peskov told reporters. "If for many months the Americans couldn't do anything about it, then, probably, one shouldn't unfoundedly blame the Russians for everything."
Federal agencies have long been attractive targets for foreign hackers looking to gain insight into American government personnel and policymaking.
Hackers linked to Russia, for instance, were able to break into the State Department's email system in 2014, infecting it so thoroughly that it had to be cut off from the internet while experts worked to eliminate the infestation. A year later, a hack at the U.S. government's personnel office blamed on China compromised the personal information of some 22 million current, former and prospective federal employees, including highly sensitive data such as background investigations.
Cybersecurity experts said the nature and level of tradecraft involved in this latest effort suggest a foreign nation. Many have pointed out that the goal of the monthslong effort appeared to be espionage and not information that could be quickly used for profit or to simply inflict damage. Russia was the most likely culprit, though China and perhaps others are potential candidates.
In terms of scale alone, the operation seems similar to the personnel office hack, said Ben Buchanan, a Georgetown University cyberespionage expert.
"The operational tradecraft -- how the hack was carried out -- seems to have been extremely good. These operators are experienced and capable, adept at finding a systemic weakness and then exploiting it quietly for months," said Buchanan, author of "The Hacker and The State."
If it was carried out by a foreign government and the U.S. has the proof, then it becomes a question of what to do about it.
Some obvious options would include expelling diplomats of the offending country, imposing sanctions or filing criminal charges for cyberespionage, steps that Washington and the European Union have taken against Russia in the past.
"I'm sure that the departments like NSA and Cyber Command are coming up with options, that the Treasury Department is looking at sanction options , that the State Department is looking for how they will send a strong signal," Spaulding said. "Whether they will get approval for all these things from the White House remains to be seen."
In the meantime, SolarWinds and its many private-sector clients were working to close any breaches and repair the damage.
The company said in a financial filing that it believed that an unknown number of customers, though fewer than 18,000, had installed the compromised product update earlier this year.
The company said it alerted relevant customers and provided mitigation steps, including a "hotfix" update. A second update is expected to be released today, the company said.
SolarWinds has said its customers include all five branches of the U.S. military, the Pentagon, the State Department, NASA, the National Security Agency, the Department of Justice and the White House, along with the top U.S. telecommunications and accounting firms, though it hasn't identified which of its customers were using the compromised product.
"We anticipate this will be a very large event when all the information comes to light," said John Hultquist, director of threat analysis at FireEye.
Information for this article was contributed by Ben Fox and Frank Bajak of The Associated Press and by Alyza Sebenius, Kartikay Mehrotra and Michael Riley of Bloomberg News.