When the Los Angeles Department of Water and Power was hacked in 2018, it took a mere six hours. Early this year, an intruder lurked in hundreds of computers related to water systems across the U.S. In Portland, Ore., burglars installed malicious computers onto a grid providing power to a chunk of the Northwest.
Two of those cases -- Los Angeles and Portland -- were tests. The water threat was real, discovered by cybersecurity firm Dragos.
All three drive home a point long known but, until recently, little appreciated: The digital security of U.S. computer networks controlling the machines that produce and distribute water and power is woefully inadequate, a low priority for operators and regulators, posing a terrifying national threat.
"If we have a new world war tomorrow and have to worry about protecting infrastructure against a cyberattack from Russia or China, then no, I don't think we're where we'd like to be," said Andrea Carcano, co-founder of Nozomi Networks, a control system security company.
Hackers working for profit and espionage have long threatened American information systems. But in the past six months, they've targeted companies running operational networks like the Colonial Pipeline fuel system, with greater persistence. These are the systems where water can be contaminated, a gas line can spring a leak or a substation can explode.
The threat has been around for at least a decade -- and fears about it for a generation -- but cost and indifference posed obstacles to action.
It isn't entirely clear why ransomware hackers -- those who use malicious software to block access to a computer system until money has been paid -- have recently moved from small-scale universities, banks and local governments to energy companies, meatpacking plants and utilities. Experts suspect increased competition and bigger payouts as well as foreign government involvement. The shift is finally drawing serious attention to the problem.
The U.S. government began taking small steps to defend cybersecurity in 1998 when the Clinton administration identified 14 private sectors as critical infrastructure, including chemicals, defense, energy and financial services. This triggered regulation in finance and power. Other industries were slower to protect their computers, including the oil and gas sector, said Rob Lee, founder of Dragos.
One of the reasons is the operational and financial burden of pausing production and installing new tools.
Much of the infrastructure running technology systems is too old for sophisticated cybersecurity tools. Ripping and replacing hardware is costly, as are service blackouts. Network administrators fear doing the job piecemeal may be worse because it can increase a network's exposure to hackers, said Carcano.
Although the Biden administration's budget includes $20 billion to upgrade the country's grid, this takes place after a history of shoulder-shrugging from federal and local authorities. Even where companies in underregulated sectors such as oil and gas have prioritized cybersecurity, they've been met with little support.
Take the case of ONE Gas Inc. in Tulsa.
Niyo Little Thunder Pearson was overseeing cybersecurity there in January 2020 when his team was alerted to malware trying to enter its operational system -- the side that controls natural gas traffic across Oklahoma, Kansas and Texas.
For two days, his team was in a dogfight with the hackers, who moved laterally across the network. Ultimately, Pearson's team managed to expel the intruders.
When Richard Robinson at Cynalytica fed the corrupted files into his own identification program, ONE Gas learned it was dealing with malware capable of executing ransomware, exploiting industrial control systems and harvesting user credentials. At its core were digital footprints found in some of the most malicious code of the past decade.
Pearson tried to take the data to the FBI but it would accept it only on a compact disc, he said. His system couldn't burn the data onto a CD. When he alerted the Department of Homeland Security and sent it through a secure portal, he never heard back.
Robinson of Cynalytica was convinced a nation-state operator had just attacked a regional natural gas provider. So he gave a presentation to the Homeland Security Department, the Departments of Energy and Defense and intelligence agencies on a conference call. He never heard back either.
"We got zero, and that was what was really surprising," he said. "Not a single individual reached back out to find out more about what happened to ONE Gas."
The agencies didn't respond to requests for comment.
Such official indifference -- even hostility -- hasn't been uncommon.