It was late February, and Microsoft Corp. engineers had been working for weeks on a handful of alarming weaknesses in the company's popular Exchange email service. They were rushing to send out a fix, targeting the second Tuesday of March -- part of a monthly ritual known in cybersecurity circles as "patch Tuesday."
The hackers got a head start. Following weeks of discreet attacks, Chinese hackers shifted into high gear. The result was a sprawling campaign that engulfed thousands of organizations in a matter of days.
What is normally a relatively smooth process -- the one Microsoft uses regularly for identifying and fixing weaknesses in its popular software -- has morphed into a global cybersecurity crisis now getting the attention of the White House.
In all, researchers had identified four vulnerabilities and classified them as critical, meaning hackers can use them unseen to steal emails and other data.
But on Feb. 26, before the software giant released its patches, attackers began infiltrating those email systems en masse -- almost as though they knew their window of opportunity was about to close, said Ryan Kalember, executive vice president of cybersecurity strategy at the email security firm Proofpoint Inc.
Microsoft is now investigating the possibility of a leak that may have triggered these mass Exchange compromises ahead of its patch release, according to two sources with knowledge of the company's response to the attack. The sources, who weren't authorized to speak on the matter, said a leak, if indeed there was one, may have come from one of the company's security or government partners, or from independent researchers. A leak may have been malicious, or it could have been part of a separate security breach, they said.
A Microsoft spokesperson declined to comment on the investigation.
When Microsoft released its patches, a week ahead of schedule on March 2, it protected some clients, but also served as an accelerant for attacks, as more hackers piled on. In their race to break into networks before victims could lock their doors, the hackers breached banks and governments globally, as well as schools, hospitals, manufacturers and regional hotel chains.
The number of cyberespionage gangs attacking Exchange servers has now reached at least 10, cybersecurity firm Essential Security against Evolving Threats said in a recent blog post. There were at least 60,000 global victims of the hack by the end of last week, said a former U.S. official with knowledge of the investigation.
The number of attackers is probably dramatically higher now that the vulnerability has been widely distributed in criminal hacking circles, according to security researchers. "The president has been briefed and is tracking the issue closely," a spokesperson for the U.S. National Security Council said Wednesday in an email. "The White House is working around the clock with our public and private partners, keeping Congress updated, assessing the impact and defining the next steps we need to take."
ZERO-DAYS
Hackers are constantly looking for critical flaws in software, known as zero-days, because they can be used to steal data from users. The more widely used the software, the more valuable knowledge of a flaw.
Although many governments and large companies had already migrated to more modern systems, Microsoft Exchange is still in use by tens of thousands of customers around the world.
The company appears to have learned of the flaws in its Exchange email software between early January and early February. A Taiwan-based cyberresearch firm called Devcore first alerted Microsoft on Jan. 5, Devcore said.
A Virginia-based cybersecurity firm, Volexity, and a researcher known for finding such flaws -- who goes by the intentionally cryptic name Orange Tsai -- said they alerted the company to the zero-days between January and early February.
It often takes several weeks for Microsoft to create a safer version of popular software, and the company works to keep wider knowledge of any flaws secret during that time.
ATTACKS ESCALATE
About 10 days before Microsoft had planned to release fixes for its flawed email software, the number of Exchange customers being hacked suddenly jumped dramatically, according to several companies that tracked the activity.
Beginning Feb. 28, Essential Security observed five new cyberespionage groups using the Exchange zero-days. That was in addition to an advanced Chinese hacking group identified by Microsoft as Hafnium, which had been using the flaws for months.
Beijing on March 3 described Microsoft's allegation of Chinese culpability as a "groundless accusation" and called for evidence to support it.
While Essential Security hasn't done its own analysis of the groups' origins, various security researchers have published reports suggesting that the five additional groups also have connections to China -- for example, assessing that the hackers in the groups speak Chinese languages or operate from internet protocol addresses based in China.
VICTIMS COME FORWARD
The identity of most of the victims of China's attack are still unknown. The Norwegian Parliament announced it was hacked as part of the Microsoft Exchange campaign, and said significant data had been lost. The European Banking Authority said it was a victim as well, but has yet to find evidence that the hackers stole secrets.
Meanwhile, many Microsoft customers remain at risk
Proofpoint's Kalember, whose company specializes in email security, said the past couple of weeks have shown how serious the consequences from a breakdown in Microsoft's patching process can be. "There were many bad bugs that were supposed to still be a secret and internal to Microsoft," he said. "Clearly they weren't."