Fastly, the company hit by a major failure that caused many of the world's top websites to go offline briefly this week, blamed the problem on a software bug that was triggered when a single customer changed a setting.
The problem at Fastly meant internet users couldn't connect to a host of popular websites early Tuesday including The New York Times, the Guardian, Twitch, Reddit and the British government's homepage.
"We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change," Nick Rockwell, Fastly's senior vice president of engineering and infrastructure, said in a blog post late Tuesday.
He said the outage was "broad and severe" but the company quickly identified, isolated and disabled the problem and after 49 minutes, most of its network was up and running again. The bug had been included in a software update that was rolled out in May and Rockwell said the company is trying to figure out why it wasn't detected during testing.
San Francisco-based Fastly provides what's called a content delivery network -- an arrangement that allows customer websites to store data such as images and videos on various mirror servers across 26 countries. Keeping the data closer to users means it loads faster.
Content delivery networks are particularly difficult to replicate because their business model requires having physical data centers spread across several countries. Fastly itself has more than 50. Larger cloud companies like Google or Amazon, which store the bulk of the internet, have fewer, but larger data warehouses.
The outage Tuesday morning underscored the importance of little-known internet infrastructure companies like Fastly to the normal functioning of the web, and how even isolated disruptions can bring huge parts of online life to a halt. The pandemic-era shifts that sent more people online for their groceries, work, school and health care have heightened the potential for broad shutdowns to do real-world harm.
"This outage was broad and severe, and we're truly sorry for the impact to our customers and everyone who relies on them," Rockwell said.
Information for this article was contributed by staff members of The Associated Press and by Taylor Telford of The Washington Post.