Microsoft blames major outage on a July update defect – and two other factors

Microsoft has attributed last week's widespread outage of Microsoft 365 services, including Outlook, Teams, and Copilot, to three converging factors, according to a statement posted early Monday by Charles Lamanna, the company's executive vice president of Copilot, Agents, and Platform. The disruption began Thursday morning and affected access in 74 countries, as tracked by Downdetector. Microsoft denies that a cyberattack was involved, a claim Lamanna and the company's status account both made after some observers raised the possibility.
The root cause was what Lamanna described as a latent defect introduced in a July update to Microsoft 365 networking gateway servers. That update had run worldwide for more than three months without reported problems. Under certain load conditions, lock contention inside an internal process caused machines to become overwhelmed and unresponsive. Lock contention is a well-known concurrency problem: when multiple threads compete for the same resource and the coordination around it stalls, throughput can collapse even though the underlying hardware is healthy. In practice, that means a bug can sit dormant through months of normal traffic and only surface when load patterns shift.
Microsoft said its testing did not catch the flaw because test labs use a different traffic and SKU mix, and load testing concentrated on weaker hardware with fewer cores. That detail is significant for developers and platform teams: the defect appears to have hit higher-capacity configurations that normal testing under-exercised.
Two additional conditions turned the defect into a major outage, and servers were reportedly unusable in under two hours. First, a surge in load made it more likely that gateways would hit the contention. Second, Microsoft 365 runs two gateway systems; before the outage, Teams failed over from one to the other, changing its traffic patterns. At the same time, nearly 14% of networking gateway server capacity was out of service because of an unrelated issue from the previous day. Those conditions together triggered the July bug, requests piled up, and servers stopped responding. At the low point, more than 80% of the servers were unresponsive.
The failure then compounded itself. Microsoft's traffic management normally reroutes users to healthy endpoints when servers lose connectivity, but the redirected load raised the chance of the same problem on the remaining servers. With the same volume of traffic spread across fewer machines, the failure cascaded from server to server. Recovery was slow for related reasons: waiting traffic hit servers as soon as they returned and rendered them unresponsive again. Lamanna said the company's normally automated recovery processes were not built to restore service from such a broad unresponsive state, and because the code defect dated back months, standard rollback, flighting, and feature flags could not quickly revert the change. Microsoft eventually deployed a fix and rolled it out to groups of servers. A full post-incident review was promised for later in the week.
Not every service was hit equally. Teams was barely offline for many users, or was among the first to recover. Lamanna explained that services including Teams, OneDrive, SharePoint, and Office apps could fail out of the faulty gateway infrastructure, which is why they came back earlier, with Teams chats and meetings working for most of the outage. Exchange Online, by contrast, cannot fail out in the same way, which helps explain why Outlook access lingered.
The outage's scale is hard to pin down precisely. Microsoft has not said how many people were affected, though its own January figures put Microsoft 365 at more than 450 million commercial paid seats. Downdetector showed reported Outlook outages peaking near 83,000 around 1 p.m. ET Thursday, easing to just under 75,000 by 2:38 p.m. ET; Teams and Copilot reports dropped notably by Thursday evening, but users were still reporting problems into Friday morning, and Microsoft's own status page was down until about 3 p.m. ET Thursday. As of Friday afternoon, the company still had no estimate for full resolution. One think tank told CNN it estimated the outage, which for some ran through the close of business Friday, cost at least $7 billion in losses.
Larger context matters here. European organizations have been moving away from Microsoft in recent years toward locally controlled, open-source alternatives, and the former CEO of GitHub launched a decentralized alternative called Entire earlier this year, citing developer demand for reliability. This suggests the reputational cost of a multi-day productivity outage extends beyond the immediate incident, particularly for buyers weighing concentration risk in a single suite.
Why it matters: Teams responsible for Microsoft 365-dependent workflows should note that the failure mode was architectural as much as it was a code bug, so a latent defect plus a routine failover plus reduced capacity was enough to cascade. My inference is that organizations relying on a single productivity suite should review their own contingency plans for multi-day loss of email, chat, and document access, and that vendors will face harder questions about whether load testing reflects production hardware and traffic. The promised post-incident review should clarify what testing and rollback changes Microsoft intends to make.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
A surprise defect took more than 80% of Microsoft’s servers out at the outage’s peak. Here’s what happened.