
A hidden defect in Microsoft’s cloud gateway, dormant since July, was to blame for one of the largest outages in the company’s history last week. A preliminary Post-Incident Report (PIR) released by the company outlines how the latent bug, changes in traffic, and reduced server capacity combined to knock much of Microsoft 365 and Exchange infrastructure offline at the end of last week. Microsoft’s report lists the incident’s final status as “a technical issue not related to cyber or AI/agents.”
Microsoft’s preliminary report places the outage between November 12 at 16:20 UTC and November 13 at 16:10 UTC, just under 24 hours total, and additional monitoring remained in place for any affected systems through November 14.
The outage started on November 12 and quickly rippled through businesses and public offices that depend on Microsoft’s cloud and email exchange infrastructure. The outage is covered in Microsoft’s incident IDs EX1550543 and MO1550821. You need Microsoft 365 admin credentials to see the full report on the company’s status pages, but Office Watch has reproduced the preliminary report, and the University of Pennsylvania maintains an archive of Microsoft’s service-health updates.
A perfect storm
Microsoft attributes the outage to three factors. A change deployed to Microsoft’s cloud gateway starting in July created a bug where, under certain conditions, CPU and memory resources could become exhausted and unresponsive. Microsoft contends that the defect was never discovered in pre-release testing because the hardware used was less powerful than some systems already deployed in production, where higher CPU core counts on newer machines apparently contributed to the lockout.
The second factor was server load. Microsoft said that “Traffic shifts during standard network maintenance and upgrades exposed a latent bug.” The company keeps two independent gateway systems in place to access Microsoft 365 for resilience. If one system encounters trouble and users can’t get through the front door, the other is opened to keep traffic flowing. According to the report, about 15 hours prior to the outage, traffic from Teams was shifted from the alternate gateway to the primary, where Exchange Online access is maintained. The switchover was a standard fail-safe, but it left the gateway handling more traffic, with a different mix of requests than before. As requests continued to pile up, affected servers exhausted their capacity and refused new connection requests.
Leave a Reply