Summary
On March 4th, our email services experienced a critical incident caused by a cascade of unforeseen hardware failures. Multiple SSD (Solid-State Drive) disks failed in quick succession resulting in an application server becoming unresponsive, first impacting email services for some users and then spreading to other parts of the system.
Users who were already logged in to webmail experienced fewer disruptions, but new logins and IMAP connections were significantly impacted. Our support system also lost operability during the outage.
We have since replaced the failed hardware, rebuilt the affected systems, and fully restored services with additional redundancy. No user data was lost, and we have taking immediate and longer-term steps to prevent a recurrence in the future.
(more…)