Announcer Downtime Root Cause Analysis

Date & Time: Oct 30, 2022, 7:26 PM IST(Sunday) - Oct 3, 2022, 6:30 PM IST(Monday)
Duration: close to 24 hours (overlapping weekend)

Root Cause:

The Announcer for Confluence cloud app was down for about 24 hours as the application server was rebooted and the startup script did not launch the webserver correctly.

Monitoring failure:

The monitoring system (site24x7) caught this outage, but it did not send out alert notifications. The outage was first discovered by the Dev team which subsequently reached out to the ITOps team. None of the customers reported the outage.

Impact:

The app experienced a complete outage during this time and displayed a Gateway timeout error on the UI. Admins were unable to create new notifications, and users were unable to view the notifications.

Resolution:

There's no significant impact on the app's state after it was restored. The app continued to work normally after restoring the webserver configuration.