Degraded Performance Incident Report
Date of Incident: 03/11/2019
Time/Date Incident Started: 03/11/2019, 11:05 am EST
Time/Date Stability Restored: 03/11/2019, 11:53 am EST
Time/Date Incident Resolved: 03/11/2019, 11:53 am EST
Users Impacted: All users
System performance throughout all systems was impacted by a sudden increase in system response times and inability to use modules.
Root Cause Analysis:
Investigation via our monitoring systems revealed a larger-than-normal number of authentication requests.
Additional research revealed that these requests caused escalating blocks and resulted in increased latency for the authentication process.
Automated authentication process responsible for logging these requests has been disabled to allow an increase in authentication request throughput.
The system will be closely monitored for an additional next several days to ensure stability.
An in-depth review of the underlying code is underway to adjust, optimize, and stabilize performance.