Date of Incident: 02/25/2019
Time/Date Incident Started: 02/25/2019, 12:05 pm EST
Time/Date Stability Restored: 02/25/2019, 12:21 pm EST
Time/Date Incident Resolved: 02/25/2019, 12:21 pm EST
Users Impacted: Active users
System performance throughout all systems was impacted by a increase in system response times and intermittent inability to use modules .
Root Cause Analysis:
Investigation via our monitoring systems revealed a larger than normal number of report requests.
Additional research revealed that these requests/reports had suboptimal execution plans and caused transient blocks.
We forced recompilation of the reports execution plans.
We will be doing in-depth review of underlying codes to adjust, optimize and stabilize performance.
The system was closely monitored for an additional next several days to ensure stability.
We are doing in-depth review of underlying codes to adjust, optimize and stabilize performance.