Date of Incident: 05/14/2019

Time/Date Incident Started: 05/14/2019, 12:23 pm EST

Time/Date Stability Restored: 05/14/2019, 2:02 pm EST

Time/Date Incident Resolved: 05/14/2019, 4:03 pm EST

‌

Users Impacted: Active users

Frequency: Intermittent

Impact: Major

‌

Incident description:

Performance degradation throughout all systems resulting from a drastic increase in system response times due to database performance issues

Root Cause Analysis:

A recently enabled feature for WO reports resulted in higher-than-expected database server resource consumption. As requests queued, this caused an overall degradation of performance in the application.

Additional research determined that the code returning data for these requests was not optimized, causing excessive database blocking and waits.

Actions Taken:

Temporarily disabled certain customer-specific integrations to reduce excessive database load due to a long API request queue.

Identified and disabled a poorly-performing database stored procedure

Mitigation Measures:

Engineering teams investigated and refactored the code responsible for this feature.

Enabled throttling of these requests to prevent recurrence.