System Performance
Incident Report for ServiceChannel
Postmortem

System Performance Degradation Incident Report

Date of Incident: 05/14/2019

Time/Date Incident Started: 05/14/2019, 12:23 pm EST

Time/Date Stability Restored: 05/14/2019, 2:02 pm EST

Time/Date Incident Resolved: 05/14/2019, 4:03 pm EST

Users Impacted: Active users

Frequency: Intermittent

Impact: Major

Incident description:

Performance degradation throughout all systems resulting from a drastic increase in system response times due to database performance issues

Root Cause Analysis:

A recently enabled feature for WO reports resulted in higher-than-expected database server resource consumption. As requests queued, this caused an overall degradation of performance in the application.

Additional research determined that the code returning data for these requests was not optimized, causing excessive database blocking and waits.

Actions Taken:

Temporarily disabled certain customer-specific integrations to reduce excessive database load due to a long API request queue.

Identified and disabled a poorly-performing database stored procedure

Mitigation Measures:

Engineering teams investigated and refactored the code responsible for this feature.

Enabled throttling of these requests to prevent recurrence.

Posted 4 months ago. Jun 12, 2019 - 12:43 EDT

Resolved
Workforce reporting has been re-enabled. After 90 minutes of monitoring, all systems are functioning normally. We consider this incident to be resolved.
Posted 5 months ago. May 14, 2019 - 16:03 EDT
Monitoring
Our engineering team has identified the issue and services are returning to normal with the exception of some Workforce reports. Work order reports are currently not displaying the technician's information associated with the work order. Tech assigned and Tech accepted fields are not available. We are continuing to monitor and will restore the reports shortly. Thank you for your patience.
Posted 5 months ago. May 14, 2019 - 14:02 EDT
Investigating
We are currently investigating degraded system performance. We will provide an update shortly. Thank you for your patience.
Posted 5 months ago. May 14, 2019 - 12:24 EDT
This incident affected: Fixxbook, Service Automation, and Store Dashboard.