System Performance Degradation
Incident Report for ServiceChannel
Postmortem

Degraded Performance Incident Report

Date of Incident: 03/11/2019

Time/Date Incident Started: 03/11/2019, 11:05 am EST

Time/Date Stability Restored: 03/11/2019, 11:53 am EST

Time/Date Incident Resolved: 03/11/2019, 11:53 am EST

Users Impacted: All users

Frequency: Intermittent

Impact: Minor

Incident description:

System performance throughout all systems was impacted by a sudden increase in system response times and inability to use modules.

Root Cause Analysis:

Investigation via our monitoring systems revealed a larger-than-normal number of authentication requests.

Additional research revealed that these requests caused escalating blocks and resulted in increased latency for the authentication process.

Actions Taken:

Automated authentication process responsible for logging these requests has been disabled to allow an increase in authentication request throughput.

The system will be closely monitored for an additional next several days to ensure stability.

Mitigation Measures:

An in-depth review of the underlying code is underway to adjust, optimize, and stabilize performance.

Posted 3 months ago. Mar 11, 2019 - 14:38 EDT

Resolved
All services are confirmed running as expected. We consider this incident to be resolved.
Posted 3 months ago. Mar 11, 2019 - 14:23 EDT
Monitoring
Our Engineers have identified and rectified the issue with service performance. We are continuing to monitor, but do see an improvement in performance and systems are back to normal.
Posted 3 months ago. Mar 11, 2019 - 12:06 EDT
Investigating
We are currently observing degraded system performance. Our engineering teams are actively working on this and we will provide updates as soon as possible.
Posted 3 months ago. Mar 11, 2019 - 11:29 EDT
This incident affected: Call Center, eInvoicing and Integrations, Service Automation, Store Dashboard, and Contractor Integrations.