Intermittent Performance Issues Under Normal Production Load - Incident Report
Date of Incident: 06/06/2023 - 06/30/2023
Time/Date Incident Started: 06/06/2023, 06:37 am EDT
Time/Date Stability Restored: 06/28/2023, 10:00 pm EDT
Time/Date Incident Resolved: 06/30/2023, 12:30 pm EDT
Users Impacted: Many
Frequency: Intermittent
Impact: Major
Incident Description:
Support highlighted a system slowdown and degraded performance, impacting the Dashboard, Work Order operations, and Invoice reports. Despite our dedicated and consistent efforts to rectify these performance challenges, they persisted over several weeks before we successfully resolved the issues.
Root Cause Analysis:
A series of interrelated issues, typically manageable individually, collectively led to significant performance degradation during periods of increased production load. We initially struggled to identify the root causes due to their occurrence around the same time as unrelated infrastructure changes. The key issues included:
Redis Cache Timeouts:
We initially suspected the Redis cache timeouts were due to an upgrade from Redis v4 to Redis v6. However, after the timeouts persisted following a reversion to Redis v4, we discarded this theory. We traced the timeouts to a combination of connection thread exhaustion and misconfigured Redis connection timeout values. The application lacked a fallback mechanism for Redis object retrieval, causing failure instead of graceful data retrieval from the persistence layer.
Application SQL Timeouts:
Unpredictable application behavior stemmed from intermittent periods of SQL timeouts on application server nodes. The distribution of these errors across all server nodes indicated a non-application code issue. Our SRE and Application Engineering teams, working with our DBA team, traced the SQL timeout errors to long-running SQL queries on the database cluster.
Application Node Failures:
During this period, an unusually high number of application nodes failed, marked by increased response duration, maximum CPU utilization, and high memory usage. The SRE team discovered the issue stemmed from the routing algorithm, set to "LeastConnections". This algorithm led heavily loaded nodes to get locked into a high-load state, requiring manual intervention.
Excessive API Calls:
A Service Provider reported an unusually large number of Work Order schedule changes in a Work Order assigned to their organization. We traced these changes, which triggered a nuisance cycle of Work Order Notes and Notifications, to a faulty Subscriber-built integration.
Actions Taken:
During the investigation, our SRE and Application Engineering teams established a protocol for daily joint monitoring conferences. Key events are available in Appendix A. Key activities included:
Mitigation Measures: