System Performance Issues Incident Report

Date of Incident:                   11/13/2019

Time/Date Incident Started: 11/13/2019, 11:46 am EDT

Time/Date Stability Restored:   11/13/2019, 03:53 pm EDT

Time/Date Incident Resolved: 11/13/2019, 05:00 pm EDT

Users Impacted: some users

Frequency: intermittent

Impact: Minor

Incident description:

Production database master encountered higher than usual workloads which led to intermittent blocks and system slowness, particularly around invoicing activities.

‌

Root Cause Analysis:

Intermittent data sync process caused unusual database reads which led to suboptimal IO responses. This problem affected only the production database master (read replicas were unaffected).

‌

Actions Taken:

1. Offensive query was identified
2. Optimization was implemented by recompiling and updating statistics
3. Monitored and confirmed that master db requests cleared the backlog and executing within normal durations

Mitigation Measures:  

1. Perform more aggressive optimizations on a more frequent schedule
2. Assign engineering teams to review the traffic pattern responsible for the incident and optimize resource consumption