Supply Manager Outage Incident Report

Date of Incident: 02/22/2020

Time/Date Incident Started: 02/22/2020, 06:30 AM EDT

Time/Date Stability Restored: 02/22/2020, 11:23 AM EDT

Time/Date Incident Resolved: 02/22/2020, 11:51 AM EDT

‌

Users Impacted: Supply Manager enabled clients

Frequency: Limited

Impact: Minor

‌

Incident description:

Supply Manager was not accessible to subscribers, returning a 404 error.

‌

Root Cause Analysis:

During regularly scheduled maintenance, the DevOps team updated networking policies for our application gateways. These networking changes were performed to improve security and to extend networking functionality that is used by the ServiceChannel Platform. Unfortunately, these updates inadvertently affected routing for the Supply Manager component for the ServiceChannel platform.

‌

Actions Taken:

1. Reverted the networking changes to restore access
2. Performed restore of Supply Manager application components to return the system to its prior configuration

Mitigation Measures:

1. Develop additional test coverage for this specific component of the platform
2. Improved the process for restoring Supply Manager components to reduce the time to recovery.