Supply Manager Outage
Incident Report for ServiceChannel

Supply Manager Outage Incident Report

Date of Incident: 02/22/2020

Time/Date Incident Started: 02/22/2020, 06:30 AM EDT

Time/Date Stability Restored: 02/22/2020, 11:23 AM EDT

Time/Date Incident Resolved: 02/22/2020, 11:51 AM EDT

Users Impacted: Supply Manager enabled clients

Frequency: Limited

Impact: Minor

Incident description:

Supply Manager was not accessible to subscribers, returning a 404 error.

Root Cause Analysis:

During regularly scheduled maintenance, the DevOps team updated networking policies for our application gateways. These networking changes were performed to improve security and to extend networking functionality that is used by the ServiceChannel Platform. Unfortunately, these updates inadvertently affected routing for the Supply Manager component for the ServiceChannel platform.

Actions Taken:

  1. Reverted the networking changes to restore access
  2. Performed restore of Supply Manager application components to return the system to its prior configuration

Mitigation Measures:

  1. Develop additional test coverage for this specific component of the platform
  2. Improved the process for restoring Supply Manager components to reduce the time to recovery.
Posted Feb 25, 2020 - 08:54 EST

The SRE Team confirms that Supply Manager is back to normal. We consider this incident to be resolved.
Posted Feb 22, 2020 - 11:51 EST
The SRE team has implemented the fix and services have returned to normal. We will continue to monitor.
Posted Feb 22, 2020 - 11:36 EST
The issue has been identified, and the SRE team is implementing a fix.
Posted Feb 22, 2020 - 11:23 EST
We are continuing to investigate this issue.
Posted Feb 22, 2020 - 07:46 EST
The Site Reliability Engineering team is currently investigating an outage of Supply Manager. We will provide an ETA as soon as possible.
Posted Feb 22, 2020 - 07:46 EST
This incident affected: Service Automation (Supply Manager).