Provider Enterprise Networking Issue Incident Report

Date of Incident: 02/22/2020

Time/Date Incident Started: 02/22/2020, 06:30 AM EDT

Time/Date Stability Restored: 02/22/2020, 09:37 AM EDT

Time/Date Incident Resolved: 02/22/2020, 10:28 AM EDT

‌

Users Impacted: Provider Enterprise enabled clients

Frequency: Limited

Impact: Minor

‌

Incident description:

Provider Enterprise XML connector was inaccessible over the network.

‌

Root Cause Analysis:

During regularly scheduled maintenance, the DevOps team upgraded networking policies for our application gateways. These networking changes were performed to improve security and extend networking functionality that is used by the ServiceChannel Platform. Unfortunately, these updates inadvertently impacted routing for the XML connector specifically leveraged by Provider Enterprise clients.

‌

Actions Taken:

1. Verified that the network issue impacted a limited number of clients specific to functionality associated with Provider Enterprise Clients.
2. Rolled back the networking changes to restore access.
3. Engaged members of the Engineering team to identify the code responsible for this component and implement a permanent fix.
4. Resubmitted outstanding Work Orders.

Mitigation Measures:

1. Develop additional test coverage for this specific component of the platform.
2. Employed code fixes to the XML Connector to prevent recurrence.