Date of Incident: 02/20/2019
Time/Date Incident Started: 02/20/2019, 12:01 am EST
Time/Date Stability Restored: 02/20/2019, 12:15 pm EST
Time/Date Incident Resolved: 02/20/2019, 12:15 pm EST
Users Impacted: Fixxbook users logging in via fixxbook.com
In preparation for improvements in provider experience, a new fixxbook.servicechannel.com URL was introduced for unification purposes. The URL behaved properly across all test environments but in production there was a problem in the login redirect that caused an error and users could not continue to Fixxbook.
Root Cause Analysis:
The redirect from old to new URL worked properly in non-production environments
Release during the night of 2/19 into 2/20 released this change to production
The redirect in production encountered a problem that was not anticipated in other environments and started failing on the login process.
The production specific issue was related to the CMS in use for the Fixxbook brand site. In the early steps, we attempted to correct the CMS configuration.
At 6 am the issue was escalated
Code changes for Fixxbook were being tested by 10 am
A fix that caused the original issue was deployed and Fixxbook login was resolved around 12:15 PM EST of the same day.
We have improved the testing procedure post deployment to production to ensure these issues are noted right after deployment to production.
While this is a one time complex change, we’re also adding CMS administrators to troubleshoot and fix these issues quickly.