Date of Incident: 04/03/2020
Time/Date Incident Started: 04/03/2020, 04:06 am EDT
Time/Date Stability Restored: 04/03/2020, 03:16 pm EDT
Time/Date Incident Resolved: 04/03/2020, 04:37 pm EDT
Users Impacted: All Subscribers accessing Dashboard
Some Work Order Notes added through the Dashboard displaying 4 hours into the future.
Root Cause Analysis:
As a part of an overall program to roll out application delivery improvements, the ServiceChannel Devops team made a change to the way the Dashboard module is served. These changes were infrastructural and not related to application logic.
Although this change was tested in a non-production setting, since the changes were not related to changes in application logic, the exit criteria for the test did not include a confirmation that timestamps were recorded against the appropriate timezone. Because the timezone configuration variable for the change was set to NULL, the data was rendered four hours later than the actual time of the event. This was confirmed in non-production test environments.
While infrastructural changes are rigorously tested before going to production, in this case, because the issue discovered was related to how the data was displayed and not a break in application logic, the test suite used by the Devops team did not show an application error. Consequently, the incorrect configuration was released to production.
A review of Work Orders edited within the impacted timeframe found that 3,178 Work Order Notes were impacted and would require their timestamps to have the appropriate timezone applied retroactively.