Incorrect Work Order Timezone
Incident Report for ServiceChannel
Postmortem

Date of Incident: 04/03/2020

Time/Date Incident Started: 04/03/2020, 04:06 am EDT

Time/Date Stability Restored: 04/03/2020, 03:16 pm EDT

Time/Date Incident Resolved: 04/03/2020, 04:37 pm EDT

Users Impacted: All Subscribers accessing Dashboard

Frequency: Sustained

Impact: Minor

Incident description:

Some Work Order Notes added through the Dashboard displaying 4 hours into the future.

Root Cause Analysis:

As a part of an overall program to roll out application delivery improvements, the ServiceChannel Devops team made a change to the way the Dashboard module is served. These changes were infrastructural and not related to application logic.

Although this change was tested in a non-production setting, since the changes were not related to changes in application logic, the exit criteria for the test did not include a confirmation that timestamps were recorded against the appropriate timezone. Because the timezone configuration variable for the change was set to NULL, the data was rendered four hours later than the actual time of the event. This was confirmed in non-production test environments.

While infrastructural changes are rigorously tested before going to production, in this case, because the issue discovered was related to how the data was displayed and not a break in application logic, the test suite used by the Devops team did not show an application error. Consequently, the incorrect configuration was released to production.

A review of Work Orders edited within the impacted timeframe found that 3,178 Work Order Notes were impacted and would require their timestamps to have the appropriate timezone applied retroactively.

Actions Taken:

  1. Received a notification from customers that Word Order Notes were recorded with an incorrect time.
  2. Confirmed that the issue was the result of the misconfiguration of a variable and not an underlying issue in application logic.
  3. Rolled back the infrastructural change to restore the correct timezone variable configuration.
  4. Found all Work Orders updated during the affected period and applied a retroactive timezone update without affecting business logic to all impacted Work Order Notes.

Mitigation Measures:

  1. Mandated more testing within the DevOps team for infrastructural configuration changes about how time is handled in lower environments.
  2. Added a timestamp test to the Production change smoke test.
Posted Apr 08, 2020 - 17:50 EDT

Resolved
Some Work Order Notes added through the Dashboard displaying 4 hours into the future.
Posted Apr 03, 2020 - 16:37 EDT