Email Processing for Inbound Requests
Incident Report for ServiceChannel
Postmortem

Degraded Performance Incident Report

Date of Incident: 03/18/2019

Time/Date Incident Started: 03/18/2019, 5:01 pm EST

Time/Date Stability Restored: 03/18/2019, 5:01 pm EST

Time/Date Incident Resolved: 03/19/2019, 12:55 am EST

Users Impacted: Some users

Frequency: Intermittent

Impact: Minor

Incident description:

The ServiceChannel Support team received a small number of reports of bounced email messages when they replying to WO notes, service requests, or sending emails to attachments. Users received the following error:

Error received: 5.3.0 – Other mail system problem – 550-‘5.6.0 Lone CR or LF in headers (see RFC2822 section 2.2)

Root Cause Analysis:

An internal investigation demonstrated that no impacting systems changes were made to the ServiceChannel platform, and none of our mail hosting partners reported any changes or issues.

Further research on the internet showed a widespread problem (as of March 18, 2019) related to a software update to Cisco Ironport ESA devices. A recent software update to Cisco Ironport ESA can cause the appliance to insert an additional colon (“:”) in custom “X-Ironport” headers, which cause local mail transfer agents to throw the RFC2822 error outlined above.

Anecdotal details for this issue can be referenced in the below Technet forum thread: https://social.technet.microsoft.com/Forums/en-US/9aa00049-c8fe-495b-8b49-a14bd3a39707/email-bounces-550560-lone-cr-or-lf-in-headers?forum=exchangesvrgeneral

Actions Taken:

ServiceChannel support verified that no changes to ServiceChannel email infrastructure or code had been implemented within the time frame of the incident.

The issue was escalated to Mail Hosting Partners, who determined that the issue was related to the client-side email configuration.

Mitigation Measures:

As this is a client-side issue, users experiencing this error are advised to contact their internal technical resources for resolution. Organizations using the Cisco Ironport ESA as a part of their email security strategy should contact the Cisco Technical Assistance Center for a workaround or patch.

Posted 3 months ago. Mar 20, 2019 - 10:05 EDT

Resolved
After consultation with our service providers and affected clients, we have been able to isolate this issue to a client-side email configuration issue.

In situations where the bounce notification email includes the error message "5.6.0 Lone CR or LF in headers (see RFC2822 section 2.2)," the issue is the result of a configuration change on the sender's email infrastructure and requires investigation by their own technical staff.

Though this is not an issue within ServiceChannel, in an effort to assist customers who may experience this issue, we will provide details of the cause of this issue in a status postmortem update. In the meantime, we consider this issue to be closed.
Posted 3 months ago. Mar 19, 2019 - 11:55 EDT
Investigating
We are currently experiencing an issue with email-triggered replies. Our mail provider has made some changes and some Provider Users may receive a bounceback message when emailing Attachments, WO Notes, or WO Dispatches. Our team is actively working to resolve the issue. In the meantime, you can still log in and make the updates in the event that you receive the bounceback email. None of our outgoing emails are affected at this time, it is only incoming into ServiceChannel. We will update as necessary.
Posted 3 months ago. Mar 18, 2019 - 16:01 EDT
This incident affected: Email Processing.