Degraded Performance Incident Report
Date of Incident: 03/18/2019
Time/Date Incident Started: 03/18/2019, 5:01 pm EST
Time/Date Stability Restored: 03/18/2019, 5:01 pm EST
Time/Date Incident Resolved: 03/19/2019, 12:55 am EST
Users Impacted: Some users
The ServiceChannel Support team received a small number of reports of bounced email messages when they replying to WO notes, service requests, or sending emails to attachments. Users received the following error:
Error received: 5.3.0 – Other mail system problem – 550-‘5.6.0 Lone CR or LF in headers (see RFC2822 section 2.2)
Root Cause Analysis:
An internal investigation demonstrated that no impacting systems changes were made to the ServiceChannel platform, and none of our mail hosting partners reported any changes or issues.
Further research on the internet showed a widespread problem (as of March 18, 2019) related to a software update to Cisco Ironport ESA devices. A recent software update to Cisco Ironport ESA can cause the appliance to insert an additional colon (“:”) in custom “X-Ironport” headers, which cause local mail transfer agents to throw the RFC2822 error outlined above.
Anecdotal details for this issue can be referenced in the below Technet forum thread: https://social.technet.microsoft.com/Forums/en-US/9aa00049-c8fe-495b-8b49-a14bd3a39707/email-bounces-550560-lone-cr-or-lf-in-headers?forum=exchangesvrgeneral
ServiceChannel support verified that no changes to ServiceChannel email infrastructure or code had been implemented within the time frame of the incident.
The issue was escalated to Mail Hosting Partners, who determined that the issue was related to the client-side email configuration.
As this is a client-side issue, users experiencing this error are advised to contact their internal technical resources for resolution. Organizations using the Cisco Ironport ESA as a part of their email security strategy should contact the Cisco Technical Assistance Center for a workaround or patch.