Single Sign On (SAML) Incident Report

Date of Incident:                   07/06/2020

Time/Date Incident Started: 07/04/2020, 2:09 pm EDT                                                                                                                             

Time/Date Stability Restored:   07/06/2020, 6:47 pm EDT

Time/Date Incident Resolved: 07/06/2020, 7:23 pm EDT

Users Impacted: Some users

Frequency: Intermittent

Impact: Minor

‌

Incident description:

The Support team received multiple reports from clients who could not log in to ServiceChannel via SSO.

The issue was identified as related to the expiration of a wildcard SSL certificate used by some ServiceChannel SSO integrations. As a result, SSO logins from the impacted clients were denied by the ServiceChannel ACS.

Logs traces reflected following exceptions:

"Module":"SAML","Message":"Decryption failed for issuer

"Module":"SAML","Message":"Failed to decrypt assertion:

"Exception":"System.Exception: Failed to decrypt XML. ---> System.Security.Cryptography.CryptographicException: Error occurred while decoding OAEP padding.

Some clients' IdPs demonstrated this behavior when recognizing the outdated certificate on ServiceChannel side, denying initiation of user logins to ServiceChannel from the IdP end. In these cases, end users experienced the error on the IdP side, so assertions were not sent to the ServiceChannel ACS.

Root Cause Analysis:

This issue was caused by an expired wildcard SSL certificate update used by a small number of SAML SSO client integrations.

Actions Taken: 

1. Updated the Production SAML module with a new certificate (via an emergency release) and provided clients who were affected by the SSO outage with the new public key.
2. Updated all configs in the Production environment with the new certificate name 'star.servicechannel.com.07312021.pfx' inserted in the field [ServiceCertificates].
3. Confirmed that clients were able to log in successfully.
    

Mitigation Measures:  

1. Generate unique certificates for each client who elect to use encryption for SAML assertions. This will prevent an impact to multiple clients in the event of a certificate issue.
2. Implement automated notifications to send warnings about certificate expiration to ServiceChannel clients and to the ServiceChannel SSO support engineering team.
3. Update the SAML SSO user documentation to deprecate the public key certificate section.