SnapEngage Service Status


CATEGORIES

Home > Chat Portal connectivity issue

Categories:

Chat Portal connectivity outage
All systems are back to normal

 

Root cause analysis summary:

Our real-time communication service provider completed their root cause analysis of the issue which caused the Chat Portal connectivity outage on the late afternoon of June 23rd. Here is the short summary from Pusher.com: “We recently migrated our payment system and for some reasons your high message add-on wasn’t taken into account in the process. As a consequence your account didn’t have the correct limits in place. After further investigation, you were the only account affected. We are really sorry about this issue impacting your users.”.

The SnapEngage tech team identified the issue as a problem with Pusher within a couple of minutes of the outage starting. It took another 5 minutes to assess the situation and all pointed to the Pusher quota system behaving incorrectly. After contacting Pusher support, and waiting for their corrective actions, our dev team started working on an alternate solution to prepare in case of a longer than planned outage of the Pusher platform. We were working on reactivating the real-time communication solution used before switching to Pusher a few years ago. Pusher service was restore around 25 minutes before we would have been ready with the legacy communication system being back online.

We have since been working with our service provider to ensure this issue will not recur. We have reviewed with them their limit warning policies to ensure that even in a wrong limit situation, the system will warn our operation. Again, we are very sorry for the impact this outage had to our customers.

5:35 pm Mountain time The connectivity issue has been resolved. Your agents are able to connect and send/receive messages in the Chat Portal again. We’re very sorry for this outage. The tech team is gathering information to work with our real time communication service provider to post mortem this outage and put in place a solution to avoid this issue in the future.

5:33 pm Mountain time We believe we will have a solution to mitigate the connectivity outage around after 6:00 pm.

5:00 pm Mountain time We are still actively working on the connectivity outage situation. We are very sorry it is taking longer than anticipated. As the Chat Portal agents are not able to pause their chat status, we advise the administrators to disable the live chat in the Settings dashboard until this issue is resolved. See this screenshot:

 

How_to_disable_chat

We will post an update no later than 6:00 pm.

4:30 pm Mountain time We are  still working with our service provider to restore the real time communication services. We will post an update at 5:00pm

4:07 pm Mountain time The SnapEngage Chat Portal is no longer accepting agent connections. We are working with our service provider to restore the service as soon as possible. We will post an update at 4:30pm

 

Published June 23, 2015

←

→