Delay in sending chat data to integrations

Resolved: Delay in sending chat data to integrationsAll systems are back to normal

 

 

Friday February 17 2017 3:45 PM – The integration delay has been resolved and all pending case data has been delivered to our various integrations. We will continue to monitor the situation and will post any follow-up details when they become available.

 

Friday February 17 2017 2:03 PM – We are currently experiencing a delay sending transcripts to integrations. Chats are not impacted by this delay, just chat transcripts. Our dev team is working on fixing this issue and it will be resolved shortly.

 

Monthly Uptime Overview

January 2017 Availability: 100.00%

Service Uptime for the previous eleven months:

January 2017 100.00%
December 2016 100.00%
November 2016 100.00%
October 2016 100.00%
September 2016 99.95%
August 2016 100.00%
July 2016 100.00%
June 2016 100.00%
May 2016 100.00%
April 2016 100.00%
March 2016 100.00%

Delay in sending chat transcripts to integrations

Resolved: Delay in sending chat data to integrationsAll systems are back to normal

 

December 27th 10:42am MDT The integration delay has been resolved and all pending case data has been delivered to our various integrations. We will continue to monitor the situation and will post any follow-up details when they become available.

December 27th 9:03am MDT We are currently experiencing a delay sending transcripts to integrations. Chats are not impacted by this delay, just chat transcripts. Our dev team is working on fixing this issue and it will be resolved shortly.

 

Update SOLVED: Issues with Agent Status

Solved: Issues with Agent Status

                          Updated 12/13/2016All systems are back to normal

 

We have had a small number of reports about agent statues showing the incorrect status. Example; an agent is available online but is not getting chats because in the admin dash they still show as paused. The dev team is working on a fix but here is a temporary solution for this.

Have your admin go into the admin dashboard, click on Settings > Agent Settings > Chat Agents and then choose the agent with the issue and click Edit on the right hand side. In here you will be able to switch the agent into available or pause. This will correct the issue moving forward.

Update: The issue originally occurred between Dec 8-9th, which was quickly fixed. We did not anticipate however that some agents not having logged in after this issue would carry over incorrect statuses. We performed some small updates, but there were still some invalid statuses until the point we setup a monitoring solution to fix and correct the remaining few agent account status still stuck back in time.

We will continue monitoring, but we expect the issue to be fully addressed now.

 

as-1 as-2

Please check back to see when this issue is solved.

Integration Delays

Resolved: Delay in sending chat data to integrationsPartial service interruption

 

October 3rd 5:42pm MDT The integration delay has been resolved and all pending case data has been delivered to our various integrations. We will continue to monitor the situation and will post any follow-up details when they become available.

October 3rd 4:12pm MDT We are currently experiencing delays in sending data to our integrations. We will post updates as the situation improves.

 

Email Delivery Delay

Resolved: Delay in sending emailsPartial service interruption

 

September 6th 3:52pm MDT SendGrid has sent the remaining backlog of emails.

September 6th 3:30pm MDT Our email provider, SendGrid, is currently experiencing latency issues which is impacting the delivery of our emails. We have been in touch with our provider and they are working on resolving the issue. We will continue to post updates as we receive more information from them.

 

Interruption in recording Website Address URLs in visitor chats

RESOLVED: Interruption in recording Website Address URLs in visitor chats
All systems are back to normal

 

August 10 9:14am MDT We experienced an interruption in recording the Website Address URL (page from which chats were started) in visitor chats. The issue occurred from 5:40am to 8:51am MDT. We have deployed a fix and apologize for any inconvenience.

 

Integration with HubSpot failing to create timeline events

RESOLVED: Integration with HubSpot failing to create timeline events
All systems are back to normal

 

July 26 12:14pm MDT The issue came back for some clients in the past couple of hours. HubSpot’s technical team reworked some permissions. They have resolved it again now and are ensuring that the problem does not come back. We have just finished the delivery of the impacted chats and offline messages.

July 26 8:39am MDT All chats and offline messages that were not delivered during the earlier HubSpot outage have been sent and delivered.

July 26 7:28am MDT HubSpot has addressed the issue on their end. The integration is properly sending cases from new chats, and we are in the process of sending a very large batch of unsent chats and offline messages from the past 10 hours. We will post an update here once all chats and offline messages have been sent to HubSpot.

July 26 5:30am MDT HubSpot acknowledged the issue and is working on a resolution. Our team is ready to resubmit all chats and offline messages once HubSpot has resolved the API issue.

July 26 3:30am MDT We have notified HubSpot, but are still awaiting a resolution time frame from their side. Our technical team is evaluating options on our end, possibly reverting our clients from the timeline event mode to the old form submission mode. We are working to ensure that no data is missing when the integration is restored. We will post an update in a couple of hours or as soon as we have more information.

July 26 1:00am MDT A breaking change in the HubSpot API is preventing SnapEngage clients from creating timeline events in their HubSpot account. The issue seems to have started around 5pm yesterday and was first assessed to be a HubSpot transient problem. We are escalating the issue with HubSpot now.

Integrations with CRMs, Help Desks, and other 3rd parties experiencing delays

RESOLVED: Integrations with CRMs, Help Desks, and other 3rd parties experiencing delays
All systems are back to normal

 

July 13 7:40am Mountain time The backlog of all chat transcripts and offline messages from the early hours of the incident have been delivered. If you are unable to locate a transcript in your Help Desk or CRM, please come chat with us so that we can assist you locating it.

July 13 3:59am Mountain time Google confirmed they have mitigated the issue at 3:50am (9 minutes ago). We are seeing integration communications being back to normal. All chat transcripts and offline messages of the past hours are being delivered now. Chats and offline messages started prior to 2 hours ago and not yet in your CRM or Help Desk system might take a few additional hours to get delivered. Will we post an update once we can confirm that the backlog of all chat transcripts and offline messages have been delivered.

July 13 3:56am Mountain time Google communicated with us that their networking engineers are working on redirecting the Fetch traffic (used by 3rd party integrations) to a stable environment. They have not provided an ETA yet, but we anticipate a quick resolution now.

July 13 3:35am Mountain time Our hosting provider posted an update on the ongoing platform issue, saying they are still working on the issue, and will post another update at 03:30 US/Pacific.

July 13 2:50am Mountain time Our hosting provider published an update on the ongoing platform issue:

We are currently investigating an intermittent issue with Google App Engine URLFetch API service. Fetch requests to non-Google related services are failing with deadline exceeded errors.
For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 02:30 US/Pacific with current details.

We will post an update as soon as one is available from Google Cloud Platform.

July 13 2:38am Mountain time Our hosting provider, Google Cloud Platform, has localized the issue causing an elevated error rate with the infrastructure we rely on for our integrations. They are working on a fix. We will post an update as soon as one is available from them, or in an hour at the latest.

July 13 1:35am Mountain time Integrations with 3rd party providers are currently experiencing delays (social discovery, weather, help desks, CRMs, knowledge bases). We are investigating the issue with our hosting provider. We will post an update within an hour.

Website content issue

RESOLVED: Websites content no longer available (no impact on live chat)
All systems are back to normal

May 3 4:14am Mountain time The issue has been resolved. All websites are back online.

May 3 3:26am Mountain time Our website content  hosting provider posted a updated on their service disruption: https://wpenginestatus.com/read-only-on-some-servers-in-one-us-datacenter/ As the issue is not yet resolved and there is no ETA available from our provider, we are redirecting all traffic from our marketing website to the SnapEngage application signin page until the issue is resolved.

May 3 3:05am Mountain time Our website content hosting provider is currently experiencing an outage causing our websites for product information, client self support, developer documentation and service status to be unavailable. There is no impact on the SnapEngage live chat service though. We are in communication with our provider to get the issue resolved.