December 2016 Availability: 100.00%
Service Uptime for the previous ten months:
Resolved: Delay in sending chat data to integrations
December 27th 10:42am MDT The integration delay has been resolved and all pending case data has been delivered to our various integrations. We will continue to monitor the situation and will post any follow-up details when they become available.
December 27th 9:03am MDT We are currently experiencing a delay sending transcripts to integrations. Chats are not impacted by this delay, just chat transcripts. Our dev team is working on fixing this issue and it will be resolved shortly.
Solved: Issues with Agent Status
We have had a small number of reports about agent statues showing the incorrect status. Example; an agent is available online but is not getting chats because in the admin dash they still show as paused. The dev team is working on a fix but here is a temporary solution for this.
Have your admin go into the admin dashboard, click on Settings > Agent Settings > Chat Agents and then choose the agent with the issue and click Edit on the right hand side. In here you will be able to switch the agent into available or pause. This will correct the issue moving forward.
Update: The issue originally occurred between Dec 8-9th, which was quickly fixed. We did not anticipate however that some agents not having logged in after this issue would carry over incorrect statuses. We performed some small updates, but there were still some invalid statuses until the point we setup a monitoring solution to fix and correct the remaining few agent account status still stuck back in time.
We will continue monitoring, but we expect the issue to be fully addressed now.
Please check back to see when this issue is solved.
Resolved: Delay in sending chat data to integrations
October 3rd 5:42pm MDT The integration delay has been resolved and all pending case data has been delivered to our various integrations. We will continue to monitor the situation and will post any follow-up details when they become available.
October 3rd 4:12pm MDT We are currently experiencing delays in sending data to our integrations. We will post updates as the situation improves.
Resolved: Delay in sending emails
September 6th 3:52pm MDT SendGrid has sent the remaining backlog of emails.
September 6th 3:30pm MDT Our email provider, SendGrid, is currently experiencing latency issues which is impacting the delivery of our emails. We have been in touch with our provider and they are working on resolving the issue. We will continue to post updates as we receive more information from them.
RESOLVED: Interruption in recording Website Address URLs in visitor chats
August 10 9:14am MDT We experienced an interruption in recording the Website Address URL (page from which chats were started) in visitor chats. The issue occurred from 5:40am to 8:51am MDT. We have deployed a fix and apologize for any inconvenience.
RESOLVED: Integration with HubSpot failing to create timeline events
July 26 12:14pm MDT The issue came back for some clients in the past couple of hours. HubSpot’s technical team reworked some permissions. They have resolved it again now and are ensuring that the problem does not come back. We have just finished the delivery of the impacted chats and offline messages.
July 26 8:39am MDT All chats and offline messages that were not delivered during the earlier HubSpot outage have been sent and delivered.
July 26 7:28am MDT HubSpot has addressed the issue on their end. The integration is properly sending cases from new chats, and we are in the process of sending a very large batch of unsent chats and offline messages from the past 10 hours. We will post an update here once all chats and offline messages have been sent to HubSpot.
July 26 5:30am MDT HubSpot acknowledged the issue and is working on a resolution. Our team is ready to resubmit all chats and offline messages once HubSpot has resolved the API issue.
July 26 3:30am MDT We have notified HubSpot, but are still awaiting a resolution time frame from their side. Our technical team is evaluating options on our end, possibly reverting our clients from the timeline event mode to the old form submission mode. We are working to ensure that no data is missing when the integration is restored. We will post an update in a couple of hours or as soon as we have more information.
July 26 1:00am MDT A breaking change in the HubSpot API is preventing SnapEngage clients from creating timeline events in their HubSpot account. The issue seems to have started around 5pm yesterday and was first assessed to be a HubSpot transient problem. We are escalating the issue with HubSpot now.
RESOLVED: Integrations with CRMs, Help Desks, and other 3rd parties experiencing delays
July 13 7:40am Mountain time The backlog of all chat transcripts and offline messages from the early hours of the incident have been delivered. If you are unable to locate a transcript in your Help Desk or CRM, please come chat with us so that we can assist you locating it.
July 13 3:59am Mountain time Google confirmed they have mitigated the issue at 3:50am (9 minutes ago). We are seeing integration communications being back to normal. All chat transcripts and offline messages of the past hours are being delivered now. Chats and offline messages started prior to 2 hours ago and not yet in your CRM or Help Desk system might take a few additional hours to get delivered. Will we post an update once we can confirm that the backlog of all chat transcripts and offline messages have been delivered.
July 13 3:56am Mountain time Google communicated with us that their networking engineers are working on redirecting the Fetch traffic (used by 3rd party integrations) to a stable environment. They have not provided an ETA yet, but we anticipate a quick resolution now.
July 13 3:35am Mountain time Our hosting provider posted an update on the ongoing platform issue, saying they are still working on the issue, and will post another update at 03:30 US/Pacific.
July 13 2:50am Mountain time Our hosting provider published an update on the ongoing platform issue:
We are currently investigating an intermittent issue with Google App Engine URLFetch API service. Fetch requests to non-Google related services are failing with deadline exceeded errors.
For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 02:30 US/Pacific with current details.
We will post an update as soon as one is available from Google Cloud Platform.
July 13 2:38am Mountain time Our hosting provider, Google Cloud Platform, has localized the issue causing an elevated error rate with the infrastructure we rely on for our integrations. They are working on a fix. We will post an update as soon as one is available from them, or in an hour at the latest.
July 13 1:35am Mountain time Integrations with 3rd party providers are currently experiencing delays (social discovery, weather, help desks, CRMs, knowledge bases). We are investigating the issue with our hosting provider. We will post an update within an hour.
RESOLVED: Websites content no longer available (no impact on live chat)
May 3 4:14am Mountain time The issue has been resolved. All websites are back online.
May 3 3:26am Mountain time Our website content hosting provider posted a updated on their service disruption: https://wpenginestatus.com/read-only-on-some-servers-in-one-us-datacenter/ As the issue is not yet resolved and there is no ETA available from our provider, we are redirecting all traffic from our marketing website to the SnapEngage application signin page until the issue is resolved.
May 3 3:05am Mountain time Our website content hosting provider is currently experiencing an outage causing our websites for product information, client self support, developer documentation and service status to be unavailable. There is no impact on the SnapEngage live chat service though. We are in communication with our provider to get the issue resolved.
RESOLVED: We are currently experiencing issues running queries to generate reports in the SnapEngage Analytics
Feb. 29 3:49pm Mountain time The issue has been resolved. As always we will continue to monitor things and will post any follow-up information as it becomes available.
Feb. 29 12:34pm Mountain time The issue with the analytics is subsiding but we’re continuing to monitor things. We will post another update as soon as we are certain the issue has been completely resolved.
Feb. 29 8:28am Mountain time We are currently experiencing issues running queries to generate reports in the SnapEngage Analytics. We are in working with our service provider on a quick resolution.