Integrations with CRMs, Help Desks, and other 3rd parties experiencing delays

RESOLVED: Integrations with CRMs, Help Desks, and other 3rd parties experiencing delays
All systems are back to normal

 

July 13 7:40am Mountain time The backlog of all chat transcripts and offline messages from the early hours of the incident have been delivered. If you are unable to locate a transcript in your Help Desk or CRM, please come chat with us so that we can assist you locating it.

July 13 3:59am Mountain time Google confirmed they have mitigated the issue at 3:50am (9 minutes ago). We are seeing integration communications being back to normal. All chat transcripts and offline messages of the past hours are being delivered now. Chats and offline messages started prior to 2 hours ago and not yet in your CRM or Help Desk system might take a few additional hours to get delivered. Will we post an update once we can confirm that the backlog of all chat transcripts and offline messages have been delivered.

July 13 3:56am Mountain time Google communicated with us that their networking engineers are working on redirecting the Fetch traffic (used by 3rd party integrations) to a stable environment. They have not provided an ETA yet, but we anticipate a quick resolution now.

July 13 3:35am Mountain time Our hosting provider posted an update on the ongoing platform issue, saying they are still working on the issue, and will post another update at 03:30 US/Pacific.

July 13 2:50am Mountain time Our hosting provider published an update on the ongoing platform issue:

We are currently investigating an intermittent issue with Google App Engine URLFetch API service. Fetch requests to non-Google related services are failing with deadline exceeded errors.
For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 02:30 US/Pacific with current details.

We will post an update as soon as one is available from Google Cloud Platform.

July 13 2:38am Mountain time Our hosting provider, Google Cloud Platform, has localized the issue causing an elevated error rate with the infrastructure we rely on for our integrations. They are working on a fix. We will post an update as soon as one is available from them, or in an hour at the latest.

July 13 1:35am Mountain time Integrations with 3rd party providers are currently experiencing delays (social discovery, weather, help desks, CRMs, knowledge bases). We are investigating the issue with our hosting provider. We will post an update within an hour.