Timeouts on scheduling checks

Incident Report for Checkly

Postmortem

After an investigation, we tracked the issue down to our upstream cloud provider. It was a side effect of a larger migration they were executing on their side.
Our check scheduling was affected by this degradation but all the other services were operational. Our internal retry mechanism handled most of the timeouts but a handful of customers saw failures in their checks.

Posted Jun 10, 2021 - 12:05 UTC

Resolved

We can see that the timeouts have disappeared. The degradation is no longer persistent. We are still investigating together with our providers what the root cause is of this issue.

Posted Jun 04, 2021 - 07:19 UTC

Identified

We are seeing intermittent timeouts — event after standard retries — of scheduling checks to some regions. This seems to be impacting less than 1% of our check volume. We are escalating this to our infra provider.

Posted Jun 03, 2021 - 11:42 UTC

This incident affected: API.