We are currently modernizing and improving our runner infrastructure. During the rollout of the new sandbox runner, we shipped a broken deployment.

Here is what happened:

1. We shipped a broken deployment, which resulted in false positives in a subset of checks.
2. Additionally, the rollback of the deployment failed, prolonging the incident.
3. The lack of proper internal communication about the system's state and rollback led to confusion and a prolonged response even more.

The incident affected browser checks in two specific cases:

1. When using Twilio in any runtime - due to TWILIO_ACCOUNT_SID environment variable being improperly escaped. Affected check runs failed with an error message: accountSid must start with AC
2. Checks using mocha in runtimes 2022.02 and 2022.10 with Playwright. These checks failed with an error Could not load reporter /checkly/functions/src/browser-check-common/src/mocha/reporter.js:

Impact:

• A total of 10 customers were affected.
• A total of 120 browser check runs (0.5%) failed due to this incident.
• A total of 66 different checks were affected.
• The broken deployment was shipped at 15:30 and was rolled back at 16:10 UTC

We apologize for any problems this may have caused. If you have any questions or concerns feel free to contact us via support@checklyhq.com