Delayed webhook alert notifications

Started
Lasted 1h 40min
Degraded performance

Affected components

  1. Alerting

Updates

RESOLVED

lasted 1h 40min

Between October 31, 23:50 UTC and November 1, 11:51 UTC, webhook alert notifications were delayed by up to 6.4 hours. Notifications to endpoints that remained unreachable may not have been delivered. Check execution and other alert channels were not affected.

MONITORING

17min earlier

A fix has been implemented and we are monitoring the results.

IDENTIFIED

46min earlier

The backlog of delayed webhook notifications has been delivered and new webhook notifications are going out normally again. We have identified the cause and are monitoring delivery.

IDENTIFIED

1h 15min earlier

The issue has been identified and we are working on a fix.

INVESTIGATING

1h 39min earlier

Webhook alert notifications, including integrations delivered via webhook, have been delayed by up to several hours. Check execution and email, Slack and other alert channels are not affected.

INVESTIGATING

32s earlier

We are investigating the issue.