Alerts

Default alerts on every service, and email notifications when they fire.

Every deployed service gets a set of default alerts. They are evaluated every minute from the same telemetry you can query with specific query, they drive the project health indicator, and on paid plans they email you when they fire. You’ll find them under Alerts in the dashboard; nothing needs to be configured.

The default alerts

“Public services” are the ones reachable through the platform ingress (the ones with a domain); only those have request data. Preview environments get the same alerts and the same health indicator, but are never emailed.

There is deliberately no alert on error-level log lines: log severity is your application’s own semantics, and apps legitimately log errors that are not problems.

How alerts are evaluated

Each alert is one SQL query over the observability views that returns a single number, compared against a threshold. It runs every minute (every five minutes for slow responses) with the same access as specific query, so it only ever sees your own environment’s data.

Open an alert in the dashboard to see the exact query. Paste it into specific query and you get the same value the alert saw:

specific query "$(pbpaste)"

Every evaluation is recorded, so an alert’s page graphs exactly the values that were checked, and you can read the same numbers yourself from observability.alert_evaluations:

SELECT Timestamp, Value, State
FROM observability.alert_evaluations
WHERE ServiceName = 'web' AND AlertName = 'http-p99-latency'
  AND Timestamp > now() - INTERVAL 6 HOUR
ORDER BY Timestamp

An alert is either ok or firing. It resolves as soon as one evaluation is back under the threshold.

How slow responses are judged

The slow-responses alert measures time to first byte at the ingress: how long the platform waited for your service to start responding. Streamed responses and slow client downloads don’t count against you, and requests that never reached your service (a client that gave up while sending its request, for instance) are ignored.

Rather than comparing to a fixed number, it compares each route to itself. A request is “slow” if it took at least 3× that route’s own p90 over the trailing 24 hours, and at least a second. The alert fires when more than 5% of a window’s requests were slow by that definition. A report page that always takes ten seconds is therefore not an alert; a page that is suddenly slower than it usually is, is. Routes are templated (/api/users/{id}, /assets/{file}.svg), so IDs and asset file names don’t fragment the history.

Health

The health indicator on a project and its environments is nothing more than the currently firing alerts: a firing critical alert shows critical, a firing warning shows warning, and everything else is healthy. Open Alerts to see which alert is behind a status.

Notifications

On Pro and above, a firing alert sends an email within a minute of firing, with what fired, where, the numbers behind it (for example the failing routes and the request counts of that window), and a link to the alert. Resolving is not emailed, and an alert that fires repeatedly is emailed at most once every 30 minutes.

Where alerts go is set up under Alert notifications in organization settings. A destination is a name and a list of email addresses; the organization’s members are suggested as you type, and everyone starts on the default destination.

Which destinations an alert uses is decided on the alert’s own page, from the Notifications button in its header. The default destination reaches every alert; a destination you add reaches no alert until you switch it on there, so adding an on-call address never pages for everything at once. Switching every destination off is how you silence an alert.

Investigating an alert

An alert’s page has what you need to decide whether it matters:

• The graph of the alert’s value over the last six hours, with the threshold marked.
• A per-route breakdown for the request-based alerts, so you can tell a spike that has passed from a route that is still slow or failing.
• History: every time the alert fired or resolved, with the facts captured at that moment.
• A prompt you can copy into your coding agent that carries the alert’s context and its query.

For the same investigation from the CLI, a few starting points:

-- What fired across this environment in the last 7 days
SELECT ServiceName, AlertName, count() AS fires
FROM observability.alert_evaluations
WHERE Outcome = 'value' AND State = 'firing' AND Timestamp > now() - INTERVAL 7 DAY
GROUP BY ServiceName, AlertName ORDER BY fires DESC

-- The slowest routes of a service in the last hour (time to first byte)
SELECT SpanAttributes['http.method'] AS method, SpanAttributes['http.route'] AS route,
  count() AS requests,
  round(quantile(0.9)(1000 * toFloat64OrZero(SpanAttributes['nginx.upstream_header_time']))) AS p90_ms
FROM observability.traces
WHERE Source = 'ingress' AND ServiceName = 'web' AND Timestamp > now() - INTERVAL 1 HOUR
GROUP BY method, route ORDER BY p90_ms DESC LIMIT 10

-- Why the health checks failed
SELECT Timestamp, ResourceAttributes['k8s.pod.name'] AS pod, Body
FROM observability.events
WHERE ServiceName = 'web' AND LogAttributes['k8s.event.reason'] = 'Unhealthy'
  AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC

See Observability for the full schema of these views.