Observability

Query your environments' logs and metrics with SQL.

Every deployed environment streams logs and metrics into a queryable store. Use specific query to run SQL against this data when debugging production incidents, investigating regressions, or doing ad-hoc analytics.

Running a query

# Inline
specific query "SELECT count() FROM observability.logs"

# Against a specific environment
specific query --environment staging "SELECT * FROM observability.logs LIMIT 5"

# From a file via stdin
cat queries/p99.sql | specific query

# Machine-readable output for agents and scripts
specific query --format jsonl "SELECT Timestamp, Body FROM observability.logs LIMIT 5"

Flags:

Queries are automatically scoped to the selected environment: you can’t see data from other environments and don’t need to filter on environment yourself.

Queries are read-only: INSERT, UPDATE, DELETE, and DDL are rejected. Each query is capped at 30 seconds of execution time, returns at most 100,000 rows, and the SQL string is limited to 50,000 characters.

The table format truncates wide values for readability. Agents and scripts should use jsonl, json, or csv for complete, untruncated values. Notices are printed to stderr so stdout remains parseable.

Choosing an environment

specific query runs against exactly one environment. Run specific status to discover every environment and its ID. The output has two sections:

• Environments - long-lived environments such as production and staging.
• Preview environments - ephemeral environments created for pull requests (plus any manually created previews), each with its name, env_... ID, PR number and title, deployed URL, and expiry.

Preview environments stream logs and metrics exactly like long-lived ones. To inspect one, pass its name or ID to --environment; that’s the only way to reach a preview’s data, since queries are auto-scoped:

specific query --environment preview-a1b2c3d4 \
  "SELECT Timestamp, ServiceName, Body
   FROM observability.logs
   WHERE SeverityNumber >= 17
   ORDER BY Timestamp DESC
   LIMIT 50"

Schema

Four views are exposed in the observability database: logs, metrics, traces and events. Column names follow OpenTelemetry’s ClickHouse exporter conventions and are case-sensitive, so use SeverityText, ServiceName, and ResourceAttributes exactly as shown. If unsure, run DESCRIBE observability.logs.

observability.logs

Unified log stream from services. Retention: up to 30 days (your plan may restrict how far back queries can read; a cutoff prints a note on stderr).

Also exposed but rarely queried directly: EnvironmentId, ProjectId (used for auto-scoping), TraceFlags.

observability.metrics

Unified metrics from services. Retention: up to 90 days.

Service metrics

Emitted for every running service container:

observability.traces

OpenTelemetry spans. Today every row is the edge span the platform’s ingress records for a request to one of your public services (Source = 'ingress'): one span per request, so counting rows counts requests. Requests between your services never pass the ingress and are not represented; neither are responses served from the CDN edge cache. Retention: up to 30 days.

SpanAttributes keys (values are strings):

observability.events

Kubernetes events about your service pods (probe failures, restarts, out-of-memory kills), in the OpenTelemetry log-record shape. Retention: up to 30 days.

observability.alert_evaluations

Every evaluation of your default alerts. This is what the alert graph in the dashboard shows; query it to see the exact numbers that did or didn’t fire. Retention: 30 days.

Default alerts

Every deployed service gets a set of default alerts, evaluated every minute from these same views and listed under Alerts in the dashboard; the project health indicator reflects whatever is firing. Each alert is a plain SQL query returning one number, compared against a threshold: open it in the dashboard to see the exact query, and paste it into specific query to get the same value the alert saw. What the alerts measure, how slow responses are judged, and how notifications work is covered in the Alerts guide.

Debugging recipes

Recent errors for a service:

SELECT Timestamp, Body
FROM observability.logs
WHERE ServiceName = 'api'
  AND SeverityNumber >= 17
  AND Timestamp >= now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC
LIMIT 50

Search log bodies for a substring:

SELECT Timestamp, ServiceName, Body
FROM observability.logs
WHERE positionCaseInsensitive(Body, 'connection refused') > 0
  AND Timestamp >= now() - INTERVAL 6 HOUR
ORDER BY Timestamp DESC
LIMIT 100

Service CPU timeseries in 1-minute buckets:

SELECT
  toStartOfMinute(TimeUnix) AS bucket,
  avg(Value)                AS cpu_cores
FROM observability.metrics
WHERE MetricName = 'container.cpu.usage'
  AND ServiceName = 'api'
  AND TimeUnix >= now() - INTERVAL 1 HOUR
GROUP BY bucket
ORDER BY bucket

Correlate all logs belonging to one trace:

SELECT Timestamp, ServiceName, Body
FROM observability.logs
WHERE TraceId = '<trace-id>'
ORDER BY Timestamp

Requests, error count and p95 latency per endpoint over the last hour:

SELECT
  ServiceName,
  SpanAttributes['http.method'] AS method,
  SpanAttributes['http.route']  AS route,
  count()                                                            AS requests,
  countIf(toUInt16OrZero(SpanAttributes['http.status_code']) >= 500) AS errors,
  round(quantile(0.95)(Duration) / 1e6)                              AS p95_ms
FROM observability.traces
WHERE Source = 'ingress'
  AND Timestamp > now() - INTERVAL 1 HOUR
GROUP BY ServiceName, method, route
ORDER BY requests DESC
LIMIT 50

Performance

The views are partitioned and ordered to make environment-scoped, time-bounded queries fast. To stay within the 30-second limit:

• Always filter by time - Timestamp >= for logs, traces and events, TimeUnix >= for metrics. Unbounded scans across 30/90 days will time out.
• Add ServiceName and/or MetricName to narrow the partition whenever you can.
• Select only the columns you need - Body, LogAttributes, and ResourceAttributes are the heavyweight columns.
• LIMIT exploratory queries while shaping them.

Discovering data

When you don’t know what’s there, ask the database:

SHOW TABLES FROM observability;
DESCRIBE observability.logs;

SELECT DISTINCT ServiceName FROM observability.logs LIMIT 50;
SELECT DISTINCT MetricName FROM observability.metrics LIMIT 100;

ClickHouse dialect

The observability store is ClickHouse, so specific query without --db uses the ClickHouse SQL dialect (unlike --db, which is standard Postgres). Commonly needed functions:

• now(), now64() - current time
• toStartOfMinute(t), toStartOfFiveMinutes(t), toStartOfHour(t) - bucket timestamps
• positionCaseInsensitive(haystack, needle) - case-insensitive substring search
• JSONExtractString(s, 'field'), JSONExtractInt, JSONExtractFloat - pull fields out of JSON log bodies
• LogAttributes['key'], Attributes['key'], ResourceAttributes['key'] - read map columns