Observability

Logs live in Loki and nowhere else: app instance output, build output and generic sandbox output are pushed there, and the web UI, the admin API and deno deploy logs read them back from it. Apps also export OpenTelemetry traces and metrics through Deno's built-in OTLP exporter, using the same channel Deno Deploy uses: vsock to the host. The app never sees an endpoint, a tenant or a credential; hh.pe attributes data to the sandbox it came from and adds those on the host.

Logs

Source Loki stream labels Structured metadata

App instance stdout/stderr and lifecycle events (starting, serving, idle, stopped, exit codes)

service_name=<app>, source=app

rev, replica, stream (out, err, sandboxo)

Build output: shell trace (+ ...), command output, limits, peak memory, image size, failure reason

service_name=<app>, source=build

rev, stream=build

Generic sandbox agent and envd output, lifecycle events

service_name=sandbox, source=sandbox

sandbox, stream

OTel log records an app emits itself (OTLP, see below)

service_name=<app>, service_instance_id=<replica> and Loki's other OTLP labels

service_version=<rev>, severity_text, trace_id, ...

Lines are batched for 250 ms (1000 lines max) and retried every 2 s while Loki is unreachable, keeping at most 20000 pending lines per instance, build or sandbox. Every build step's output is flushed before the build status changes, so a finished build's log is complete. Console output goes through stdout and stderr (OTEL_DENO_CONSOLE=ignore): Deno's OTLP log batches are lost when an instance is stopped before they are exported, stdout is not.

Readers: the app page shows the last 1000 lines of the last 7 days, build and sandbox pages everything since the revision or sandbox was created, GET /apps/<app>/logs?lines=N and GET /apps/<app>/revisions/<rev>/log return plain text, and deno deploy logs prints the history and then follows. Queries span at most 30 days (Loki's default max_query_length). Every reader goes through one place in sandboxod (sandboxod/otel.ts): identical windows asked for within 2 s share one Loki query, and the org page's usage totals share one Mimir query per 15 s.

The web UI renders those lines with the page and then streams new ones as server-sent events from /otel/logql; deno deploy logs continues the same way. Every follower of the same log, in the UI or the CLI, shares one Loki live tail (/loki/api/v1/tail, a WebSocket; delay_for holds lines for 1 s so late writers still arrive in order), so a new line shows up about a second after it was written. Loki allows 10 tails per tenant by default (max_concurrent_tail_requests) and ends each after an hour (tail_max_duration); sandboxod reconnects, and polls query_range every second while Loki refuses a tail. Metric charts refresh from /otel/promql every step while the tab is visible; identical chart queries from several viewers share one Mimir request per step. Both endpoints only run the query and tenant the page signed (see signing_secret below).

Without telemetry.loki logs are dropped (with a warning at startup) and the readers report that log storage is not configured.

OpenTelemetry

app (deno, OTEL_DENO=true, OTEL_DENO_VSOCK=2:4318)
  -> AF_VSOCK connect(cid 2, port 4318)        gVisor sentry (bundled runsc)
  -> /run/vsock/2/4318 in the sandbox
  -> slots/<network>/sandbox-<id>/vsock/2/4318 on the host
  -> hh.pe OtlpForwarder (one receiver per running instance)
       /v1/traces  -> telemetry.otlp.traces
       /v1/metrics -> telemetry.mimir.url + /otlp/v1/metrics
       /v1/logs    -> telemetry.loki.url + /otlp/v1/logs
       + X-Scope-OrgID: <tenant>, + configured headers

  • gVisor has no vsock. The bundled runsc ships a sentry with an AF_VSOCK socket provider: vsock sockets are unix stream sockets underneath, and (cid, port) maps to /run/vsock/<cid>/<port> (guest CID 3, host CID 2), the same convention Firecracker uses for guest-initiated vsock connections. The runtime sandboxes mount a host directory at /run/vsock and run with --host-uds=all.

  • When an instance starts, hh.pe opens an HTTP server on that sandbox's vsock/2/4318 and sends these variables in the start line, after the app's own env so the app config cannot override them: OTEL_DENO=true, OTEL_DENO_VSOCK=2:4318, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, OTEL_DENO_CONSOLE=ignore, OTEL_SERVICE_NAME=<app>, OTEL_RESOURCE_ATTRIBUTES=service.version=<rev>,service.instance.id=<replica>. Other OTEL_* settings from the app env apply, for example OTEL_METRIC_EXPORT_INTERVAL.

  • The forwarder accepts OTLP/HTTP (protobuf or JSON, compressed or not) up to 16 MiB per request, forwards it with a 10 s timeout and answers 503 when the backend is unreachable, so the exporter retries. Signals without a configured URL are accepted and dropped.

  • The tenant is set by the host, so an app can only write into its own tenant, even if it speaks OTLP over /run/vsock itself.

Only app instances export OpenTelemetry; builds and generic sandboxes only log. Export is on when Loki, Mimir or a traces URL is configured; otherwise apps get no OTLP variables. Runtime sandboxes always have vsock: the cron socket (vsock/2/4320, DENO_UNSTABLE_CRON_SOCK) uses it too.

Usage

The organization page (/<org>) shows requests, CPU time, outbound and inbound traffic of the last 24 hours, in total and requests per app. hh.pe meters this itself, so it does not depend on what an app exports:

  • requests, inbound and outbound bytes are counted by the zone for every request it forwards to an app: request line, headers and body in, status line, headers and body out (as sent to the client, after decompression). Traffic of an app's own outgoing connections is not counted;

  • CPU time is the host CPU of the instance's sandbox (sentry and application), read from its cgroup cpu.stat every flush and when it stops, minus the value when it was taken from the pool, so the warm V8 boot is not billed. This needs runtime sandboxes in cgroups (runtime_cgroup_parent, auto by default); zones without them are named on the organization page and their CPU time stays -. gVisor's own accounting (runsc events --stats) is not used: it reports about half a core for an idle Deno;

  • every telemetry.usage_interval_ms (15 s) the deltas since the last flush are pushed to Mimir as OTLP gauges sandboxo.app.requests, sandboxo.app.cpu.seconds, sandboxo.app.inbound.bytes and sandboxo.app.outbound.bytes with an app attribute, in the org tenant. The page sums them with sum_over_time(...[24h]), which is exact and survives restarts. Deltas that fail to push are kept for the next flush.

Metrics

/<org>/<app>/metrics charts the app over the last 15 minutes, hour, 6 hours, day or week, about 120 points per chart (step 15 s for 15 minutes, 30 s for an hour, 5 minutes for a day, 1.5 hours for a week), times in UTC:

Chart Source Query

Requests

usage, org tenant

sum(sum_over_time(sandboxo_app_requests{app}[step])), bars per step

Latency

Deno, app tenant

histogram_quantile 0.5, 0.95, 0.99 of rate(http_server_request_duration_bucket{job}[max(step, 2m)])

Responses

Deno, app tenant

rate(http_server_request_duration_count{job}) summed by status class (2xx ... 5xx)

CPU

usage, org tenant

sandboxo_app_cpu_seconds per step as percent of one core; needs runtime cgroups

Traffic

usage, org tenant

sandboxo_app_inbound_bytes and _outbound_bytes per step, in bytes per second

Memory

Deno, app tenant

sum(v8js_memory_heap_size{job}) over all instances

Charts are server-rendered SVG in a MetricChart island that adds a crosshair and the values under the cursor. Deno exports metrics every 60 s by default (OTEL_METRIC_EXPORT_INTERVAL), so the latency and response charts lag by up to a minute and stay empty for apps without OTel export.

Traces

/<org>/<app>/traces lists the app's requests from Tempo (telemetry.tempo, queried with the app's tenant): one row per root span, newest first, with the revision identicon, method and path, HTTP status and duration. It shows up to 100 traces of the last 15 minutes to 7 days, optionally only failed requests (span status error or a 5xx response) or only one revision ("Traces of this revision" on a build page). The query is TraceQL:

{ resource.service.name = "<app>" && nestedSetParent < 0 [&& ...] }
  | select(span.http.request.method, span.url.path,
           span.http.response.status_code, span.cron.name,
           resource.service.version, status)
  with (most_recent=true)

The search box takes the cron filters of Deno Deploy: kind:cron adds span.cron.name =~ ".+", cron.name:<name> and cron.schedule:<schedule> (quote values with spaces, cron.schedule:"0 * * * *") match those span attributes; span.cron.name is selected as well. Tempo reports selected attributes a span does not have as the string nil, which the list treats as missing.

Cron runs are traced from the platform side: the scheduler mints the trace id and passes it to deno as traceparent, and the zone exports a root span cron <name> (internal, cron.name, cron.schedule, cron.attempt, cron.run.id, error status with the failure reason) as OTLP/JSON to the traces URL with the app's tenant and resource. deno's own deno.cron span, with the handler's exception, and everything the handler does nest under it. The Cron pages link each run to its trace.

A trace page (/<org>/<app>/traces/<id>, from Tempo's /api/v2/traces/<id>) shows the spans as a waterfall: nested by parent, bars on the trace's time axis, colored by kind (server, client, internal) and red on error. Opening a row shows its attributes, events (such as recorded exceptions) and status message. Below are the app's logs from 20 ms before the trace to 100 ms after it, for the trace's revision: console lines carry no trace id, so this is a time window and can include lines of concurrent requests. Times on these pages are UTC.

Configuration

Zones push: app, build and sandbox logs, the OTLP forwarder and usage metering run in zoned, so a zone needs the Loki, Mimir and traces URLs and headers. Usage gauges carry a zone attribute. sandboxod pushes only the build lines it writes itself (git fetch, dispatch failures) and queries Loki, Mimir and Tempo for the web UI and the APIs. Both take the same telemetry section in their config (see Configuration); header values are secrets:

"telemetry": {
  "loki": { "url": "https://loki.example.com", "headers": { "Authorization": { "_env": "LOKI_AUTH" } } },
  "mimir": { "url": "http://127.0.0.1:9009" },
  "tempo": { "url": "http://127.0.0.1:3200" },
  "otlp": { "traces": "http://127.0.0.1:4318/v1/traces" }
}
Key Meaning

telemetry.loki.url

Loki base URL, for example http://127.0.0.1:3100: push API, query API and OTLP logs endpoint

telemetry.loki.headers

Extra headers for every Loki request, for example credentials of a hosted Loki

telemetry.otlp.traces

Full OTLP/HTTP traces URL, for Tempo http://127.0.0.1:4318/v1/traces

telemetry.otlp.headers

Extra headers for forwarded traces

telemetry.otlp.max_body_bytes, telemetry.otlp.timeout_ms

Largest forwarded OTLP request (16 MiB) and the forward timeout (10 s)

telemetry.tempo.url

Tempo query API base URL, for example http://127.0.0.1:3200, for the traces pages (ingest goes to telemetry.otlp.traces)

telemetry.tempo.headers

Extra headers for Tempo queries

telemetry.mimir.url

Mimir base URL, for example http://127.0.0.1:9009: OTLP metrics endpoint for app metrics and usage, Prometheus query API for the organization page

telemetry.mimir.headers

Extra headers for every Mimir request

telemetry.tenant

X-Scope-OrgID value: app (default, one tenant per app; generic sandboxes use the org), org, or a fixed string

telemetry.usage_interval_ms

How often zones push usage deltas (15 s)

runtime_cgroup_parent (zoned)

Cgroup parent for runtime sandboxes, auto (default), a path or null (see Configuration). Enables CPU time usage

signing_secret (sandboxod)

Required, a secret of at least 32 characters, keep it stable: it signs the query URLs pages hand to the log and chart islands (valid for 12 hours and only with the same login) and the e2b template upload links. Rotating it stops open pages from refreshing until they are reloaded. The dev config reads a fixed value from the dev shell's SANDBOXO_SIGNING_SECRET; the NixOS module generates one into /var/lib/sandboxo-secrets/ unless services.sandboxod.signingSecretFile is set

Local backends

process-compose.yaml runs Grafana Mimir, Tempo and Loki from nixpkgs (all in the dev shell), single binary each, multi-tenant, listening on 127.0.0.1 only. Their object storage is a single-node Garage with buckets loki, mimir and tempo, next to the sandboxo bucket for hh.pe blobs; garage-init (nix/garage-init.sh) assigns the layout, imports the dev key and creates the buckets before the backends start. Redis (AOF persistence) holds hh.pe records. The dev shell sets the shared dev credentials (GARAGE_RPC_SECRET, SANDBOXO_S3_ACCESS_KEY_ID, SANDBOXO_S3_SECRET_ACCESS_KEY) and SANDBOXO_REDIS_URL, which the dev config reads through _env placeholders, so run process-compose and dev.ts from it. Local state (WAL, Garage metadata and data, the AOF) lives in observability/data/, configs in observability/:

Backend HTTP Other ports

Mimir

9009

gRPC 9095

Tempo

3200

OTLP gRPC 4317, OTLP HTTP 4318, gRPC 9195

Loki

3100

gRPC 9096

Garage

S3 3900

RPC 3901, admin 3903

Redis

-

6380

The dev config (SANDBOXO_DEV_CONFIG, see dev.ts) points hh.pe at them through its telemetry section, and the dev shell configures process-compose itself (PC_CONFIG_FILES=process-compose.yaml, API on PC_PORT_NUM=18080):

nix develop
process-compose up            # TUI; add -t=false to run headless
SANDBOXO_TOKEN=secret deno task dev

Every query needs the tenant header, for an app hello:

curl -H 'X-Scope-OrgID: hello' 'http://127.0.0.1:3200/api/search?tags=service.name%3Dhello'
curl -H 'X-Scope-OrgID: hello' 'http://127.0.0.1:3100/loki/api/v1/query_range?query=%7Bservice_name%3D%22hello%22%7D'
curl -H 'X-Scope-OrgID: hello' 'http://127.0.0.1:9009/prometheus/api/v1/label/__name__/values'

In Grafana, add one Prometheus (http://127.0.0.1:9009/prometheus), Tempo and Loki data source per tenant with the X-Scope-OrgID header set.

Mimir must advertise 127.0.0.1 for every ring and for the query frontend (frontend.address); with an auto-detected interface address its queriers cannot report finished queries and every query hangs. Loki's ingestion limits are raised (32 MB/s, 64 MB bursts) so large build logs are not rejected.

The smoke tests that read logs (smoke/deploy.ts, smoke/compat.ts, smoke/build_limits.ts, smoke/ui.ts) need the backends running and exit early otherwise. smoke/otel.ts also checks that an app's traces, logs and HTTP metrics arrive under its tenant, that another tenant sees none, that its requests are counted exactly on the organization page and, with SANDBOXO_RUNTIME_CGROUP set (auto works in a user session; the smoke passes it as runtime_cgroup_parent), that its CPU time is plausible.