Zones

hh.pe runs as two programs plus obice routers. sandboxod is the control plane: it owns every record (Redis), takes uploads and git fetches into blob storage (S3), serves the APIs, the web UI and the SSH gateway, and serves the routers their routes. zoned runs the gVisor sandboxes of one availability zone: app instances, builds, generic sandboxes, template builds and btrfs volumes. Routers are obice instances (envoy driven by obice's own xDS server) that take the public traffic for apps and sandbox ports over HTTPS and send it straight to the zones. A zone talks to S3, to the telemetry backends and to sandboxod; it never touches Redis and takes no inbound connections: its own obice dials a reverse tunnel to every router, so zones can sit behind NAT.

zones apps, sandbox ports APIs, UI, ssh GET routes zone API (egress socket) reverse tunnels (HTTP/2) register clientsbrowsers, deno deploy, @deno/sandbox, e2b, ssh routers (obice)the control host's, standalone ones sandboxodAPIs, UI, ssh, placement, scaling, records obice + zoned ord1 obice + zoned ams1 Redis S3 OTLP (Mimir, Loki, Tempo)

Zones and regions

A zone has a name (a DNS label of at most 32 characters, for example ord1) and belongs to one region (three lowercase letters, ord). Ids keep the region (sbx_ord_..., vol_ams_...); records store the zone. A zone's region is fixed once it has registered: registering the same name with another region is rejected. sandboxod's region is the default region for new sandboxes and volumes.

Registration

zoned starts with a new random boot id and keeps only slots/ and its run directory ephemeral: paused checkpoints, volumes, egress state and caches survive restarts. It posts its ZoneInfo (name, region, URL, boot id, protocol version, org, tenant mode, profiles, features, limits) to <control>/zone/v1/register with its token in x-sandboxo-zone-token, every 30 s, and every 2 s while that fails. The answer lists the routers the zone should tunnel to.

sandboxod accepts the registration when the token matches the zone's record (zones/<name> in Redis), the org and tenant mode match its own, the region matches the recorded one and the protocol version is supported. It then dials GET /v1/events on the zone through its router: an NDJSON stream that starts with a hello snapshot of everything the zone holds, then carries events, a ping with current capacity every 5 s and a load with the in-flight requests of every replica. A zone is ready while the stream is open and something arrived in the last 15 s; operations on a zone that is not ready answer 503 ZONE_UNAVAILABLE, and nothing new is placed there. sandboxod reconnects every second. Known zones are dialed again when sandboxod starts, before they register.

Tokens:

  • zone_tokens in sandboxod's config, for example {"ord1": {"_file": "/run/keys/ord1"}}, writes those tokens into the zone records at startup (the NixOS module, dev.ts and the smoke harness use this);

  • PUT /zones/<name> on the API port (with the API token) creates or rotates a zone's token and returns it once: {"name", "token"}.

zoned reads its own from token in its config. The zone's envoy adds the token to every request that arrives over a tunnel, so routers never hold zone tokens. Tunnels, route fetches and registration are not authenticated beyond that yet: keep routers, zones and sandboxod on a private network or put TLS in front. Registration and route fetches are served on sandboxod's API port, so that port must only be reachable from that network.

GET /zones lists zones with status (ready, disconnected, unregistered), region, url, cordoned, bootId, features, limits and the last reported capacity. POST /zones/<name>/cordon stops new placements in a zone; {"cordoned": false} as the body lifts it. Work pinned to a cordoned zone (its volumes, snapshots and paused sandboxes) still runs there.

Routers and tunnels

A router is an obice. Its envoy bootstrap (sandboxod router-bootstrap, or nix/router-bootstrap.nix in the NixOS modules) holds the tunnel listener that zones dial and, on the router next to sandboxod, an egress listener on a unix socket through which sandboxod reaches the zone API. The zone's envoy adds the zone token to everything that comes through its tunnels, so the socket is mode 0660 and only sandboxod's group may use it (/run/sandboxo-egress/egress.sock, group sandboxo-egress, in the NixOS module). Everything else is an obice remote source: the router long-polls GET /router/v1/source?name=<router>&tunnel=<host:port> on sandboxod's API port and gets a document in obice's source format:

  • one zone-<name> tunnel cluster per zone, reaching that zone's tunnels;

  • routes on obice's https listener, matched on the host (port ignored) in the order sandboxod's front door always had: running sandbox port labels (<port>-sbx-<region>-<id>.<domain>) and exposure labels, the e2b-sandbox-id header of running sandboxes, 404 sandbox not found for other sandbox labels and ids, then apps (<app>.<domain>, the root app on <domain>, previews <app>--<rev>.<domain>), 404 app is not deployed, 503 app is unavailable for an app without a zone in its region, and 404 unknown app for the rest of <domain> and *.<domain>. Other hosts are left to the virtual hosts of the router's own obice config.

Routes add x-sandboxo-route, the original URL and the client address. The router's obice config drops client headers starting with x-sandboxo- (http.strip_headers), and obice sets x-forwarded-for, x-forwarded-proto, x-forwarded-host and x-real-ip and answers with HSTS. sandboxod rebuilds the document on every promotion, placement change and sandbox change. Promotions, sandbox creation, resume and exposing a port only return once every router that polled in the last 5 s reports (Obice-Accepted) that its envoy applied the new routes, so the next request finds them.

A poll registers the router: sandboxod keeps it in Redis (routers/<name>) and pushes the router list to the zones (PUT /v1/routers), so every zone dials every router. A router that stops polling stays in the list for 10 minutes, so a sandboxod restart does not make zones drop their tunnels.

Routers serve <domain> and *.<domain> with the certificate in their obice config (certificates.sandboxo), from obice's ACME or from files. In development, process-compose up runs the router local on port 8443 (obice run --http-port, without a plain http listener) with the packages.dev-tls certificate for sandboxo.localhost, and the dev config sets domain and port to match. A router elsewhere:

sandboxod router-bootstrap --name edge1 --tunnel '[::]:8005' \
  --log-dir /var/log/router > bootstrap.json
obice run --bootstrap bootstrap.json --config obice.json \
  --remote-source 'sandboxo=http://control.example.com:8001/router/v1/source?name=edge1&tunnel=edge1.example.com:8005' \
  -- --concurrency 2

where obice.json sets http.strip_headers: ["x-sandboxo-*"] and the certificate. On NixOS, services.sandboxo-router sets this up in the host's services.obice, and services.sandboxod makes the host's obice the router local (router.enable).

Each zone runs its own obice as a child of zoned and serves it the router list as a remote source on loopback. It keeps 4 tunnels per router: envoy keeps tunnels per worker thread, so routers must not run more than 4 workers. The modules pass --concurrency 2, which also applies to everything else the host's obice serves. zoned itself only listens on loopback.

Placement

There is no replicated filesystem, so volumes, snapshots and paused checkpoints stay in the zone that created them, and so does everything that uses them:

  • a sandbox mounting volumes runs in their zone; volumes from two zones in one sandbox are rejected with 400 VOLUMES_IN_DIFFERENT_ZONES (volume 'a' (zone ord1) and volume 'b' (zone ams1) are in different zones);

  • a snapshot is taken in its volume's zone, a volume created from a snapshot lives in the snapshot's zone;

  • a paused sandbox resumes in the zone that holds its checkpoint;

  • if that zone is not ready the request fails with 503 ZONE_UNAVAILABLE; if an explicit region does not match it, with 400 REGION_MISMATCH.

Everything else is free to move:

  • unpinned sandboxes and volumes go to a ready, uncordoned zone with the feature (sandboxes, volumes, egress policies need egress), in the requested region if one is given (400 INVALID_REGION for an unknown one), otherwise preferably in the default region and in any region when it has none. Sandboxes prefer the zone with the most free sandbox slots, then the most warm sandboxes; ties are broken at random. When a zone answers LIMIT_REACHED, the next one is tried;

  • apps run as replicas, see [scaling];

  • a build goes to the zone with the most free build slots; a template build to the zone running the fewest template builds. Images and logs are stored in S3, so a template built in one zone boots in every other.

Scaling

An app runs as one or more replicas, each an isolated runtime sandbox with its own id (6 characters, [0-9a-z]), the way Deno Deploy runs several instances of an application. Two settings in the app config (UI Configuration page, admin API) control them; like the rest of the config they apply to the next deploy:

region

A region code (ord) or global (default). Replicas only run in zones of that region; with no ready zone there the app answers 503. deno deploy create --region only accepts us, eu and global, so from the CLI only global is taken over.

maxReplicas

1 (default) to 16.

sandboxod decides how many replicas each zone runs; the zones pick the replica for every request:

  • Every deployed app is assigned a home zone, preferably in the default region (sandboxod's region), then the zone that built the revision, then where most warm runtime sandboxes wait. sandboxod sends the zone the instance spec and its replica count (PUT /v1/instances/<key>); the zone starts a replica on the first request.

  • Routers spread an app's requests over its zones by their replica counts. Inside a zone, a request goes to the replica with the fewest in-flight requests; when the zone runs fewer replicas than it was given, it starts the missing ones.

  • Zones report the in-flight requests of every replica, including requests waiting for a replica to start and cron runs, every second. When they exceed scaling.target_inflight (20) per replica, sandboxod adds a replica (at most one every 2 s, up to maxReplicas) in the region with the fewest replicas of the app, then the zone with the fewest, so a global app spreads over regions and zones.

  • When the smoothed load would fit one replica fewer at half the target for scaling.scale_down_ms (60 s), the zone with the most replicas gets one less; it drains its least loaded replica (up to zoned's drain_ms) and stops it. A lower maxReplicas or a changed region shrinks the same way.

  • A zone that gets a request for an app it holds no replicas for answers 409 unknown-instance before reading the body, and the router retries in another zone. A zone that does not know an app at all, after a restart or for a preview, asks sandboxod for the spec (GET /zone/v1/instances/<key>).

  • Idle replicas stop after idle_ms on their zone; an app without traffic runs nothing, and the next request starts a replica in its zone.

  • A zone that stops responding loses its replicas, and they move to another allowed zone. After a sandboxod restart the zones' hello snapshots bring the replica counts back.

  • A preview runs in the zone that first asked for it.

Every app log line carries the replica as structured metadata (replica), and the app's own OpenTelemetry data carries service.instance.id, so logs show a second, round identicon for the replica next to the revision's. GET /apps/<app> and the app overview list the replicas the zones report with their zone, in-flight requests and state.

Failures

Zone down

Its sandboxes answer 503, listing them still works. Apps move to other zones; routers retry requests that cannot reach a zone in another one. After 60 s without the zone, its builds go back to the queue (at most two attempts per revision) and its template builds fail; if the zone comes back still running them, they are cancelled there.

Zone restart (new boot id)

Sandboxes that were running there are recorded as stopped (zone restarted). Paused checkpoints and volumes are kept and resume there. Builds lost by the restart are queued again, template builds fail.

Router down

Clients of that router lose their connections; zones keep their tunnels to the others. A router that comes back fetches its routes again and the zones dial it again.

sandboxod restart

Zones keep their sandboxes running and keep enforcing deadlines (sandboxod sends absolute times, so clocks must be synchronized). Deno session sandboxes, those without a timeout, end with their owning connection and are stopped. Routers keep serving the routes they have and fetch new ones once sandboxod is back.

Reconciliation

On every hello sandboxod compares the snapshot with its records: tombstones of sandboxes that ended while it was away are applied, running records the zone no longer has become stopped (or paused, if it holds their checkpoint), sandboxes the zone runs without a record are killed (after a 60 s grace for creations in flight), paused records whose checkpoint is gone become stopped (checkpoint lost), and orphan checkpoints are discarded. Finished builds, cron runs and tombstones are kept by the zone until sandboxod acknowledges them, and the first result applied wins. Cron runs the zone no longer knows fail; runs still running after a sandboxod restart fail if no zone claims them within 30 s.

Zone API

All requests carry x-sandboxo-zone-token; ids are minted by sandboxod and every operation is idempotent by id. Errors are {"error": {"code", "message"}}.

Endpoint Meaning

GET /v1/info

zone info and capacity

GET /v1/events

NDJSON: hello, ping, load, sandbox.ended, build.finished, templateBuild.log, templateBuild.finished, cron.finished

POST /v1/ack

{"sandboxes"?, "builds"?, "templateBuilds"?, "cronRuns"?}: forget delivered results

PUT /v1/routers

[{"name", "address"}]: routers to tunnel to

PUT, DELETE /v1/instances/<key>

{"spec", "replicas"}: app instance spec (image, config, env, cron names) and how many replicas the zone runs, 0 drains them; stop all replicas with ?drain=1

PUT /v1/cron-runs/<id>

run a cron on one of the app's replicas (started if needed); 202, the result arrives as cron.finished; 409 UNKNOWN_INSTANCE when the zone runs no replicas of the app

PUT, DELETE /v1/builds/<id>

start or cancel a build

PUT /v1/template-builds/<id>, GET .../log?offset=, DELETE

template builds

PUT /v1/sandboxes/<id>, DELETE, POST .../pause, POST .../resume, PUT .../deadline

generic sandbox lifecycle

GET /v1/sandboxes/<id>/agent

WebSocket, NDJSON agent protocol

GET /v1/sandboxes/<id>/tcp/<port>

WebSocket byte tunnel to a port, text frame eof half-closes

PUT, DELETE /v1/volumes/<id>, GET .../tar, PUT, DELETE /v1/snapshots/<id>

volumes and snapshots

Proxied traffic (app requests, sandbox ports, envd) is any request with x-sandboxo-route: kind=instance&key=<app>&rev=<rev>, kind=preview&app=<app> (the revision comes from the host), or kind=sandbox&id=<id>&port=<port> where the port is a number, host (from the port label) or header (from e2b-sandbox-port, 49983 without it). The original URL is in x-sandboxo-url and the client address in x-sandboxo-client; zoned strips every x-sandboxo-* header before forwarding.

Running

Both programs and dev.ts take a JSON config file, see Configuration. SANDBOXO_TOKEN=secret deno task dev starts sandboxod and zones in one process for development (dev.ts), and process-compose up runs the router. As separate processes, with ord1.token holding the zone token:

{
  "token": { "_env": "SANDBOXO_TOKEN" },
  "signing_secret": { "_env": "SANDBOXO_SIGNING_SECRET" },
  "zone_tokens": { "ord1": { "_file": "ord1.token" } },
  "blobs": { "backend": "fs" }
}
{
  "name": "ord1",
  "token": { "_file": "ord1.token" },
  "state": "state/zones/ord1",
  "blobs": { "backend": "fs", "dir": "state/blobs" }
}
echo dev-zone-token > ord1.token
SANDBOXO_TOKEN=secret deno run -A sandboxod/main.ts --config sandboxod.json &
deno run -A zoned/main.ts --config zoned.json

The flake builds packages.sandboxod (no gVisor, profiles or passt in its closure, checked by checks.closures), packages.zoned, packages.sandboxo (the all-in-one launcher) and packages.dev-tls (the development CA and certificate for sandboxo.localhost). The NixOS module imports obice's and offers services.sandboxo for one host (sandboxod, Redis, Garage and a zone local1 with a generated token), or the parts:

# control host
services.sandboxod = {
  enable = true;
  domain = "deploy.example.com";
  tokenFile = "/run/keys/sandboxo-token";
  apiHost = "10.0.0.1";
  tunnel.address = "10.0.0.1:8005";
  router.enableACME = true;
  garage.s3Address = "10.0.0.1:3900";
  zones.ord1.tokenFile = "/run/keys/zone-ord1";
};
services.obice.acme.nameserver = "10.0.0.53:53";

# zone host, behind NAT is fine
services.zoned = {
  enable = true;
  name = "ord1";
  region = "ord";
  tokenFile = "/run/keys/zone-ord1";
  controlUrl = "http://10.0.0.1:8001";
  settings.blobs = {
    backend = "s3";
    endpoint = "http://10.0.0.1:3900";
    access_key_id._env = "SANDBOXO_S3_ACCESS_KEY_ID";
    secret_access_key._env = "SANDBOXO_S3_SECRET_ACCESS_KEY";
  };
  environmentFiles = [ "/run/keys/zone-s3.env" ];
};

# another router
services.sandboxo-router = {
  enable = true;
  name = "edge1";
  domain = "deploy.example.com";
  tunnel.address = "edge1.example.com:8005";
  controlAddress = "10.0.0.1:8001";
  enableACME = true;
  openFirewall = true;
};
services.obice.acme.nameserver = "10.0.0.53:53";

The zone's S3 key needs read and write access to the bucket. /var/lib/zoned should be on btrfs for volumes.

Tests

sandboxod/placement_test.ts covers the placement rules, scaler_test.ts the replica counts, routes_test.ts the route order and that obice check takes the source, and router_test.ts runs obice with the router bootstrap and checks that envoy accepts the generated routes and serves them over HTTPS; common/obice_test.ts covers the route source server. smoke/multizone.ts runs sandboxod and zones ord1 and ams1 in one process and checks app failover, volume placement, the mixed-zone rejection, snapshots and forks, pausing and resuming across a zone restart, a sandboxod restart and a template built in one zone used in the other. smoke/routers.ts adds a standalone router and checks that both zones reach it and that the routers serve without each other. smoke/tunnel.ts measures the path through the tunnel and a tunnel loss. smoke/cron.ts covers cron discovery, scheduled and manual runs, retries, timeouts, a sandboxod restart and a promotion during a run. checks.module boots the single-host module; checks.multizone a control VM, two zone VMs that take no inbound connections and a router VM.