State directory

Persistent server state goes through three interfaces: records through a key-value store (Kv in sandboxod/kv.ts), content-addressed blobs through BlobStorage (common/blobs.ts) and volume data through VolumeStorage (zoned/btrfs.ts). By default records live in Redis (RedisKv) and blobs in an S3 bucket (S3BlobStorage, Garage in the dev shell and the NixOS module); "kv": {"backend": "fs"} and "blobs": {"backend": "fs"} keep them as plain files instead.

Records belong to sandboxod alone; zones never read Redis (see Zones). Both programs have a state directory, state in their config (see Configuration): sandboxod's (default ./state) holds its git cache and SSH host key, plus records and blobs with the fs backends. A zone's (default ./state/zone, state/zones/<name>/ with the all-in-one dev.ts) holds everything that runs there: volumes as btrfs subvolumes, paused checkpoints, sandbox bundles and a local cache of S3 blobs. With fs blobs every zone must see sandboxod's blobs/ directory (blobs.dir), so fs only works on one host. Logs are not stored here: app, build and sandbox logs go to Loki only, see Observability.

kv is sandboxod's only; blobs is required in both programs and must point at the same storage:

Key Default Meaning

kv.backend

redis

redis or fs (state/kv/)

kv.url

redis://127.0.0.1:6379

secret, redis://[:password@]host[:port][/db], rediss:// for TLS

kv.prefix

sandboxo:

prefix of every Redis key, so servers can share a database

blobs.backend

required

s3 or fs

blobs.dir

<state>/blobs

blob directory for fs

blobs.endpoint

http://127.0.0.1:3900

S3 endpoint URL, path-style requests

blobs.region

garage

signing region

blobs.bucket

sandboxo

bucket, must exist

blobs.access_key_id, blobs.secret_access_key

required for s3

secrets, credentials with read, write and delete on the bucket

blobs.prefix

empty

prefix of every object key

blobs.concurrency

32

parallel S3 requests

Two servers must not share a Redis prefix or an S3 prefix: garbage collection deletes every blob under its prefix that its own records do not reference.

Layout

sandboxod:

state/
  kv/                            fs kv only: records, see Key-value store
    apps/<app>/
      config.json                app config
      meta.json                  app id and timestamps
      current.json               promoted revision id
      playground.json            playgrounds only: code volume and creation time
      env/<NAME>.json            one environment variable
      revisions/<rev>.json       revision record
      status/<rev>.json          build status
      crons/<name>.json          one cron job's recent runs (name base64url)
    sandboxes/<sandbox id>.json  generic sandbox record
    volumes/<volume id>.json     volume record
    snapshots/<snapshot id>.json snapshot record
    org-slugs/<org>/<slug>.json  volume and snapshot slug index, per organization
    templates/<template id>.json e2b template record
    template-builds/<template id>/<build id>.json   template build record
    template-files/<template id>/<files hash>.json  uploaded COPY archive
    org-template-names/<org>/<name>.json  template name index, per organization
    zones/<name>.json            zone record and token
    routers/<name>.json          router, its tunnel address and when it last fetched routes
    editors/<org>/<user>.json    VS Code user settings, synced from editors
    sessions/<sha256(id)>.json   UI login session (oidc only)
    meta/legacy-org.json         organization that owns pre-tenancy records
    .trash/                    subtrees being deleted
  blobs/                         fs blobs only, shared with the zones
    <sha256>                     file contents, tree objects, build and template images, manifests, logs
    .tmp-<uuid>                  blob being written
  blob-cache/                    s3 blobs only
    .tmp-<uuid>                  upload being hashed
  git/
    <hash(url)>.git/             bare repository cache (gix)
  ssh_host_ed25519_key           SSH gateway host key

zoned:

state/zone/
  blob-cache/                    s3 blobs only
    <sha256>                     local copy of a blob opened as a file
    .tmp-<uuid>                  blob being hashed before upload
    <sha256>.<uuid>.part         blob being downloaded
  volumes/                       btrfs only
    <volume id>/                 volume data, a subvolume
    <snapshot id>/               read-only snapshot of a volume
    .trash/                      subvolumes being deleted
  paused/
    <sandbox id>/                runsc checkpoint of a paused sandbox
    <sandbox id>.json            its spec and creation time
  egress/
    ca/                          iron-proxy CA
    sandboxes/<sandbox id>/      proxy config and secrets of one sandbox
  templates/
    layers/                      registry layer cache
    bases/                       unpacked base images
    work/                        template builds in progress, wiped on startup
  nix/
    <encoded flake ref>          GC root symlink to a sandbox profile
    sandbox-store/nix/store/     private Nix store, mounted at /nix/store
  slots/                         sandbox bundles, wiped on startup
    none/sandbox-<id>/           warm runtime sandboxes, network none
    sandboxed/sandbox-<id>/      warm runtime sandboxes, network sandboxed
    build/<id>/                  build sandboxes
    sandbox-<network>/sandbox-<id>/   generic sandboxes (e2b, Deno Sandbox)
    volume-tar/                  sandboxes packing volume archives
    templates/                   template build sandboxes
  tmp/
  tools/
    inspect.js                   repository inspection, run in build sandboxes
    agent.js                     generic sandbox agent
  envoy/                         the zone's obice: bootstrap.json (holds the zone token), obice.log, envoy.log, tunnels.log
Path Lifetime Created by

kv/, blobs/

persistent (sandboxod)

FsKv.init, FsBlobStorage.init (sandboxod/kv.ts, common/blobs.ts)

volumes/, paused/, egress/

persistent (zone)

BtrfsVolumeStorage.init (zoned/btrfs.ts), SandboxHost (zoned/sandboxes.ts), Egress.create (@in3pre3/sandbox/egress)

blob-cache/

cache, safe to delete while the program is stopped

S3BlobStorage.init (common/blobs.ts)

git/

cache, safe to delete

GitCache.init (sandboxod/git.ts), git_cache moves it

nix/

cache, rebuilt on startup

SandboxTemplate.prepare (@in3pre3/sandbox), zoned's gc_root_dir moves it

templates/

cache, work/ removed on every startup

TemplateBuilder (zoned/templates.ts)

slots/

ephemeral, removed on every zone startup

sandbox pools (zoned/pool.ts, zoned/)

tools/

regenerated on every startup

bundleTools (zoned/tools.ts, Deno.bundle)

ssh_host_ed25519_key

persistent, created once (mode 0600)

SSH gateway (sandboxod/ssh.ts), unless sandbox.ssh is null

Key-value store

Kv (sandboxod/kv.ts) stores JSON values under keys that are lists of segments matching [A-Za-z0-9_-]+. Store (sandboxod/store.ts) is the only user; it validates records with zod on read. Operations: get, set, create (only if absent), update (read-modify-write), delete, deleteTree (a key and everything below it) and list (the immediate children of a prefix, sorted). Keys with other segments read as missing; writing them throws.

FsKv maps key [a, b, c] to the file kv/a/b/c.json and its children to the directory kv/a/b/c/, so a key can hold a value and have children.

  • set writes .tmp-<uuid> next to the target (mode 0600) and renames it into place.

  • create writes the temp file and hardlinks it to the target, which fails atomically if the target exists.

  • update runs one at a time per key within the process.

  • deleteTree renames the subtree into kv/.trash/<uuid> (atomic), deletes it there, then deletes the key's own value.

  • On startup FsKv.init empties .trash/ and removes stray .tmp-* files.

RedisKv keeps the value of [a, b, c] as a JSON string under <prefix>v:a/b/c and the children of every prefix in a set, <prefix>i:a/b holding c, <prefix>i:a holding b and <prefix>i: holding a.

  • set, create and the final write of update are one Lua script each, so the value and the parent sets change atomically. create is SET only if absent.

  • update reads the value, runs the function and writes only if the value is still the one it read (compare-and-set in the script), retrying on conflict, so it is atomic across processes. The function may run more than once.

  • delete removes the value and, when the key has no children, its name from the parent set, in one script.

  • deleteTree walks the sets below the prefix and deletes every value and set it finds; it is not atomic.

  • list reads the prefix's set. A name stays listed while it has children, like a directory in FsKv.

Apps

kv/apps/<app>/ exists for every app; <app> is a lowercase DNS label of at most 32 characters, so preview hosts <app>--<rev> stay within the 63-character label limit. It starts with a letter, so it never looks like a sandbox port host (<port>-sbx-<region>-<20>); ssh and names shaped like exposeHttp labels (x-<region>-<20>) are reserved.

config.json

The app config. Fields: entrypoint, args, installCommand, buildCommand, preDeployCommand, buildDirectory, network (none or sandboxed), buildTimeout (minutes or null), buildMemoryLimit (MiB or null), region (3-letter region code or global) and maxReplicas (1 to 16), see Scaling. Missing fields take their defaults (appConfig in common/appconfig.ts), so older files stay readable. The environment is not stored here: readConfig fills env from env/ and writeConfig drops it.

env/<NAME>.json

One environment variable per key, so every change is a single atomic write: {"value", "secret", "local", "updatedAt"}. <NAME> matches [A-Za-z_][A-Za-z0-9_]*. local puts the variable in the Local context (context_ids: null in the deno deploy API, otherwise production and preview). A secret variable is write-only: its value is never sent to a browser, the admin API or the deno deploy CLI, it cannot become non-secret again, its local flag is fixed, and saving it with an empty value keeps the sealed one. Values are stored as plain text; builds and instances get all variables, and revision records keep a snapshot of them.

meta.json

{"id", "createdAt", "updatedAt"}: a UUID used as the app id by the deno deploy API, and ISO timestamps. updatedAt changes on every config write.

current.json

The promoted revision id as a JSON string. Promotion is a single set, so the switch is atomic. Missing until the first promotion.

playground.json

Playgrounds only: {"volume", "createdAt"}, the volume holding the code.

Builds and revisions

A revision id is <milliseconds in base 36>-<8 hex> (for example mvfobbyg-21e28c6c), so ids sort by creation time. Both live under kv/apps/<app>/; revisions are listed by their status/ keys.

status/<rev>.json

{"status", "updatedAt"}: status is uploading, queued, building, ok or failed, updatedAt the ISO time of the last change.

revisions/<rev>.json

The revision record:

{
	"prod": true,
	"createdAt": "2026-10-19T19:54:54.664Z",
	"config": { "entrypoint": "...", "...": "..." },
	"source": { "tree": "<sha256 of the root tree>" },
	"git": { "url": "https://...", "ref": "main", "commit": "<sha1>" },
	"manifest": { "blob": "<sha256>", "files": 42 },
	"detect": { "install": true, "build": true, "runtime": true },
	"build": { "image": "<sha256>", "size": 80928768, "nodeModules": true },
	"crons": [{ "name": "cleanup", "schedule": "0 * * * *", "backoffSchedule": [1000] }]
}

config is a snapshot of the app config (env included) taken when the revision was created; later config changes apply to the next revision. source.tree is the root of the source tree in blobs/; it is set once the upload completes or the git fetch finishes, and a redeploy copies it. git is set for quick-deploy revisions (ref only when given, commit after the fetch). manifest points to the deno deploy upload manifest, stored in blobs/ because it can list hundreds of thousands of files, and keeps its file count. It stays after the upload. detect marks the settings to fill from repository inspection. build is set by a successful build: image is the build output as an EROFS image in blobs/ (the source copy plus everything the build produced: node_modules, .deno_dir with the module cache, framework output such as .next), mounted read-only at /app in runtime sandboxes; nodeModules adds --node-modules-dir=manual to the start line. crons lists the Deno.cron() jobs the revision registered when the build booted it once (the warmup); a revision without the field was built before cron support and runs no crons. Failed builds keep no output.

crons/<name>.json

{"name", "runs"} for one cron job, keyed by the base64url of its name (cron names may contain spaces). runs holds the last cron.history runs, oldest first:

{
	"id": "mvhngj9o-1a2b3c4d",
	"schedule": "0 * * * *",
	"rev": "mvhngbru-139140f2",
	"attempt": 1,
	"scheduledAt": "2026-10-21T05:00:00.000Z",
	"startedAt": "2026-10-21T05:00:00.012Z",
	"finishedAt": "2026-10-21T05:00:00.140Z",
	"status": "ok",
	"traceId": "<32 hex>",
	"zone": "ord1"
}

status is running, ok or error (with error); a retry is a new run with attempt + 1.

Blob storage

blobs/ holds content-addressed objects named by the hex sha256 of their bytes (FsBlobStorage in common/blobs.ts):

  • file blobs: raw file contents, the same hashes the deno deploy CLI uses in upload manifests;

  • tree blobs: one directory each, canonical JSON {"tree":1,"entries":[...]} sorted by name, with entries {name, type: "file", mode, size, hash}, {name, type: "dir", hash} or {name, type: "symlink", target} (common/tree.ts);

  • build images: uncompressed EROFS images of build output, packed inside the build sandbox (mkfs.erofs, erofsMkfsArgs in @in3pre3/sandbox);

  • upload manifests: JSON {"<path>": "<sha256>"} from the deno deploy CLI (Sources.putManifest).

New blobs are written to .tmp-<uuid>, hashed while streaming and renamed into place with mode 0444. Blobs from before the read-only mode are 0644. The directory is flat.

S3BlobStorage stores the same objects under <blobs.prefix><sha256>. Byte arrays are hashed in memory; streams are written to blob-cache/.tmp-<uuid> while hashing, uploaded (multipart above 64 MiB) unless the object already exists, and the file becomes the cache entry. Readers that need a file (blob mounts, the runtime /app image, template layers) go through local, which downloads into blob-cache/<sha256> once; FsBlobStorage returns its own file. Deleting a blob deletes the object and the cache entry. A missing object reads as Deno.errors.NotFound, as a missing file does.

Garbage collection (Sources.gc on sandboxod, every gc_interval_ms) keeps blobs reachable from any revision's source.tree, every build.image, every manifest.blob, the files listed in the manifests of revisions created less than a day ago that have no tree yet (uploads in progress), template build images and logs, uploaded template files, and the images of sandboxes that are not stopped, and deletes the rest once they are older than gc_grace_ms; stale .tmp-* files go too. Zones only add blobs (build and template images, template logs); their blob-cache/ copies of deleted blobs stay until the cache is cleared. Build sandboxes see blobs through hardlinks in their blob mount directory (below), so deleting a blob does not break a running build.

Git cache

git/<first 32 hex of sha256(url)>.git/ is a bare repository per source URL, created and fetched with gix (crates/gitclone). Fetches are shallow (depth 1), go to refs/sandboxo/fetched (+<ref or HEAD>:refs/sandboxo/fetched, no tags) and use only the repository's own config. Nothing is checked out; the fetched commit is imported into blobs/ and the revision records the tree. The cache only speeds up refetches and can be deleted at any time.

Nix

nix/<encodeURIComponent(flake ref)> are GC root symlinks to the built sandbox profiles (SANDBOXO_PROFILE, SANDBOXO_SANDBOX_PROFILE), so the host Nix garbage collector keeps them. nix/sandbox-store/ is a private Nix store (nix copy --to local?root=...) holding only the profiles' closures; its nix/store is bind-mounted read-only at /nix/store in every sandbox. The directory belongs to the zone and can be moved with gc_root_dir.

Slots

slots/ holds the runsc bundles of all live sandboxes of a zone and is deleted when the zone starts. Every bundle directory (sandbox-<16 hex>/) has:

sandbox-<id>/
  config.json                    OCI spec passed to runsc
  rootfs/                        root skeleton: /etc files, mountpoints
  state/                         runsc --root: <container>.state, .lock, runsc-<container>.sock
  slots/<name>/                  host side of bind-mounted slots without a hostDir
  blobfs/<n>/                    blob mount control dir (build sandboxes)
    blobfs.sock                  served by sandboxo, used by the blobfs gofer
    files/                       hardlinks of blobs opened in the sandbox
  vsock/                         mounted at /run/vsock (runtime sandboxes)
    2/4318                       host OTLP receiver (guest vsock connect to cid 2)
    2/4320                       host cron socket (DENO_UNSTABLE_CRON_SOCK)
    3/<port>                     guest vsock listeners (host connects here)
slots/none/, slots/sandboxed/

Warm runtime sandboxes per network flavor. /app is an empty mountpoint in the rootfs; on assignment the revision's build image is mounted there (runsc debug --mount).

slots/build/<8 hex>/

One directory per build sandbox: out/ is the read-write /out slot that receives app.erofs (kept on shutdown, then streamed into blobs/) next to the bundle. The read-write /app slot is the bundle's slots/app/. The bundle's blobfs/0/ backs the read-only /src mount that serves the revision's source tree.

slots/sandbox-<network>/

Generic sandboxes (e2b and Deno Sandbox APIs). Their rootfs has an FHS layout (/usr/local/bin, /home/...) and a sparse .gvisor.filestore.<container> file (1 GiB apparent size) backing the writable root overlay.

Generic sandboxes and volumes

kv/sandboxes/<id>.json

One record per generic sandbox, kept after it stops. Ids are sbx_<region>_<20 base32 chars>. Fields: id, api (e2b or deno), template, labels, network, memoryMb, cpuCount, volumes ([{volume, path}]), createdAt, stopAt, stoppedAt, pausedAt, autoPause, envdAccessToken, zone, profile and envdVersion (of the zone that runs it, kept so a checkpoint resumes with the same profile), stopReason, image ({build, hash, user, workdir} for template sandboxes), egress (policy without secret values) and the front-door state that has to survive a sandboxod restart: exposures ([{label, port}] of exposeHttp hosts, port is null while a label is reserved but not bound), sshUsers and vscode ({label, pid}).

kv/volumes/<id>.json

Volume record: id, slug, capacity, createdAt, zone, and baseSnapshot ({id, slug}) for volumes created from a snapshot. Ids are vol_<region>_<20 base32 chars>. capacity is not enforced.

kv/snapshots/<id>.json

Snapshot record: id, slug, volume ({id, slug} of the source volume), createdAt, zone. Ids are snp_<region>_<20 base32 chars>.

kv/zones/<name>.json

{"name", "token", "region"?, "cordoned"?, "url"?, "bootId"?, "registeredAt"?}, see Zones. The token is stored in plain text.

kv/templates/<id>.json, kv/template-builds/<id>/<build>.json

e2b templates: {id, name, createdAt, tags: {<tag>: {buildID, createdAt}}} and one record per build with status (waiting, building, ready, error), tags, cpuCount, memoryMB, createdAt, finishedAt, reason, zone, and once ready image ({hash, size} of the EROFS image in blob storage), config ({env, workdir, user}) and logBlob (the build log as JSON lines).

kv/sessions/<sha256(id)>.json

{sub, name, email, picture, idToken, createdAt, expiresAt}: a UI login. The cookie holds the random id, the key only its hash. The ID token is kept as the id_token_hint for logout. Expired sessions are dropped when read and swept on the blob GC interval.

kv/meta/legacy-org.json

The organization that owns records without an org field (written before multi-tenancy) and records of the single-org org. Startup moves the old slugs/, template-names/ and editor/user keys under it and refuses to start when oidc.legacy_org names a different organization than the stored one.

Ownership

App metas, sandbox, volume, snapshot and template records carry org, the owning organization id. Every API resolves ids and slugs within the caller's organization, so an id of another organization reads as not found.

kv/template-files/<id>/<files hash>.json

{"blob", "size", "createdAt"}: a COPY archive uploaded by the e2b SDK, stored in blob storage under its own hash.

kv/editors/<org>/<user>.json

{"settings.json"?, "keybindings.json"?}: the raw text (comments kept) of the VS Code user files of one user in one organization (admin for the admin token), at most 256 KiB each. Written into every VS Code that user opens from the UI and updated when one of them changes the files.

kv/org-slugs/<org>/<slug>.json

{"id"} of a volume or snapshot. Volumes and snapshots of an organization share one slug namespace, as in the Deno Sandbox API. The slug is claimed with create before anything else is written and released again if creation fails, so two objects never share a slug. Lookups by slug go through it.

volumes/<vol id>/, volumes/<snp id>/

BtrfsVolumeStorage keeps every volume as a btrfs subvolume (btrfs subvolume create), bind-mounted read-write into the sandboxes that attach it. A snapshot is a read-only snapshot of that subvolume (btrfs subvolume snapshot -r). A volume created from a snapshot is a writable snapshot of it, so it shares data copy-on-write with its base and outlives it. Snapshots of a volume in use are crash-consistent: sandbox volume mounts use runsc's default --file-access-mounts=shared, so gVisor writes through to the host and btrfs flushes pending writes when it takes the snapshot.

Deleting a volume or snapshot removes its record and slug, clears the read-only flag of a snapshot (a read-only subvolume cannot be moved), moves the subvolume into volumes/.trash/ and destroys it there: btrfs subvolume delete when permitted (root, or user_subvol_rm_allowed), otherwise a recursive delete, which works rootless. Startup destroys whatever is left in .trash/.

The btrfs binary (btrfs-progs, in the dev shell) must be on PATH. If volumes/ is not on btrfs the server logs a warning at startup and creating volumes or snapshots fails; everything else works.

Outside the state directory

$XDG_RUNTIME_DIR/sandboxo-<8 hex of sha256(zone state path)>/ (or /tmp/...) holds the sockets a zone shares with its sandboxes, because unix socket paths are limited to 108 bytes. It is recreated on startup and removed on shutdown:

  • <network>-<8 hex>/: runtime sandbox /run/sbx, with ctl.sock (Deno prewarm control) and http.sock (the app's HTTP server);

  • s-<8 hex>/: generic sandbox /run/sbx, with ctl.sock, agent.sock and p<port>.sock for exposed ports.

Build cgroups (build.cgroup_parent) and sandbox cgroups (sandbox.cgroup_parent) live in the cgroup hierarchy, not in the state directory.

Filesystem requirements

  • A zone's blob-cache/ (or the shared blobs/ with fs blobs) should be on the same filesystem as its slots/: blob mounts hardlink blobs into slots/build/.../blobfs/<n>/files/ and fall back to copying.

  • volumes/ must be on btrfs for volumes and snapshots. Smokes that use volumes keep their state in SANDBOXO_SMOKE_DIR (default ./.smoke/) instead of $TMPDIR, which is often tmpfs.

  • State directories from before blob storage may still contain uploads/; nothing reads it, and revisions without source.tree cannot be redeployed.

  • State directories from before the key-value store keep records in apps/, builds/, sandboxes/ and volumes/<id>.json; nothing reads them. Their volume directories are plain directories, not subvolumes, and cannot be snapshotted.

  • State directories from before zones keep volumes/, paused/, slots/ and nix/ at the top level, where no zone looks. Volume and sandbox records without zone cannot be placed; recreate them.