State directory
Persistent server state goes through three interfaces: records through a
key-value store (Kv in sandboxod/kv.ts), content-addressed blobs through
BlobStorage (common/blobs.ts) and volume data through VolumeStorage
(zoned/btrfs.ts). By default records live in Redis (RedisKv) and blobs in an
S3 bucket (S3BlobStorage, Garage in the dev shell and the NixOS module);
"kv": {"backend": "fs"} and "blobs": {"backend": "fs"} keep them as plain
files instead.
Records belong to sandboxod alone; zones never read Redis (see
Zones). Both programs have a state directory, state in their
config (see Configuration): sandboxod's (default ./state)
holds its git cache and SSH host key, plus records and blobs with the fs
backends. A zone's (default ./state/zone, state/zones/<name>/ with the
all-in-one dev.ts) holds everything that runs there: volumes as btrfs
subvolumes, paused checkpoints, sandbox bundles and a local cache of S3 blobs.
With fs blobs every zone must see sandboxod's blobs/ directory
(blobs.dir), so fs only works on one host. Logs are not stored here: app,
build and sandbox logs go to Loki only, see
Observability.
kv is sandboxod's only; blobs is required in both programs and must point
at the same storage:
| Key | Default | Meaning |
|---|---|---|
|
|
|
|
|
secret, |
|
|
prefix of every Redis key, so servers can share a database |
|
required |
|
|
|
blob directory for |
|
S3 endpoint URL, path-style requests |
|
|
|
signing region |
|
|
bucket, must exist |
|
required for |
secrets, credentials with read, write and delete on the bucket |
|
empty |
prefix of every object key |
|
|
parallel S3 requests |
Two servers must not share a Redis prefix or an S3 prefix: garbage collection deletes every blob under its prefix that its own records do not reference.
Layout
sandboxod:
state/
kv/ fs kv only: records, see Key-value store
apps/<app>/
config.json app config
meta.json app id and timestamps
current.json promoted revision id
playground.json playgrounds only: code volume and creation time
env/<NAME>.json one environment variable
revisions/<rev>.json revision record
status/<rev>.json build status
crons/<name>.json one cron job's recent runs (name base64url)
sandboxes/<sandbox id>.json generic sandbox record
volumes/<volume id>.json volume record
snapshots/<snapshot id>.json snapshot record
org-slugs/<org>/<slug>.json volume and snapshot slug index, per organization
templates/<template id>.json e2b template record
template-builds/<template id>/<build id>.json template build record
template-files/<template id>/<files hash>.json uploaded COPY archive
org-template-names/<org>/<name>.json template name index, per organization
zones/<name>.json zone record and token
routers/<name>.json router, its tunnel address and when it last fetched routes
editors/<org>/<user>.json VS Code user settings, synced from editors
sessions/<sha256(id)>.json UI login session (oidc only)
meta/legacy-org.json organization that owns pre-tenancy records
.trash/ subtrees being deleted
blobs/ fs blobs only, shared with the zones
<sha256> file contents, tree objects, build and template images, manifests, logs
.tmp-<uuid> blob being written
blob-cache/ s3 blobs only
.tmp-<uuid> upload being hashed
git/
<hash(url)>.git/ bare repository cache (gix)
ssh_host_ed25519_key SSH gateway host key
zoned:
state/zone/
blob-cache/ s3 blobs only
<sha256> local copy of a blob opened as a file
.tmp-<uuid> blob being hashed before upload
<sha256>.<uuid>.part blob being downloaded
volumes/ btrfs only
<volume id>/ volume data, a subvolume
<snapshot id>/ read-only snapshot of a volume
.trash/ subvolumes being deleted
paused/
<sandbox id>/ runsc checkpoint of a paused sandbox
<sandbox id>.json its spec and creation time
egress/
ca/ iron-proxy CA
sandboxes/<sandbox id>/ proxy config and secrets of one sandbox
templates/
layers/ registry layer cache
bases/ unpacked base images
work/ template builds in progress, wiped on startup
nix/
<encoded flake ref> GC root symlink to a sandbox profile
sandbox-store/nix/store/ private Nix store, mounted at /nix/store
slots/ sandbox bundles, wiped on startup
none/sandbox-<id>/ warm runtime sandboxes, network none
sandboxed/sandbox-<id>/ warm runtime sandboxes, network sandboxed
build/<id>/ build sandboxes
sandbox-<network>/sandbox-<id>/ generic sandboxes (e2b, Deno Sandbox)
volume-tar/ sandboxes packing volume archives
templates/ template build sandboxes
tmp/
tools/
inspect.js repository inspection, run in build sandboxes
agent.js generic sandbox agent
envoy/ the zone's obice: bootstrap.json (holds the zone token), obice.log, envoy.log, tunnels.log
| Path | Lifetime | Created by |
|---|---|---|
|
persistent (sandboxod) |
|
|
persistent (zone) |
|
|
cache, safe to delete while the program is stopped |
|
|
cache, safe to delete |
|
|
cache, rebuilt on startup |
|
|
cache, |
|
|
ephemeral, removed on every zone startup |
sandbox pools ( |
|
regenerated on every startup |
|
|
persistent, created once (mode 0600) |
SSH gateway ( |
Key-value store
Kv (sandboxod/kv.ts) stores JSON values under keys that are lists of segments
matching [A-Za-z0-9_-]+. Store (sandboxod/store.ts) is the only user; it validates
records with zod on read. Operations: get, set, create (only if absent),
update (read-modify-write), delete, deleteTree (a key and everything
below it) and list (the immediate children of a prefix, sorted). Keys with
other segments read as missing; writing them throws.
FsKv maps key [a, b, c] to the file kv/a/b/c.json and its children to
the directory kv/a/b/c/, so a key can hold a value and have children.
-
setwrites.tmp-<uuid>next to the target (mode 0600) and renames it into place. -
createwrites the temp file and hardlinks it to the target, which fails atomically if the target exists. -
updateruns one at a time per key within the process. -
deleteTreerenames the subtree intokv/.trash/<uuid>(atomic), deletes it there, then deletes the key's own value. -
On startup
FsKv.initempties.trash/and removes stray.tmp-*files.
RedisKv keeps the value of [a, b, c] as a JSON string under
<prefix>v:a/b/c and the children of every prefix in a set, <prefix>i:a/b
holding c, <prefix>i:a holding b and <prefix>i: holding a.
-
set,createand the final write ofupdateare one Lua script each, so the value and the parent sets change atomically.createisSETonly if absent. -
updatereads the value, runs the function and writes only if the value is still the one it read (compare-and-set in the script), retrying on conflict, so it is atomic across processes. The function may run more than once. -
deleteremoves the value and, when the key has no children, its name from the parent set, in one script. -
deleteTreewalks the sets below the prefix and deletes every value and set it finds; it is not atomic. -
listreads the prefix's set. A name stays listed while it has children, like a directory inFsKv.
Apps
kv/apps/<app>/ exists for every app; <app> is a lowercase DNS label of at most
32 characters, so preview hosts <app>--<rev> stay within the 63-character
label limit. It starts with a letter, so it never looks like a sandbox port host
(<port>-sbx-<region>-<20>); ssh and names shaped like exposeHttp labels
(x-<region>-<20>) are reserved.
config.json-
The app config. Fields:
entrypoint,args,installCommand,buildCommand,preDeployCommand,buildDirectory,network(noneorsandboxed),buildTimeout(minutes ornull),buildMemoryLimit(MiB ornull),region(3-letter region code orglobal) andmaxReplicas(1 to 16), see Scaling. Missing fields take their defaults (appConfigincommon/appconfig.ts), so older files stay readable. The environment is not stored here:readConfigfillsenvfromenv/andwriteConfigdrops it. env/<NAME>.json-
One environment variable per key, so every change is a single atomic write:
{"value", "secret", "local", "updatedAt"}.<NAME>matches[A-Za-z_][A-Za-z0-9_]*.localputs the variable in the Local context (context_ids: nullin the deno deploy API, otherwise production and preview). Asecretvariable is write-only: its value is never sent to a browser, the admin API or the deno deploy CLI, it cannot become non-secret again, itslocalflag is fixed, and saving it with an empty value keeps the sealed one. Values are stored as plain text; builds and instances get all variables, and revision records keep a snapshot of them. meta.json-
{"id", "createdAt", "updatedAt"}: a UUID used as the app id by the deno deploy API, and ISO timestamps.updatedAtchanges on every config write. current.json-
The promoted revision id as a JSON string. Promotion is a single
set, so the switch is atomic. Missing until the first promotion. playground.json-
Playgrounds only:
{"volume", "createdAt"}, the volume holding the code.
Builds and revisions
A revision id is <milliseconds in base 36>-<8 hex> (for example
mvfobbyg-21e28c6c), so ids sort by creation time. Both live under
kv/apps/<app>/; revisions are listed by their status/ keys.
status/<rev>.json-
{"status", "updatedAt"}:statusisuploading,queued,building,okorfailed,updatedAtthe ISO time of the last change. revisions/<rev>.json-
The revision record:
{ "prod": true, "createdAt": "2026-10-19T19:54:54.664Z", "config": { "entrypoint": "...", "...": "..." }, "source": { "tree": "<sha256 of the root tree>" }, "git": { "url": "https://...", "ref": "main", "commit": "<sha1>" }, "manifest": { "blob": "<sha256>", "files": 42 }, "detect": { "install": true, "build": true, "runtime": true }, "build": { "image": "<sha256>", "size": 80928768, "nodeModules": true }, "crons": [{ "name": "cleanup", "schedule": "0 * * * *", "backoffSchedule": [1000] }] }configis a snapshot of the app config (env included) taken when the revision was created; later config changes apply to the next revision.source.treeis the root of the source tree inblobs/; it is set once the upload completes or the git fetch finishes, and a redeploy copies it.gitis set for quick-deploy revisions (refonly when given,commitafter the fetch).manifestpoints to the deno deploy upload manifest, stored inblobs/because it can list hundreds of thousands of files, and keeps its file count. It stays after the upload.detectmarks the settings to fill from repository inspection.buildis set by a successful build:imageis the build output as an EROFS image inblobs/(the source copy plus everything the build produced:node_modules,.deno_dirwith the module cache, framework output such as.next), mounted read-only at/appin runtime sandboxes;nodeModulesadds--node-modules-dir=manualto the start line.cronslists theDeno.cron()jobs the revision registered when the build booted it once (the warmup); a revision without the field was built before cron support and runs no crons. Failed builds keep no output. crons/<name>.json-
{"name", "runs"}for one cron job, keyed by the base64url of its name (cron names may contain spaces).runsholds the lastcron.historyruns, oldest first:{ "id": "mvhngj9o-1a2b3c4d", "schedule": "0 * * * *", "rev": "mvhngbru-139140f2", "attempt": 1, "scheduledAt": "2026-10-21T05:00:00.000Z", "startedAt": "2026-10-21T05:00:00.012Z", "finishedAt": "2026-10-21T05:00:00.140Z", "status": "ok", "traceId": "<32 hex>", "zone": "ord1" }statusisrunning,okorerror(witherror); a retry is a new run withattempt+ 1.
Blob storage
blobs/ holds content-addressed objects named by the hex sha256 of their
bytes (FsBlobStorage in common/blobs.ts):
-
file blobs: raw file contents, the same hashes the deno deploy CLI uses in upload manifests;
-
tree blobs: one directory each, canonical JSON
{"tree":1,"entries":[...]}sorted by name, with entries{name, type: "file", mode, size, hash},{name, type: "dir", hash}or{name, type: "symlink", target}(common/tree.ts); -
build images: uncompressed EROFS images of build output, packed inside the build sandbox (
mkfs.erofs,erofsMkfsArgsin@in3pre3/sandbox); -
upload manifests: JSON
{"<path>": "<sha256>"}from the deno deploy CLI (Sources.putManifest).
New blobs are written to .tmp-<uuid>, hashed while streaming and renamed into
place with mode 0444. Blobs from before the read-only mode are 0644. The
directory is flat.
S3BlobStorage stores the same objects under <blobs.prefix><sha256>.
Byte arrays are hashed in memory; streams are written to
blob-cache/.tmp-<uuid> while hashing, uploaded (multipart above 64 MiB) unless
the object already exists, and the file becomes the cache entry. Readers that
need a file (blob mounts, the runtime /app image, template layers) go through
local, which downloads into blob-cache/<sha256> once; FsBlobStorage
returns its own file. Deleting a blob deletes the object and the cache entry.
A missing object reads as Deno.errors.NotFound, as a missing file does.
Garbage collection (Sources.gc on sandboxod, every gc_interval_ms)
keeps blobs reachable from any revision's source.tree, every build.image,
every manifest.blob, the files listed in the manifests of revisions created
less than a day ago that have no tree yet (uploads in progress), template build
images and logs, uploaded template files, and the images of sandboxes that are
not stopped, and deletes the rest once they are older than
gc_grace_ms; stale .tmp-* files go too. Zones only add blobs
(build and template images, template logs); their blob-cache/ copies of
deleted blobs stay until the cache is cleared. Build sandboxes see blobs through hardlinks in their
blob mount directory (below), so deleting a blob does not break a running
build.
Git cache
git/<first 32 hex of sha256(url)>.git/ is a bare repository per source URL,
created and fetched with gix (crates/gitclone). Fetches are shallow (depth
1), go to refs/sandboxo/fetched (+<ref or HEAD>:refs/sandboxo/fetched, no
tags) and use only the repository's own config. Nothing is checked out; the
fetched commit is imported into blobs/ and the revision records the tree.
The cache only speeds up refetches and can be deleted at any time.
Nix
nix/<encodeURIComponent(flake ref)> are GC root symlinks to the built sandbox
profiles (SANDBOXO_PROFILE, SANDBOXO_SANDBOX_PROFILE), so the host Nix
garbage collector keeps them. nix/sandbox-store/ is a private Nix store
(nix copy --to local?root=...) holding only the profiles' closures; its
nix/store is bind-mounted read-only at /nix/store in every sandbox. The
directory belongs to the zone and can be moved with gc_root_dir.
Slots
slots/ holds the runsc bundles of all live sandboxes of a zone and is deleted
when the zone starts. Every bundle directory (sandbox-<16 hex>/) has:
sandbox-<id>/
config.json OCI spec passed to runsc
rootfs/ root skeleton: /etc files, mountpoints
state/ runsc --root: <container>.state, .lock, runsc-<container>.sock
slots/<name>/ host side of bind-mounted slots without a hostDir
blobfs/<n>/ blob mount control dir (build sandboxes)
blobfs.sock served by sandboxo, used by the blobfs gofer
files/ hardlinks of blobs opened in the sandbox
vsock/ mounted at /run/vsock (runtime sandboxes)
2/4318 host OTLP receiver (guest vsock connect to cid 2)
2/4320 host cron socket (DENO_UNSTABLE_CRON_SOCK)
3/<port> guest vsock listeners (host connects here)
slots/none/,slots/sandboxed/-
Warm runtime sandboxes per network flavor.
/appis an empty mountpoint in the rootfs; on assignment the revision's build image is mounted there (runsc debug --mount). slots/build/<8 hex>/-
One directory per build sandbox:
out/is the read-write/outslot that receivesapp.erofs(kept on shutdown, then streamed intoblobs/) next to the bundle. The read-write/appslot is the bundle'sslots/app/. The bundle'sblobfs/0/backs the read-only/srcmount that serves the revision's source tree. slots/sandbox-<network>/-
Generic sandboxes (e2b and Deno Sandbox APIs). Their rootfs has an FHS layout (
/usr/local/bin,/home/...) and a sparse.gvisor.filestore.<container>file (1 GiB apparent size) backing the writable root overlay.
Generic sandboxes and volumes
kv/sandboxes/<id>.json-
One record per generic sandbox, kept after it stops. Ids are
sbx_<region>_<20 base32 chars>. Fields:id,api(e2bordeno),template,labels,network,memoryMb,cpuCount,volumes([{volume, path}]),createdAt,stopAt,stoppedAt,pausedAt,autoPause,envdAccessToken,zone,profileandenvdVersion(of the zone that runs it, kept so a checkpoint resumes with the same profile),stopReason,image({build, hash, user, workdir}for template sandboxes),egress(policy without secret values) and the front-door state that has to survive a sandboxod restart:exposures([{label, port}]ofexposeHttphosts,portisnullwhile a label is reserved but not bound),sshUsersandvscode({label, pid}). kv/volumes/<id>.json-
Volume record:
id,slug,capacity,createdAt,zone, andbaseSnapshot({id, slug}) for volumes created from a snapshot. Ids arevol_<region>_<20 base32 chars>.capacityis not enforced. kv/snapshots/<id>.json-
Snapshot record:
id,slug,volume({id, slug}of the source volume),createdAt,zone. Ids aresnp_<region>_<20 base32 chars>. kv/zones/<name>.json-
{"name", "token", "region"?, "cordoned"?, "url"?, "bootId"?, "registeredAt"?}, see Zones. The token is stored in plain text. kv/templates/<id>.json,kv/template-builds/<id>/<build>.json-
e2b templates:
{id, name, createdAt, tags: {<tag>: {buildID, createdAt}}}and one record per build withstatus(waiting,building,ready,error),tags,cpuCount,memoryMB,createdAt,finishedAt,reason,zone, and once readyimage({hash, size}of the EROFS image in blob storage),config({env, workdir, user}) andlogBlob(the build log as JSON lines). kv/sessions/<sha256(id)>.json-
{sub, name, email, picture, idToken, createdAt, expiresAt}: a UI login. The cookie holds the random id, the key only its hash. The ID token is kept as theid_token_hintfor logout. Expired sessions are dropped when read and swept on the blob GC interval. kv/meta/legacy-org.json-
The organization that owns records without an
orgfield (written before multi-tenancy) and records of the single-orgorg. Startup moves the oldslugs/,template-names/andeditor/userkeys under it and refuses to start whenoidc.legacy_orgnames a different organization than the stored one. - Ownership
-
App metas, sandbox, volume, snapshot and template records carry
org, the owning organization id. Every API resolves ids and slugs within the caller's organization, so an id of another organization reads as not found. kv/template-files/<id>/<files hash>.json-
{"blob", "size", "createdAt"}: aCOPYarchive uploaded by the e2b SDK, stored in blob storage under its own hash. kv/editors/<org>/<user>.json-
{"settings.json"?, "keybindings.json"?}: the raw text (comments kept) of the VS Code user files of one user in one organization (adminfor the admin token), at most 256 KiB each. Written into every VS Code that user opens from the UI and updated when one of them changes the files. kv/org-slugs/<org>/<slug>.json-
{"id"}of a volume or snapshot. Volumes and snapshots of an organization share one slug namespace, as in the Deno Sandbox API. The slug is claimed withcreatebefore anything else is written and released again if creation fails, so two objects never share a slug. Lookups by slug go through it. volumes/<vol id>/,volumes/<snp id>/-
BtrfsVolumeStoragekeeps every volume as a btrfs subvolume (btrfs subvolume create), bind-mounted read-write into the sandboxes that attach it. A snapshot is a read-only snapshot of that subvolume (btrfs subvolume snapshot -r). A volume created from a snapshot is a writable snapshot of it, so it shares data copy-on-write with its base and outlives it. Snapshots of a volume in use are crash-consistent: sandbox volume mounts use runsc's default--file-access-mounts=shared, so gVisor writes through to the host and btrfs flushes pending writes when it takes the snapshot.Deleting a volume or snapshot removes its record and slug, clears the read-only flag of a snapshot (a read-only subvolume cannot be moved), moves the subvolume into
volumes/.trash/and destroys it there:btrfs subvolume deletewhen permitted (root, oruser_subvol_rm_allowed), otherwise a recursive delete, which works rootless. Startup destroys whatever is left in.trash/.The
btrfsbinary (btrfs-progs, in the dev shell) must be onPATH. Ifvolumes/is not on btrfs the server logs a warning at startup and creating volumes or snapshots fails; everything else works.
Outside the state directory
$XDG_RUNTIME_DIR/sandboxo-<8 hex of sha256(zone state path)>/ (or
/tmp/...) holds the sockets a zone shares with its sandboxes, because unix
socket paths are limited to 108 bytes. It is recreated on startup and removed
on shutdown:
-
<network>-<8 hex>/: runtime sandbox/run/sbx, withctl.sock(Deno prewarm control) andhttp.sock(the app's HTTP server); -
s-<8 hex>/: generic sandbox/run/sbx, withctl.sock,agent.sockandp<port>.sockfor exposed ports.
Build cgroups (build.cgroup_parent) and sandbox cgroups
(sandbox.cgroup_parent) live in the cgroup hierarchy, not in the state
directory.
Filesystem requirements
-
A zone's
blob-cache/(or the sharedblobs/withfsblobs) should be on the same filesystem as itsslots/: blob mounts hardlink blobs intoslots/build/.../blobfs/<n>/files/and fall back to copying. -
volumes/must be on btrfs for volumes and snapshots. Smokes that use volumes keep their state inSANDBOXO_SMOKE_DIR(default./.smoke/) instead of$TMPDIR, which is often tmpfs. -
State directories from before blob storage may still contain
uploads/; nothing reads it, and revisions withoutsource.treecannot be redeployed. -
State directories from before the key-value store keep records in
apps/,builds/,sandboxes/andvolumes/<id>.json; nothing reads them. Their volume directories are plain directories, not subvolumes, and cannot be snapshotted. -
State directories from before zones keep
volumes/,paused/,slots/andnix/at the top level, where no zone looks. Volume and sandbox records withoutzonecannot be placed; recreate them.