Configuration

sandboxod, zoned and the all-in-one dev.ts each read one JSON file given with --config <file>. Keys are snake_case and unknown keys are errors. Every key not marked required has a default, so a minimal sandboxod config is:

{
  "token": { "_env": "SANDBOXO_TOKEN" },
  "signing_secret": { "_file": "/run/keys/sandboxo-signing" },
  "blobs": { "backend": "fs" }
}

Errors name keys in camelCase (build.timeoutMs for build.timeout_ms). Relative paths resolve against the working directory.

Secrets

Keys marked secret take a plain string or a placeholder, the same ones obice uses:

{ "_env": "NAME" }

the value of the environment variable NAME.

{ "_file": "/path" }

the content of the file, without trailing newlines.

A missing variable or an unreadable file fails startup. Secret keys: sandboxod's token, signing_secret, the values of zone_tokens and kv.url, zoned's token, and in both blobs.access_key_id, blobs.secret_access_key and the values of every headers in telemetry.

Environment

Only the Nix profiles come from the environment, and binaries from PATH. The zoned and sandboxo packages set both in their wrappers; the dev shell has them for deno run. zoned and dev.ts fail at startup when one is missing.

Variable Meaning

SANDBOXO_PROFILE

required, Nix profile of build and runtime sandboxes

SANDBOXO_SANDBOX_PROFILE

required, Nix profile of generic sandboxes (e2b, Deno Sandbox)

zoned needs on PATH: sbx-runsc (sbx-gvisor), pasta (passt), mkfs.erofs (erofs-utils), iron-proxy for sandbox egress policies, obice for the zone's tunnels (its envoy comes from obice's own PATH), btrfs, unshare and nsenter.

sandboxod

Key Default Meaning

token

required, secret

admin token for the API, the UI and /zones, characters [A-Za-z0-9._~-] only; with oidc set it stays a break-glass superadmin

signing_secret

required, secret

signs UI query links and upload links, see Observability

blobs

required

blob storage, see State directory

kv

Redis on 127.0.0.1:6379

record storage, see State directory

telemetry

none

Loki, Mimir, Tempo and OTLP, see Observability

state

state

state directory, see State directory

org

local

deployment id ([A-Za-z0-9_-]): zones must use the same, it names the shared telemetry tenant and cannot be used as an app name. Without oidc it is also the one organization

domain

localhost

apps are served on <app>.<domain>

port

443

https port of the routers, used in app and sandbox URLs

api_hostname, api_port

127.0.0.1, 8001

API and UI, also where zones register and routers fetch their routes

egress

/run/sandboxo-egress/egress.sock

unix socket of the egress listener of the router on this host, sandboxod reaches the zones through it

ui

production

production (needs ui/dist), development (bundles and hot-reloads the UI) or off

region

ord

default region for new sandboxes, volumes and apps

zone_tokens

{}

zone name to secret, written into the zone records at startup, see Zones

sandbox.max_lifetime_ms

1800000

longest lifetime of a generic sandbox

sandbox.template_build_timeout_ms

3600000

e2b template build timeout

sandbox.ssh

{}

SSH gateway, null turns it off

sandbox.ssh.hostname, sandbox.ssh.port

::, 2222

SSH gateway listen address

sandbox.ssh.public_host

ssh.<domain>

host the SSH commands name

scaling.target_inflight

20

in-flight requests per replica above which an app scales up, see Scaling

scaling.scale_down_ms

60000

how long the load has to stay low before a replica is removed

build.timeout_ms

600000

build timeout of apps that set none

build.max_timeout_ms

1800000

largest build timeout an app may set, at least build.timeout_ms

build.memory_mb

1024

build memory limit of apps that set none, null for no limit

build.max_memory_mb

4096

largest build memory limit an app may set

cron.max

10

cron jobs a revision may register, more fail the build

cron.timeout_ms

900000

a cron run that takes longer fails and its instance is replaced

cron.history

100

runs kept per cron job

git_cache

<state>/git

bare repository cache for git deploys

git_fetch_timeout_ms

300000

git fetch timeout

source_max_files, source_max_bytes

200000, 2147483648

limits of one upload or git tree

gc_interval_ms, gc_grace_ms

3600000, 3600000

blob garbage collection interval (0 disables it) and the age a blob needs before it is deleted; expired login sessions are swept on the same interval

oidc

none

multi-tenant mode with users and organizations from an OpenID provider (pykrete), see OpenID Connect

OpenID Connect

With oidc set, users log in through the provider and every organization they belong to there is a tenant: its members share its apps, sandboxes, volumes, snapshots and templates. API keys are minted, listed and revoked through the provider's backend API (a Clerk-compatible subset) and checked with RFC 7662 introspection. telemetry.tenant must be app, and redirect_base must not be under domain (apps would receive the session cookie).

Key Default Meaning

oidc.issuer

required

issuer URL; discovery must report the same issuer

oidc.client_id, oidc.client_secret

required, secret

confidential client; the secret is also the backend API key. The client needs "Backend API" (set by the client owner) and "All organizations" (first-party, set by a pykrete admin), otherwise every organization has to enable the client and only users who authorized it are visible

oidc.token_auth_method

client_secret_post

or client_secret_basic

oidc.redirect_base

required

public origin of the UI; the redirect URI is <redirect_base>/login/callback and the post-logout URI <redirect_base>/login

oidc.api

<issuer>api/v1/

backend API base URL

oidc.legacy_org

org

provider organization id that owns records created before multi-tenancy (and in single-org mode); it cannot change once set

oidc.scopes

openid profile email organizations

requested scopes

oidc.session_ttl_ms

604800000

lifetime of a UI login

oidc.introspect_ttl_ms, oidc.membership_ttl_ms

60000, 30000

how long API key checks and organization memberships are cached, so how late a revoked key or a removed member is noticed

oidc.admin_login

false

also offer the admin token form on /login

oidc.org_url, oidc.account_url

none

links to the provider's organization page ({slug}, {id} are replaced) and account page, for members, invitations and API keys

zoned

Key Default Meaning

name

required

zone name, a DNS label of at most 32 characters

token

required, secret

zone token, see Zones

blobs

required

blob storage, the same as sandboxod's, see State directory

telemetry

none

where logs, traces, metrics and usage are pushed, see Observability

region

ord

region of the zone

org

local

must match sandboxod's

hostname, port

127.0.0.1, 8002

zone API listen address, only the zone's envoy connects

url

the listen address

zone URL reported to sandboxod

control

http://127.0.0.1:8001

sandboxod API listener

state

state/zone

zone state directory, see State directory

gc_root_dir

<state>/nix

GC roots and private store of the profiles

pool.none, pool.sandboxed

20, 4

warm runtime sandboxes per network flavor

pool.build

2

warm build sandboxes

build.concurrency

2

builds the zone runs at once

build.memory_mb

1024

build memory limit of apps that set none, null for no limit

build.memory_overhead_mb

1024

added to the app's limit for the build sandbox's cgroup

build.cpus

none

cpu.max of build cgroups, in cores

build.cgroup_parent, runtime_cgroup_parent, sandbox.cgroup_parent

auto

cgroup parents of build, runtime and generic sandboxes: a path, auto or null (no cgroups). auto uses the cgroup of zoned's unit when it is delegated (Delegate=yes, as the NixOS module sets), otherwise one systemd-run --user --scope unit per sandbox under app.slice. Without a systemd user manager auto fails: the zone logs a warning and runs without cgroups. Limits and CPU usage need them

start_timeout_ms

10000

how long a replica may take to start

idle_ms

300000

idle replicas stop after this long without requests

drain_ms

30000

how long a replica that is scaled down or replaced finishes its requests

sandbox.pool_size

2

warm generic sandboxes

sandbox.max

10

generic sandboxes the zone runs at once

dev.ts

dev.ts starts sandboxod and its zones in one process. Its config has three keys:

sandboxod

a sandboxod config.

zoned

zoned keys shared by every zone, except the ones dev.ts sets itself: name, region, token, org, hostname, port, url, control, state, blobs and telemetry.

zones

[{"name", "region"?}], default <region>1 and <region>2 of sandboxod's region.

Every zone gets a random token, a free loopback port, sandboxod's org, telemetry and blob storage (fs blobs in <state>/blobs), its state in <state>/zones/<name>/ and gc_root_dir <state>/nix unless set. The dev shell exports SANDBOXO_DEV_CONFIG (and SANDBOXO_DEV_UI_CONFIG with ui: development), generated from flake.nix, which points at the process-compose backends and reads the token from SANDBOXO_TOKEN:

SANDBOXO_TOKEN=secret deno task dev

NixOS

services.sandboxod.settings and services.zoned.settings are written to the config file. The modules set the keys their options cover (domain, name, ports, state directory, ...), the tokens as _file placeholders on systemd credentials, and for the local Redis and Garage of services.sandboxod kv and blobs with _env placeholders. Secrets for other placeholders go in environmentFiles:

services.zoned = {
  enable = true;
  name = "ord1";
  tokenFile = "/run/keys/zone-ord1";
  controlUrl = "http://10.0.0.1:8001";
  settings = {
    pool.none = 4;
    blobs = {
      backend = "s3";
      endpoint = "http://10.0.0.1:3900";
      access_key_id._env = "SANDBOXO_S3_ACCESS_KEY_ID";
      secret_access_key._env = "SANDBOXO_S3_SECRET_ACCESS_KEY";
    };
  };
  environmentFiles = [ "/run/keys/zone-s3.env" ];
};