Jails, done the ZFS way

• A project is a logical namespace. Networks and storage are separate, host-managed resources that a project is granted, optionally as a sub-range with a quota. A project also sets quotas for jails, CPU, memory, PIDs and addresses.
• Identities are pinned client certificates, API tokens and users of the web console. Each has a role (readonly, operator or admin) and a project scope. An empty project list never means access to everything; global access is an explicit flag that only a global admin can grant.
• Trust is built without trust-on-first-use: an admin creates a single-use trust token bound to a name, with an expiry, and the new client pins the server fingerprint from it.
• Every route has a permission entry, or the daemon refuses to start. State changes are audited. Dangerous parameters (raw jail parameters, host mounts, raw PF) need the admin role.
• A separate dataset is not a full security boundary. Jails share the host kernel, so a high-security tenant needs a dedicated pool, a dedicated host, or both.

# on the server
$ jailops trust add --name laptop --role operator --projects web

# on the client, with the token it printed
$ jailops remote add host.example --token <token>
$ jailops project switch web
$ jailops whoami

• An image is a line of immutable generations built from pkgbase. Each generation is a dataset with a read-only snapshot that never changes.
• A thin jail is a ZFS clone of one generation. What must survive an update (/etc, /usr/local, selected /var paths, /home, /root, volumes) lives in separate datasets mounted under the root. A thick layout, a full independent copy, is also available.
• Upgrade takes one atomic snapshot of the jail, clones the new generation, merges /etc three ways, swaps the roots with zfs rename, health-checks, and swaps back on failure. An unmergeable conflict leaves the jail on the old generation.
• Rollback has scopes. root swaps the OS back and keeps your data. data and all discard what was written since, show how many bytes are at stake, and need the jail name typed to confirm.
• Storage is a named ZFS location on a host, with quotas per project and per jail. Each pool that holds thin jails keeps its own copy of the generations it uses.

$ jailops storage create fast --dataset tank/jailops
$ jailops image build freebsd-15
$ jailops jail upgrade web --dry-run
$ jailops jail upgrade web
$ jailops jail rollback web --data --snapshot before-upgrade
# shows the bytes that would be discarded, asks for the jail name

On FreeBSD 15.1 a prototype built a pkgbase userland into an empty dataset, started and health-checked a thin jail cloned from it, upgraded it with etcupdate, swapped roots, and showed that a root rollback keeps data written after the upgrade. One question, how /var/db/pkg behaves after a root swap, is still open.

• Every jail uses VNET and an epair on a bridge that JailOps created. Shared-IP and host-networked jails are not supported. JailOps never modifies a bridge it did not create.
• A network has a subnet, a gateway that is a real address on the bridge, and an explicit allocation range. JailOps allocates from the range and writes the address into the jail spec.
• No DHCP, no router advertisements, no SLAAC. accept_rtadv is disabled in the jail. Ordinary IPv6 neighbor discovery still works because it is part of IPv6, not a service.
• IPv6 routed mode, where the upstream routes a prefix to the host, must work fully without any proxy. A routed prefix can be carved into /80 networks, and unallocated space gets a reject route so packets do not loop.
• The built-in IPv6 NDP proxy for switched /64 networks is experimental, off by default, and not yet proven on FreeBSD.
• PF only, one anchor per jail. A ruleset is rendered, checked with pfctl -n, loaded in one operation and read back. JailOps never owns pf.conf; you add three anchor lines once.

$ jailops network create dmz --subnet 192.0.2.0/24 \
    --gateway 192.0.2.1 --range 192.0.2.10-192.0.2.40
$ jailops network leases dmz
$ jailops firewall address-set create office \
    --address 192.0.2.0/24 --address 198.51.100.7
$ jailops firewall policy create web --default-in deny \
    --default-out allow --rule allow in tcp port 80,443

On FreeBSD 15.1, anchors load, replace and flush without touching the administrator's ruleset, and a reload is atomic. On the arm64 test host, PF between projects on a shared bridge and per-jail source-address enforcement worked, with nested anchors needing anchor "*" in every parent. Ethernet-level protection (ARP, NDP, MAC spoofing) is not covered and remains open.

• The host does the cryptography. It cuts data into content-defined chunks, compresses them with zstd, and encrypts them with XChaCha20-Poly1305. The chunk id is a keyed HMAC, so the server cannot even confirm that it holds known data.
• The backup node, jailops-backupd, is keyless. It stores chunks on a directory or an S3 bucket, verifies them by trailer without any key, and runs retention, garbage collection and quarantine. It links no jail, ZFS or decryption code, and a test enforces that.
• Every snapshot is self-contained: a list of references to chunks. There are no incremental chains to break, and only new chunks are uploaded.
• The key is a file on the host. A recovery key is shown once, and the master key can be wrapped to offline recovery recipients, so a restore on a new host needs only a recovery key.
• The server sees object counts and sizes, the deduplication graph, snapshot times, and host and jail identity. It never sees file names, contents or keys.
• This is not ZFS send and not ZFS native encryption. ZFS snapshots only give the backup a consistent view. Replication between trusted hosts is a separate feature, below.

$ jailops backup target create nightly --kind node \
    --address https://b.example:8443 --fingerprint <sha256>
$ jailops backup policy create nightly --selector env=prod \
    --target nightly --schedule "0 2 * * *" --keep-daily 14 --keep-weekly 8
$ jailops backup list
$ jailops backup verify <backup>

The daemon is not root: file agents of the privileged helper, confined by chroot to one checked directory, read the snapshot and write the restored jail. On the arm64 test host a jail was backed up to a node, verified at three depths and restored as a new jail. Snapshot metadata matched live stat, and inode and ctime were stable across snapshots; zfs diff misses hard-link name changes, so capture rescans those.

• ZFS send and receive between hosts that trust each other, built on incremental chains with guid continuity, bookmarks, holds and resume tokens, and never recv -F over existing snapshots.
• On FreeBSD 15.1 a prototype confirmed the ZFS behaviours this relies on, and found that an incremental on top of a receiver with a newer snapshot is accepted without -F. JailOps therefore enforces guid continuity itself instead of trusting recv to refuse.
• A compromised source can corrupt a replica, which is exactly why backup is a separate, keyless, append-only repository. The replication code exists as a library and is not yet exposed through the API or CLI.

• A fleet is a git repository of YAML: hosts, host groups, jail templates, networks, policies and rollouts. Secrets are encrypted with age.
• jailops-sync plans before it applies. It writes only objects it owns (marked managed_by: sync:<fleet-id>), never changes or deletes anything else, and deleting even its own objects needs an explicit --prune. Every apply writes a JSONL log.
• Hand edits to an object that sync owns are reported as drift and reverted on apply. adopt writes live objects into the repository as YAML, and prune-check lists what is live but undeclared, without deleting.
• Rollouts run staged upgrades with a canary, a batch size, a pause and a health gate.
• A hub is a mode of jailopsd. It keeps a registry of hosts, authenticates the caller and forwards requests over pinned mTLS (jailops --host host-b jail list). If the hub is down, hosts keep working. There is no shared storage and no live migration.
• It is built only on the SDK, and a test enforces that it imports nothing from the daemon. There is no scheduler: a jail is declared on a named host.

$ jailops-sync validate
$ jailops-sync plan
$ jailops-sync drift
$ jailops-sync sync --apply
$ jailops-sync rollout list

# rollouts/web-upgrade.yaml
apiVersion: jailops/v1
kind: Rollout
metadata:
  name: web-upgrade
spec:
  selector: {tier: web}
  canary: 1
  batch_size: 2
  pause: 30s
  health_gate: {timeout: 2m, interval: 5s}

• REST under /1.0. Mutations may be synchronous or asynchronous, with an operations resource you can wait on. Updates use ETag and If-Match, events stream as server-sent events, features are negotiated by api_extensions, and error codes are stable.
• Authentication: a unix socket with peer credentials, mutual TLS with pinned fingerprints, API tokens, and web sessions with CSRF protection.
• Go SDK: a separate module with no third-party dependencies, context everywhere, typed wire types, Wait for operations, iterators for lists and event streams, and an in-memory fake server for tests. The CLI and jailops-sync use nothing else.
• CLI: jailops prints tables for people and -o json, yaml or name for scripts, has stable exit codes, a --dry-run flag, and jailops explain CODE for every error code.
• Web consoles: a host console (also the hub console) embedded in jailopsd, and a backup console embedded in jailops-backupd. They are Vue 3 apps with a light and a dark theme, and each may only call its own API prefixes.

$ jailops jail list -o json
$ jailops jail list --state running -o name
$ jailops events --type lifecycle -o json
$ jailops explain not_found

• Linux jails (Alpine and Debian) through the Linuxulator. Experimental. Alpine and Debian 13 (glibc) ran as jails on FreeBSD 15.1 arm64 with IPv4 and IPv6; amd64 and other glibc distributions are not yet verified.
• Provisioning and managed updates. Today you upgrade a jail on demand with jailops jail upgrade, and jailops-sync rollouts stage upgrades across hosts. Update policies with schedules and notifications are Planned.
• Cluster mode with Raft and self-healing fleets are Planned. Failover of stateful workloads will not happen automatically without fencing.
• Out of scope by design: virtual machines, bhyve, OCI containers, live migration, and any storage other than ZFS.