Articles

Jail updates without cron

How JailOps keeps FreeBSD jails up to date without a cron script: one image per release, the patch levels as generations, and a nightly upgrade you turn on, with a health check, a rollback and a record of every run. What it leaves alone, and what was tested.

Updating FreeBSD jails with a cron script

It works until the night it does not.

Many hosts with jails have one: a script that cron starts at night. It upgrades the base system of each jail, runs pkg upgrade in every jail, maybe restarts a service or two, and mails its output to someone.

It does the work, and it leaves out the three things that matter on the night something breaks:

  • No health check. The script finishes when the commands return, not when the services answer. A jail whose service no longer starts looks like every other jail.
  • No rollback. When an upgrade breaks a jail, the way back is whatever you prepared by hand, if you did.
  • No record. What changed, in which jail, and what failed lives in a mail, if the mail arrived and someone read it.

And it rolls the same change over every jail, the broken ones included. JailOps replaces such a script with a nightly upgrade per image, which the daemon runs itself. Before it, the model it rests on.

pkgbase jails: one image, one release

Patch levels are generations.

A thin jail's root is a ZFS clone of an image generation: an immutable pkgbase userland. What must survive an upgrade (/etc, /usr/local, /var/db, /home, /root and the jail's volumes) lives on datasets of its own. Upgrading a jail swaps the root for a clone of a newer generation and keeps those datasets.

  • An image is one FreeBSD release and is named after it: freebsd-15-1 is 15.1-RELEASE.
  • Its generations are the patch levels of that release. Each upgrade builds the next one, with the release's current patches and packages.
  • A new release is a new image (freebsd-15-2), never a generation of the old one. Nothing upgrades a jail from one release to the next in place.
  • The host first. A jail runs on the host's kernel, so the host must run a release at least as new as the image's.

A jail made with --image freebsd-15-1 follows the image's newest generation. The commands are the same whether you run them by hand or let the nightly run do them:

patch levels by hand
$ jailops image upgrade freebsd-15-1 --dry-run   # the plan, and which jails would move
$ jailops image upgrade freebsd-15-1             # the next generation, then every thin jail, one at a time
$ jailops image generation list freebsd-15-1  # which jail is on which generation

Each jail move is the same as jailops jail upgrade: an atomic snapshot of the jail, the new root, a three-way merge of /etc, the swap, and the jail's health check. A jail that fails it is swapped back to its old root. The run stops at the first jail that fails, so a bad generation is not rolled across the host.

Automatic jail upgrades, every night

Off until you turn it on.

The nightly auto upgrade is set per image. Nothing is upgraded unless you turned it on: not when you create an image, not in the first-run wizard, where the box is unticked.

turn it on, look, turn it off
$ jailops image set freebsd-15-1 --auto-upgrade --upgrade-at 03:00   # --upgrade-packages=false leaves the packages in the jails alone
$ jailops image show freebsd-15-1                          # the next run and the latest runs
$ jailops image set freebsd-15-1 --auto-upgrade=false

In the console it is the checkbox Upgrade automatically every night on the image's card of the Images page, with the time and Also upgrade the packages inside the jails, and the same box in Create image. Only a global administrator changes it; everyone else sees it.

The Images page of the JailOps host console in dark theme: the image freebsd-15-1 with four generations, the box Upgrade automatically every night ticked, the time 03:00, the box Also upgrade the packages inside the jails ticked, the next run, and the recent runs: two nights with nothing new and one that upgraded to freebsd-15-1#g4, moving five jails. Demo data.The Images page of the JailOps host console in light theme: the image freebsd-15-1 with four generations, the box Upgrade automatically every night ticked, the time 03:00, the box Also upgrade the packages inside the jails ticked, the next run, and the recent runs: two nights with nothing new and one that upgraded to freebsd-15-1#g4, moving five jails. Demo data.
The nightly upgrade on the image's card: the setting, the next run, the recent runs. Demo data.

Every night at the chosen time (the host's time zone; 03:00 unless you choose another), a run does three things:

  1. It builds the image's next generation, with the patch updates of the release. When nothing changed, the run is recorded as unchanged.
  2. It moves the thin jails of the image to it, one at a time, exactly as jailops image upgrade does: snapshot, new root, /etc merge, health check, and the automatic rollback of a jail whose check fails. It stops at the first jail that fails; the jails after it stay where they are until the next run. Jails that an earlier, stopped run left behind are moved too, even on a night when nothing new was built.
  3. With packages on (the default), it upgrades the packages inside the running jails of the image. For a FreeBSD jail the host runs pkg -r on the jail's root and names the jail's own packages, so a base system package never comes in this way: the base moves only with the generation.

It leaves alone:

  • jails whose update strategy is manual or in place, thick jails, Linux jails (their root) and jails pinned to a generation such as freebsd-15-1#g3 (their root);
  • the packages of a stopped jail;
  • the host. Upgrading the host is yours to do, and it comes first.

A host that was off at the chosen time runs the upgrade when it is back, if that is within four hours; later than that it waits for the next night. Every run is kept with the image: when it started, its result (ok, unchanged or failed, with what failed) and how many jails were moved, upgraded and failed. A failed run raises the event image-upgrade-failed and stays on the image's card as a notice until a later run succeeds.

The image card of freebsd-15-1 in the JailOps host console in dark theme, with a red notice: The last nightly upgrade failed. Jail db: health check /usr/local/bin/check exited 1; rolled back to freebsd-15-1#g4. Below it the recent runs, the newest one marked Failed with 2 moved, 1 failed. Demo data.The image card of freebsd-15-1 in the JailOps host console in light theme, with a red notice: The last nightly upgrade failed. Jail db: health check /usr/local/bin/check exited 1; rolled back to freebsd-15-1#g4. Below it the recent runs, the newest one marked Failed with 2 moved, 1 failed. Demo data.
A night that failed: the jail that failed its health check went back to its old generation, and the run stopped there. Demo data.

Moving from your cron script: remove the entry and turn the nightly upgrade on for the image at the same hour. The daemon runs it itself, so it needs no token or client on the host; a host that was off catches up within four hours; the packages are upgraded in the same run; and each result is kept with the image instead of in a mail.

What a run prints

From a real run, and from the tests.

A run looks like this, on a host with the image freebsd-15-2. The jail up was on the first generation, one of its packages was outdated, and the nightly time was two minutes ahead, so the daemon's clock started the run:

before: off until asked for
$ jailops image show freebsd-15-2
Image freebsd-15-2
  System:    freebsd 15.2-PRERELEASE (amd64)
  Source:    pkgbase
  Aliases:   none
  Keeps:     every generation
  Age:       1m
  Nightly:   off ('jailops image set freebsd-15-2 --auto-upgrade' turns it on)
after the run
$ jailops image show freebsd-15-2
Image freebsd-15-2
  System:    freebsd 15.2-PRERELEASE (amd64)
  Source:    pkgbase
  Aliases:   none
  Keeps:     every generation
  Age:       3m
  Nightly:   every night at 14:49: generation and jails and their packages; next 2026-11-04 14:49

Automatic upgrades
  STARTED            RESULT   WHAT
  2026-11-03 14:49   ok       2 moved, 1 packages upgraded

Generations
  GEN   PACKAGES   AGE   USED BY
  g1    47         3m    demo/up,demo/web
  g2    47         2m    demo/up,demo/web

$ jailops jail list --project demo
NAME   STATE     IMAGE          GEN               ADDRESSES   AGE
up     running   freebsd-15-2   freebsd-15-2#g2   10.77.0.3   2m
web    running   freebsd-15-2   freebsd-15-2#g2   10.77.0.2   3m

The run built nothing new (the repository had nothing new for the image), moved the two jails that were behind to the newest generation, and upgraded the outdated package inside the one jail that had it. The test then checked that the package's version had changed and that the jail still ran.

A failed run looks like this when you run the same upgrade by hand. This output is from the command line's tests, with their fake server:

a jail fails its health check
$ jailops image upgrade freebsd-15-1
generation/freebsd-15-1#g5 built from freebsd-15-1#g4
JAIL   PROJECT   STATUS          FROM              TO                REASON
pin    main      skipped         freebsd-15-1#g1   -                 pinned to freebsd-15-1#g1
web    main      failed          freebsd-15-1#g2   freebsd-15-1#g5   health check failed (the jail was swapped back to freebsd-15-1#g2; application data was not touched)
api    shop      not_attempted   freebsd-15-1#g3   -                 stopped after main/web failed
! error: jail web failed; the jails after it were not attempted
! hint: 'jailops update history web --project main' shows what happened; run the upgrade again once it is resolved

$ jailops update history web
ID         JAIL   KIND       RESULT   CHANGE       SNAPSHOT      AGE
upd-<id>   web    packages   ok       2 packages   update-<id>   3h

The ids are placeholders of the tests. A jail's history lists every update with its result and the snapshot taken before it.

Alpine and Debian jails, and unattended-upgrades

apk yes, apt never.

  • Alpine jails: their packages are checked and upgraded with apk, run in the jail, by jailops update, the managed schedules and the nightly run. A Linux jail is always a full copy, so it never moves to a new generation.
  • Debian jails: JailOps never runs apt in them, and no update of JailOps touches their packages. A Debian jail keeps its own packages up to date, for instance with unattended-upgrades inside the jail, as on any Debian system.

The reason is the same as for everything else here: an update that JailOps runs must be one it can check, roll back and record. For a Debian jail it does not do that yet, so it does not pretend to.

A new FreeBSD release: a new image

15.2, after the host.

When 15.2-RELEASE comes, nothing moves on its own. The order:

  1. Upgrade the host, then run jailops doctor. A jail is never moved to a generation of a newer release than the host's kernel.
  2. Create and build the image of the new release, and turn its nightly upgrade on if you want it: each release is its own image, with its own setting.
  3. Create new jails on it. jailops app install takes the image of the newest release the host can run when you name none.
after the host runs 15.2
$ jailops image create freebsd-15-2 --release 15.2-RELEASE
$ jailops image build freebsd-15-2
$ jailops image set freebsd-15-2 --auto-upgrade --upgrade-at 03:00

The console points it out: when the host runs a newer release than its newest FreeBSD image, the Images page and the Overview say so and open Create image filled in for it.

The Images page of the JailOps host console in dark theme with a notice: This host runs FreeBSD 15.2-RELEASE, and no image has it yet. Create the image for it. A new release is a new image: the jails that exist stay on the release of their image and keep getting its patch levels. A button reads Create the image for 15.2-RELEASE; below it, the image freebsd-15-1. Demo data.The Images page of the JailOps host console in light theme with a notice: This host runs FreeBSD 15.2-RELEASE, and no image has it yet. Create the image for it. A new release is a new image: the jails that exist stay on the release of their image and keep getting its patch levels. A button reads Create the image for 15.2-RELEASE; below it, the image freebsd-15-1. Demo data.
The host runs 15.2, the newest image is 15.1: the console says so. Demo data.
The Create image dialog of the JailOps host console in dark theme: system FreeBSD, the name freebsd-15-2, the release 15.2-RELEASE (this host), the architecture amd64, the box Upgrade automatically every night not ticked, the box Build the first generation now ticked, and the button Create and build. Demo data.The Create image dialog of the JailOps host console in light theme: system FreeBSD, the name freebsd-15-2, the release 15.2-RELEASE (this host), the architecture amd64, the box Upgrade automatically every night not ticked, the box Build the first generation now ticked, and the button Create and build. Demo data.
Create image, filled in for the host's release; the nightly upgrade stays off until you tick it. Demo data.

Moving a jail's data to a jail of the new release is not automated. Create the new jail next to the old one, take a backup of the old one, copy or export the data with the application's own tools, check it, move the addresses or ports over, and delete the old jail when you no longer need it. The old image keeps getting its patch levels for as long as you keep it; delete it once no jail uses it.

Limits

So you can judge.

  • The host is not upgraded. Upgrading the host's base system with boot environments is designed, not built.
  • No live migration. A jail shares its host's kernel, so a move between releases or hosts means a new jail; moving a jail to another host is planned (the roadmap, a plan and not a promise).
  • A failed package upgrade is not rolled back by itself. Packages live on the jail's datasets, so a root rollback does not undo them. The way back is jailops jail rollback JAIL --all --snapshot SNAPSHOT with the snapshot the update recorded, which discards what the jail wrote since.
  • No notification yet. A failed run is an event and a notice in the console; mail and webhook notifications are planned.

The details are in docs/upgrade.md.

More articles

Try it on one server.

Try the console from the screenshots, install on a clean FreeBSD 15.1 machine, or add JailOps to a host you already run.