Help

If this happens, do this.

What you see, why it happens, and the command that fixes it. Start with jailops doctor: it checks the host and names most of these by their id.

Show answers about

Install and first start

The daemon, the helper, the first doctor run.

the socket does not exist: is jailopsd running?

Cause

The daemon is not running, so the client finds no socket. Most often it refused to start because of its configuration.

Fix

Ask rc, then start it and read what it prints: the rc script validates the configuration first. field X not found is a key the daemon does not know; remove it.

# service jailopsd status
# service jailopsd start
# service jailops_priv checkconfig
Link to this answer

unauthorized: authentication required on the host itself

Cause

On the local socket only root is recognised. Any other local user is anonymous, like an unauthenticated remote client.

Fix

Run the command as root (FreeBSD has no sudo in the base system: log in as root or use su -), or give it an identity: a token in JAILOPS_TOKEN or a client certificate. Root replaces a lost password without the old one. These commands run as root:

# jailops token create NAME --role operator --projects web
# jailops user set-password NAME
Link to this answer

Every ZFS or jail operation fails with forbidden

Cause

The privileged helper refuses what priv.conf does not allow. The message names the line to add (admin-host-mount and so on).

Fix

Add the line to /usr/local/etc/jailops/priv.conf as root and restart the helper. service jailops_priv checkconfig prints the effective roots and allow-lists.

# service jailopsd restart
# service jailops_priv checkconfig
Link to this answer

a project is required, or a jail needs a network

Cause

There is no default project, network or storage. Nothing is inferred.

Fix

Name them, or choose a current project once.

$ jailops project switch web
$ jailops create site --image freebsd-15-1 --network lan --storage default --start --autostart
Link to this answer

jailops doctor fails host.release

Cause

The host runs a FreeBSD release below 15.1. JailOps targets 15.1 and later, and nothing older.

Fix

Upgrade the host to FreeBSD 15.1 or later, then run the doctor again.

Link to this answer

jailops doctor reports a missing kernel.module

Cause

if_epair, pf or zfs is not loaded.

Fix

Load it now and at boot.

# kldload -n if_epair pf zfs
# sysrc -f /boot/loader.conf if_epair_load=YES pf_load=YES zfs_load=YES
Link to this answer

jailops doctor warns sysctl.persistent

Cause

Forwarding (or, with a switched IPv6 network, net.inet6.icmp6.nd6_onlink_ns_rfc4861) is on now but will not survive a reboot. JailOps does not write these settings to rc.conf or sysctl.conf (the optional host modules write only their own variables).

Fix

Make it persistent.

# sysrc gateway_enable=YES ipv6_gateway_enable=YES
# echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf
Link to this answer

Firewall policies have no effect

Cause

The anchor line is missing from pf.conf (pf.anchors), or a quick pass rule comes before anchor "jailops/*" and passes the traffic for good (pf.order).

Fix

Add the line among your filter rules, move the anchor before broad quick passes, check, then load.

# /etc/pf.conf
# ... your options and scrub, then ...
anchor "jailops/*"

# pfctl -nf /etc/pf.conf
# pfctl -f /etc/pf.conf
Link to this answer

Jails and storage

Starting, stopping, volumes, logs and limits.

A jail does not start, or jail -c hangs until it times out

Cause

An rc script inside the jail starts its daemon through daemon(8) without -f. The supervisor keeps the descriptors it was given, here the pipes of whoever ran jail -c, and the caller waits for the end of an output that never comes. A forgejo jail showed it as WaitDelay expired before I/O complete.

Fix

JailOps sets exec.consolelog in every jail's jail.conf: the output of /etc/rc and the hooks goes to a file on the host instead of the caller's pipes. Upgrade, then read that file when a start fails.

# pkg upgrade jailops
# service jailopsd restart
# service jailopsd restart
# tail -n 50 /var/log/jail_PROJECT_JAIL_console.log

For a jail of your own outside JailOps, the same line in its jail.conf does it; see the recipe.

Link to this answer

A volume looks empty after a reboot

Cause

The volume's dataset has canmount=on, from before JailOps set canmount=noauto on volumes. ZFS mounted it at boot, before the jail root it lies in, and the root's mount then covered it. jailops doctor warns jail.volumes.

Fix

Run the zfs set the doctor's hint lists, for each dataset it names, then stop and start the jail so the volume is mounted after its root. What the jail wrote in the meantime went to the dataset underneath: look there before you delete anything.

These commands run as root on the host.

# zfs get -r canmount POOL/jailops/projects/PROJECT/jails/JAIL
# zfs set canmount=noauto DATASET
# jailops restart JAIL
Link to this answer

Stopping or deleting a jail reports busy

Cause

Something inside still holds a file open, so the unmount fails. Stopping retries a busy unmount for a few seconds before it gives up. busy is also the answer while another operation holds the jail.

Fix

Find the process with fstat on the jail's root, end it, and stop again. For a lock, wait for the running operation.

These commands run as root on the host.

# fstat -f /jailops/projects/PROJECT/jails/JAIL/root
# jailops op list
Link to this answer

busy: jail "x" is not mounted when you read its logs

Cause

A stopped jail's root is mounted only while it runs, so there is nothing to read.

Fix

Start the jail, or mount the dataset the message names with zfs mount on the host, and read again.

Link to this answer

A rotated log answers not readable (compressed)

Cause

The logs API reads plain files. A compressed rotated copy (messages.1.gz) is listed but not read.

Fix

Read it from inside the jail.

$ jailops exec JAIL -- zcat /var/log/messages.1.gz
Link to this answer

quota_exceeded

Cause

A project quota (jails, CPU, memory, PIDs, addresses or storage) would be exceeded. It is checked before anything is created.

Fix

See the usage, free something up, or ask a global administrator to raise the quota.

$ jailops project show NAME
Link to this answer

ip_conflict or pool_exhausted

Cause

The address you asked for is taken or outside the range, or the range is full.

Fix

Use auto, look at the leases, or widen the range.

$ jailops network leases NAME
Link to this answer

A jail has no bash, or shows the condition ShellSetup

Cause

A new FreeBSD jail gets bash from the host's package repository at its first start. The host could not reach the repository then; the start succeeded anyway, and the jail says so. A jail made before has neither bash nor the prompt: nothing changes on its own.

Fix

Fix the host's package repository, then give the jail bash and the prompt. The jail must run. In the console: Install bash and the prompt on the jail's Console tab.

$ jailops jail shell-setup JAIL
Link to this answer

Any other error code

Cause

Every error prints as code: message with a hint.

Fix

jailops explain CODE explains one code, jailops explain lists them all, and jailops doctor is always the first thing to run.

Link to this answer

Networking and IPv6

Addresses, neighbour discovery and the firewall.

The host is unreachable over IPv6 for a few minutes, every few hours

Cause

On a switched /64, the provider's routers check that your address is still there with unicast neighbour solicitations sent from their global addresses. The host holds a /128 and no router advertisement describes the prefix, so FreeBSD's default (net.inet6.icmp6.nd6_onlink_ns_rfc4861=0) drops them as off-link. The router marks the host unreachable until its next multicast solicitation, and the jails behind it go with it. jailops doctor warns while a proxy-mode network exists.

Fix

One sysctl, now and at every boot. The packet must still arrive with hop limit 255, so nothing from beyond the link gets in.

These commands run as root on the host.

# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1
# echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf
# netstat -s -p icmp6 | grep 'bad neighbor solicitation'

The whole story, with captures: JailOps on a netcup server with IPv6.

Link to this answer

A jail on a switched IPv6 network is not reachable from outside

Cause

Nobody answers the router's neighbour solicitation for the jail's address: the proxy is not approved in priv.conf, not running, or the solicitation never reaches the host.

Fix

Run the doctor (network.ndproxy) and look at the network's NDPProxy condition. Then watch the uplink while the router asks: a solicitation for the jail's address and an advertisement from the host's MAC should follow within a millisecond. If the solicitation never arrives, the provider filters by MAC or does not deliver the prefix to you, and no proxy can fix that.

These commands run as root on the host.

# jailops doctor
# jailops network show NAME
# tcpdump -n -e -i vtnet0 'icmp6 and (ip6[40] == 135 or ip6[40] == 136)'
Link to this answer

The first packets to a new jail are lost

Cause

The router resolves a new address on its own timers. The proxy announces each address when it appears, which usually avoids the wait.

Fix

Usually nothing to fix: it settles by itself once the router has resolved the address. If it does not, see the entry above.

Link to this answer

A jail starts and has no network

Cause

Forwarding is off for a routed network, a kernel module is missing, or the pf anchors are missing.

Fix

Look at the network's conditions and the doctor's sysctl.*, kernel.module.* and pf.anchors lines.

These commands run as root on the host.

# jailops network show NAME
# jailops doctor
# sysctl net.inet.ip.forwarding net.inet6.ip6.forwarding
Link to this answer

A service in a jail is not reachable from the internet

Cause

The host's pf.conf blocks inbound traffic on the uplink by default. Unless you hand pf.conf to the optional host firewall module, JailOps does not edit that file, so it does not open the door for you.

Fix

Admit the jails' addresses and only the ports you want, with a rule of your own. The jail's own anchor still decides what reaches the jail.

pass in quick on vtnet0 inet6 proto tcp from any to 2001:db8:1:2:100::/80 port 8080
Link to this answer

Jails behind IPv4 NAT lose most of their packets

Cause

On a host whose pf.conf blocks by default and NATs the jails, forwarded packets were translated early by pf's floating state and then dropped.

Fix

Put set state-policy if-bound at the top of pf.conf and reload.

Link to this answer

Your SSH session drops when you reload the firewall

Cause

service pf restart and pfctl -F states flush the state table, your session included.

Fix

Check the file, then load it in place.

# pfctl -nf /etc/pf.conf
# pfctl -f /etc/pf.conf
Link to this answer

A jail cannot reach a jail of another project

Cause

By design: projects are closed to each other, like an external host. Only addresses cross.

Fix

Open exactly what you need, with the addresses written on both sides: an address set and a policy rule in the project that serves, and a rule that allows the address out in the project that connects.

$ jailops firewall policy create web --default-in deny --default-out allow
$ jailops firewall rule add web in --from set:office --port 22 \
    --comment "ssh from the office"
Link to this answer

network N is already granted to project P

Cause

A network belongs to exactly one project. Storage may be shared; networks may not.

Fix

Use another network, or revoke the grant first.

$ jailops network revoke NETWORK --project PROJECT
Link to this answer

Apps

Installing from the catalog, and writing a definition of your own.

An app's service start never returns

Cause

The service's rc script starts its daemon through daemon(8) without -f, and the supervisor holds the pipes of the command that started it.

Fix

The installer starts services through a fixed wrapper that sends the rc script's output to a temporary file in the jail and prints it only when the start fails. Upgrade JailOps. In an rc script of your own, start the daemon with daemon -f.

Link to this answer

PostgreSQL's initdb stops at shmget: Failed system call

Cause

A jail has no System V shared memory, semaphores or message queues unless its parameters allow them.

Fix

In a definition, sysvipc: true. For an existing jail, the key of the same name; a restart applies it. The jail gets namespaces of its own (new), never the host's.

$ jailops config set JAIL sysvipc=true --restart
Link to this answer

A service cannot write its log or pid directory: it belongs to root

Cause

pkg -r installs from the host, where the package's user does not exist, so its directories come out root-owned (or owned by a number that means something else). pkg's database records no owners, so nothing can restore them generically. Data volumes start root-owned too.

Fix

In a definition, give the directories to their user in setup, with mkdir -p first. By hand, the same in the jail.

# in a definition
setup:
  - [/bin/mkdir, -p, /var/log/valkey, /var/run/valkey]
  - [/usr/sbin/chown, "valkey:valkey", /var/db/valkey, /var/log/valkey, /var/run/valkey]

$ jailops exec JAIL -- chown valkey:valkey /var/log/valkey
Link to this answer

Shared object "..." not found after a package went in with pkg -r

Cause

pkg -r does not run the jail's library trigger, so the running jail's library cache does not know the new libraries.

Fix

jailops app install refreshes it for you. After a pkg -r of your own, refresh it in the jail.

$ jailops exec JAIL -- /sbin/ldconfig -m /usr/local/lib
Link to this answer

nginx answers, but its welcome page is missing

Cause

The package ships its default site as /usr/local/www/nginx-dist and links /usr/local/www/nginx to it in a post-install step of its package list. Installed from the host into the jail, that step did not take effect.

Fix

The catalog's nginx definition makes the directory and copies the default site in its setup. By hand:

$ jailops exec nginx -- mkdir -p /usr/local/www/nginx
$ jailops exec nginx -- cp -Rn /usr/local/www/nginx-dist/. /usr/local/www/nginx/
Link to this answer

ssh-keygen is not found in an app jail

Cause

The thin FreeBSD image has no ssh-keygen.

Fix

List openssh-portable as a package of the definition (from the FreeBSD repository), as the catalog's isms does; the program is then /usr/local/bin/ssh-keygen.

Link to this answer

Postfix logs improper command pipelining, or Dovecot rejects its configuration

Cause

Postfix refuses commands that arrive before its 220 greeting: a probe or script that talks first is cut off. Dovecot's configuration syntax differs between major versions, and a file written for one is rejected by the other.

Fix

Make clients wait for the greeting. Write Dovecot's configuration for the version the package installs, and check it in the jail before you start the service.

$ jailops exec mail -- /usr/local/bin/doveconf -n
$ jailops exec mail -- /usr/local/sbin/postfix check
Link to this answer

An app is listed as invalid

Cause

The definition fails validation: the reason is in the list. A definition written for a newer JailOps (a key, a format version or a category this daemon does not know) is invalid until JailOps is upgraded.

Fix

Read the reason, check the files offline, and upgrade JailOps when the definition needs it.

These commands run as root on the host.

# jailops app list
# jailops app validate /usr/local/share/jailops/apps
# pkg upgrade jailops
Link to this answer

jailops app install or jailops create says there is no image

Cause

Without --image the installer takes the FreeBSD image of the newest release the host can run (of two images of the same release, the first by name). There is no FreeBSD image yet, or every one is of a newer release than the host. jailops create names its image, and that image does not exist. The hint names the commands that make the image, with the host's release filled in, or lists the images there are.

Fix

Create and build the image of the host's release (the console: Images, Create image), or name an image that exists with --image. An image is one FreeBSD release: freebsd-15-1 is 15.1-RELEASE.

# jailops image create freebsd-15-1 --release 15.1-RELEASE
# jailops image build freebsd-15-1
# jailops app install nginx --image freebsd-15-1 --network lan --storage default
Link to this answer

An installation failed and the jail is stopped

Cause

A step after the jail's first start failed: a package, a setup command, a service, or the health check within its wait. The jail is kept for you to look inside.

Fix

The error names the step. Look inside, then delete the jail and try again.

$ jailops start JAIL
$ jailops jail logs JAIL
$ jailops shell JAIL
$ jailops delete JAIL
Link to this answer

An app from the JailOps repository fails at the package step

Cause

The definition installs with -r jailops, and the host's pkg configuration has no repository named jailops. The installer does not check beforehand; the step fails with pkg's own message.

Fix

Configure the repository as the packages repository's README shows, then delete the kept jail and install again.

Link to this answer

busy: the app is being installed

Cause

While an installation holds the jail, every exec, shell and console session is refused: pkg -r works on the jail's root from the host.

Fix

Wait for the installation to finish.

Link to this answer

Upgrades and backup

Going forward, going back, and the backups.

An app broke after an update

Cause

A failed package update is not rolled back by itself. The update recorded a snapshot first, and the install left app-installed.

Fix

Find the snapshot in the update history, or go back to the fresh installation. Both discard what the jail wrote since: the command shows how much and asks you to type the jail name.

$ jailops update history JAIL
$ jailops jail rollback JAIL --all --snapshot SNAPSHOT
$ jailops jail rollback JAIL --all --snapshot app-installed
Link to this answer

The nightly upgrade of an image failed

Cause

A jail that moved to the new generation failed its health check, or its /etc could not be merged. That jail was put back on its old generation, and the run stopped there: the jails after it were not attempted. The image card in the console keeps the failure as a notice until a later run succeeds.

Fix

jailops image show IMAGE lists the run and names the jail; the jail's update history says what happened. Fix the jail, then run the upgrade again, or let the next night do it.

$ jailops image show freebsd-15-1
$ jailops update history JAIL --project PROJECT
$ jailops image upgrade freebsd-15-1
Link to this answer

data_loss_unconfirmed

Cause

The operation would discard data and was sent without confirmation.

Fix

Check what would be lost (the CLI shows the bytes written since the snapshot), then type the jail name, or pass --confirm-data-loss in a script. -y does not apply to data loss.

Link to this answer

merge_conflict during an upgrade

Cause

The /etc merge of the new generation needs a person. The jail stays on the old generation.

Fix

Resolve the listed files in the jail, then upgrade again.

$ jailops jail upgrade JAIL
Link to this answer

image_in_use

Cause

A jail still uses the image or generation. The error names the jails; an alias alone never blocks.

Fix

Upgrade or delete those jails, then retry.

Link to this answer

chain_broken

Cause

A backup chain has a missing or corrupt link.

Fix

Verify the backup, then take a new full backup.

$ jailops backup verify BACKUP
Link to this answer

key_not_escrowed

Cause

A raw backup was refused because the backup key has no escrow.

Fix

Escrow the key to a recipient you control.

$ jailops backup key escrow --recipient age1...
Link to this answer

jailops doctor fails zfs.compat

Cause

The pool has features beyond zfs.compat_file, so replication to an older host would fail.

Fix

Check the pool features on both ends. JailOps never runs zpool upgrade; do not run it on a pool that replicates to an older host.

Link to this answer

jailops doctor fails db.integrity

Cause

The daemon's SQLite database is damaged.

Fix

Restore from the latest copy in <data_dir>/backups/, which the daemon writes with VACUUM INTO.

Link to this answer

jailops doctor fails operations.journal

Cause

A rollback did not complete.

Fix

Inspect the datasets of the jails it names, then remove the journal record.

Link to this answer

Not here? Ask.

Open an issue with the command, its output and what jailops doctor says. The troubleshooting guide in the repository has every doctor check.