Help
If this happens, do this.
What you see, why it happens, and the command that fixes it. Start with jailops doctor: it checks the host and names most of these by their id.
- IPv6 drops every few hours
- A jail start hangs
- initdb: Failed system call
- Directories owned by root
- Shared object not found
- A volume looks empty
- Roll an app back
Install and first start
The daemon, the helper, the first doctor run.
the socket does not exist: is jailopsd running?
the socket does not exist: is jailopsd running?- Cause
The daemon is not running, so the client finds no socket. Most often it refused to start because of its configuration.
- Fix
Ask rc, then start it and read what it prints: the rc script validates the configuration first.
field X not foundis a key the daemon does not know; remove it.# service jailopsd status # service jailopsd start # service jailops_priv checkconfig
unauthorized: authentication required on the host itself
unauthorized: authentication required on the host itself- Cause
On the local socket only root is recognised. Any other local user is anonymous, like an unauthenticated remote client.
- Fix
Run the command as root (FreeBSD has no
sudoin the base system: log in as root or usesu -), or give it an identity: a token inJAILOPS_TOKENor a client certificate. Root replaces a lost password without the old one. These commands run as root:# jailops token create NAME --role operator --projects web # jailops user set-password NAME
Every ZFS or jail operation fails with forbidden
forbidden- Cause
The privileged helper refuses what
priv.confdoes not allow. The message names the line to add (admin-host-mountand so on).- Fix
Add the line to
/usr/local/etc/jailops/priv.confas root and restart the helper.service jailops_priv checkconfigprints the effective roots and allow-lists.# service jailopsd restart # service jailops_priv checkconfig
a project is required, or a jail needs a network
a project is required, or a jail needs a network- Cause
There is no default project, network or storage. Nothing is inferred.
- Fix
Name them, or choose a current project once.
$ jailops project switch web $ jailops create site --image freebsd-15-1 --network lan --storage default --start --autostart
jailops doctor fails host.release
jailops doctor fails host.release- Cause
The host runs a FreeBSD release below 15.1. JailOps targets 15.1 and later, and nothing older.
- Fix
Upgrade the host to FreeBSD 15.1 or later, then run the doctor again.
jailops doctor reports a missing kernel.module
jailops doctor reports a missing kernel.module- Cause
if_epair,pforzfsis not loaded.- Fix
Load it now and at boot.
# kldload -n if_epair pf zfs # sysrc -f /boot/loader.conf if_epair_load=YES pf_load=YES zfs_load=YES
jailops doctor warns sysctl.persistent
jailops doctor warns sysctl.persistent- Cause
Forwarding (or, with a switched IPv6 network,
net.inet6.icmp6.nd6_onlink_ns_rfc4861) is on now but will not survive a reboot. JailOps does not write these settings torc.conforsysctl.conf(the optional host modules write only their own variables).- Fix
Make it persistent.
# sysrc gateway_enable=YES ipv6_gateway_enable=YES # echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf
Firewall policies have no effect
- Cause
The anchor line is missing from
pf.conf(pf.anchors), or aquickpass rule comes beforeanchor "jailops/*"and passes the traffic for good (pf.order).- Fix
Add the line among your filter rules, move the anchor before broad
quickpasses, check, then load.# /etc/pf.conf # ... your options and scrub, then ... anchor "jailops/*" # pfctl -nf /etc/pf.conf # pfctl -f /etc/pf.conf
Jails and storage
Starting, stopping, volumes, logs and limits.
A jail does not start, or jail -c hangs until it times out
jail -c hangs until it times out- Cause
An rc script inside the jail starts its daemon through daemon(8) without
-f. The supervisor keeps the descriptors it was given, here the pipes of whoever ranjail -c, and the caller waits for the end of an output that never comes. A forgejo jail showed it asWaitDelay expired before I/O complete.- Fix
JailOps sets
exec.consolelogin every jail'sjail.conf: the output of/etc/rcand the hooks goes to a file on the host instead of the caller's pipes. Upgrade, then read that file when a start fails.# pkg upgrade jailops # service jailopsd restart # service jailopsd restart # tail -n 50 /var/log/jail_PROJECT_JAIL_console.log
For a jail of your own outside JailOps, the same line in its
jail.confdoes it; see the recipe.
Stopping or deleting a jail reports busy
busy- Cause
Something inside still holds a file open, so the unmount fails. Stopping retries a busy unmount for a few seconds before it gives up.
busyis also the answer while another operation holds the jail.- Fix
Find the process with
fstaton the jail's root, end it, and stop again. For a lock, wait for the running operation.These commands run as root on the host.
# fstat -f /jailops/projects/PROJECT/jails/JAIL/root # jailops op list
busy: jail "x" is not mounted when you read its logs
busy: jail "x" is not mounted when you read its logs- Cause
A stopped jail's root is mounted only while it runs, so there is nothing to read.
- Fix
Start the jail, or mount the dataset the message names with
zfs mounton the host, and read again.
A rotated log answers not readable (compressed)
not readable (compressed)- Cause
The logs API reads plain files. A compressed rotated copy (
messages.1.gz) is listed but not read.- Fix
Read it from inside the jail.
$ jailops exec JAIL -- zcat /var/log/messages.1.gz
quota_exceeded
quota_exceeded- Cause
A project quota (jails, CPU, memory, PIDs, addresses or storage) would be exceeded. It is checked before anything is created.
- Fix
See the usage, free something up, or ask a global administrator to raise the quota.
$ jailops project show NAME
ip_conflict or pool_exhausted
ip_conflict or pool_exhausted- Cause
The address you asked for is taken or outside the range, or the range is full.
- Fix
Use
auto, look at the leases, or widen the range.$ jailops network leases NAME
A jail has no bash, or shows the condition ShellSetup
ShellSetup- Cause
A new FreeBSD jail gets bash from the host's package repository at its first start. The host could not reach the repository then; the start succeeded anyway, and the jail says so. A jail made before has neither bash nor the prompt: nothing changes on its own.
- Fix
Fix the host's package repository, then give the jail bash and the prompt. The jail must run. In the console: Install bash and the prompt on the jail's Console tab.
$ jailops jail shell-setup JAIL
Any other error code
- Cause
Every error prints as
code: messagewith a hint.- Fix
jailops explain CODEexplains one code,jailops explainlists them all, andjailops doctoris always the first thing to run.
Networking and IPv6
Addresses, neighbour discovery and the firewall.
The host is unreachable over IPv6 for a few minutes, every few hours
- Cause
On a switched
/64, the provider's routers check that your address is still there with unicast neighbour solicitations sent from their global addresses. The host holds a/128and no router advertisement describes the prefix, so FreeBSD's default (net.inet6.icmp6.nd6_onlink_ns_rfc4861=0) drops them as off-link. The router marks the host unreachable until its next multicast solicitation, and the jails behind it go with it.jailops doctorwarns while a proxy-mode network exists.- Fix
One sysctl, now and at every boot. The packet must still arrive with hop limit 255, so nothing from beyond the link gets in.
These commands run as root on the host.
# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 # echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf # netstat -s -p icmp6 | grep 'bad neighbor solicitation'
The whole story, with captures: JailOps on a netcup server with IPv6.
A jail on a switched IPv6 network is not reachable from outside
- Cause
Nobody answers the router's neighbour solicitation for the jail's address: the proxy is not approved in
priv.conf, not running, or the solicitation never reaches the host.- Fix
Run the doctor (
network.ndproxy) and look at the network'sNDPProxycondition. Then watch the uplink while the router asks: a solicitation for the jail's address and an advertisement from the host's MAC should follow within a millisecond. If the solicitation never arrives, the provider filters by MAC or does not deliver the prefix to you, and no proxy can fix that.These commands run as root on the host.
# jailops doctor # jailops network show NAME # tcpdump -n -e -i vtnet0 'icmp6 and (ip6[40] == 135 or ip6[40] == 136)'
The first packets to a new jail are lost
- Cause
The router resolves a new address on its own timers. The proxy announces each address when it appears, which usually avoids the wait.
- Fix
Usually nothing to fix: it settles by itself once the router has resolved the address. If it does not, see the entry above.
A jail starts and has no network
- Cause
Forwarding is off for a routed network, a kernel module is missing, or the pf anchors are missing.
- Fix
Look at the network's conditions and the doctor's
sysctl.*,kernel.module.*andpf.anchorslines.These commands run as root on the host.
# jailops network show NAME # jailops doctor # sysctl net.inet.ip.forwarding net.inet6.ip6.forwarding
A service in a jail is not reachable from the internet
- Cause
The host's
pf.confblocks inbound traffic on the uplink by default. Unless you handpf.confto the optional host firewall module, JailOps does not edit that file, so it does not open the door for you.- Fix
Admit the jails' addresses and only the ports you want, with a rule of your own. The jail's own anchor still decides what reaches the jail.
pass in quick on vtnet0 inet6 proto tcp from any to 2001:db8:1:2:100::/80 port 8080
Jails behind IPv4 NAT lose most of their packets
- Cause
On a host whose
pf.confblocks by default and NATs the jails, forwarded packets were translated early by pf's floating state and then dropped.- Fix
Put
set state-policy if-boundat the top ofpf.confand reload.
Your SSH session drops when you reload the firewall
- Cause
service pf restartandpfctl -F statesflush the state table, your session included.- Fix
Check the file, then load it in place.
# pfctl -nf /etc/pf.conf # pfctl -f /etc/pf.conf
A jail cannot reach a jail of another project
- Cause
By design: projects are closed to each other, like an external host. Only addresses cross.
- Fix
Open exactly what you need, with the addresses written on both sides: an address set and a policy rule in the project that serves, and a rule that allows the address out in the project that connects.
$ jailops firewall policy create web --default-in deny --default-out allow $ jailops firewall rule add web in --from set:office --port 22 \ --comment "ssh from the office"
network N is already granted to project P
network N is already granted to project P- Cause
A network belongs to exactly one project. Storage may be shared; networks may not.
- Fix
Use another network, or revoke the grant first.
$ jailops network revoke NETWORK --project PROJECT
jailops doctor fails network.uplink
jailops doctor fails network.uplink- Cause
The uplink of a proxy-mode network is the physical parent of a vlan, a bridge member, or a virtual interface (lo, epair, tun, gif, wg, lagg).
- Fix
Name the interface the segment is on: a physical interface without vlan children, or the vlan(4) interface itself.
Apps
Installing from the catalog, and writing a definition of your own.
An app's service start never returns
- Cause
The service's rc script starts its daemon through daemon(8) without
-f, and the supervisor holds the pipes of the command that started it.- Fix
The installer starts services through a fixed wrapper that sends the rc script's output to a temporary file in the jail and prints it only when the start fails. Upgrade JailOps. In an rc script of your own, start the daemon with
daemon -f.
PostgreSQL's initdb stops at shmget: Failed system call
initdb stops at shmget: Failed system call- Cause
A jail has no System V shared memory, semaphores or message queues unless its parameters allow them.
- Fix
In a definition,
sysvipc: true. For an existing jail, the key of the same name; a restart applies it. The jail gets namespaces of its own (new), never the host's.$ jailops config set JAIL sysvipc=true --restart
A service cannot write its log or pid directory: it belongs to root
- Cause
pkg -rinstalls from the host, where the package's user does not exist, so its directories come out root-owned (or owned by a number that means something else). pkg's database records no owners, so nothing can restore them generically. Data volumes start root-owned too.- Fix
In a definition, give the directories to their user in
setup, withmkdir -pfirst. By hand, the same in the jail.# in a definition setup: - [/bin/mkdir, -p, /var/log/valkey, /var/run/valkey] - [/usr/sbin/chown, "valkey:valkey", /var/db/valkey, /var/log/valkey, /var/run/valkey] $ jailops exec JAIL -- chown valkey:valkey /var/log/valkey
Shared object "..." not found after a package went in with pkg -r
Shared object "..." not found after a package went in with pkg -r- Cause
pkg -rdoes not run the jail's library trigger, so the running jail's library cache does not know the new libraries.- Fix
jailops app installrefreshes it for you. After apkg -rof your own, refresh it in the jail.$ jailops exec JAIL -- /sbin/ldconfig -m /usr/local/lib
nginx answers, but its welcome page is missing
- Cause
The package ships its default site as
/usr/local/www/nginx-distand links/usr/local/www/nginxto it in a post-install step of its package list. Installed from the host into the jail, that step did not take effect.- Fix
The catalog's nginx definition makes the directory and copies the default site in its
setup. By hand:$ jailops exec nginx -- mkdir -p /usr/local/www/nginx $ jailops exec nginx -- cp -Rn /usr/local/www/nginx-dist/. /usr/local/www/nginx/
ssh-keygen is not found in an app jail
ssh-keygen is not found in an app jail- Cause
The thin FreeBSD image has no
ssh-keygen.- Fix
List
openssh-portableas a package of the definition (from the FreeBSD repository), as the catalog's isms does; the program is then/usr/local/bin/ssh-keygen.
Postfix logs improper command pipelining, or Dovecot rejects its configuration
improper command pipelining, or Dovecot rejects its configuration- Cause
Postfix refuses commands that arrive before its
220greeting: a probe or script that talks first is cut off. Dovecot's configuration syntax differs between major versions, and a file written for one is rejected by the other.- Fix
Make clients wait for the greeting. Write Dovecot's configuration for the version the package installs, and check it in the jail before you start the service.
$ jailops exec mail -- /usr/local/bin/doveconf -n $ jailops exec mail -- /usr/local/sbin/postfix check
An app is listed as invalid
- Cause
The definition fails validation: the reason is in the list. A definition written for a newer JailOps (a key, a format version or a category this daemon does not know) is invalid until JailOps is upgraded.
- Fix
Read the reason, check the files offline, and upgrade JailOps when the definition needs it.
These commands run as root on the host.
# jailops app list # jailops app validate /usr/local/share/jailops/apps # pkg upgrade jailops
jailops app install or jailops create says there is no image
jailops app install or jailops create says there is no image- Cause
Without
--imagethe installer takes the FreeBSD image of the newest release the host can run (of two images of the same release, the first by name). There is no FreeBSD image yet, or every one is of a newer release than the host.jailops createnames its image, and that image does not exist. The hint names the commands that make the image, with the host's release filled in, or lists the images there are.- Fix
Create and build the image of the host's release (the console: Images, Create image), or name an image that exists with
--image. An image is one FreeBSD release:freebsd-15-1is 15.1-RELEASE.# jailops image create freebsd-15-1 --release 15.1-RELEASE # jailops image build freebsd-15-1 # jailops app install nginx --image freebsd-15-1 --network lan --storage default
An installation failed and the jail is stopped
- Cause
A step after the jail's first start failed: a package, a setup command, a service, or the health check within its wait. The jail is kept for you to look inside.
- Fix
The error names the step. Look inside, then delete the jail and try again.
$ jailops start JAIL $ jailops jail logs JAIL $ jailops shell JAIL $ jailops delete JAIL
An app from the JailOps repository fails at the package step
- Cause
The definition installs with
-r jailops, and the host's pkg configuration has no repository namedjailops. The installer does not check beforehand; the step fails with pkg's own message.- Fix
Configure the repository as the packages repository's README shows, then delete the kept jail and install again.
busy: the app is being installed
busy: the app is being installed- Cause
While an installation holds the jail, every exec, shell and console session is refused:
pkg -rworks on the jail's root from the host.- Fix
Wait for the installation to finish.
Upgrades and backup
Going forward, going back, and the backups.
An app broke after an update
- Cause
A failed package update is not rolled back by itself. The update recorded a snapshot first, and the install left
app-installed.- Fix
Find the snapshot in the update history, or go back to the fresh installation. Both discard what the jail wrote since: the command shows how much and asks you to type the jail name.
$ jailops update history JAIL $ jailops jail rollback JAIL --all --snapshot SNAPSHOT $ jailops jail rollback JAIL --all --snapshot app-installed
The nightly upgrade of an image failed
- Cause
A jail that moved to the new generation failed its health check, or its
/etccould not be merged. That jail was put back on its old generation, and the run stopped there: the jails after it were not attempted. The image card in the console keeps the failure as a notice until a later run succeeds.- Fix
jailops image show IMAGElists the run and names the jail; the jail's update history says what happened. Fix the jail, then run the upgrade again, or let the next night do it.$ jailops image show freebsd-15-1 $ jailops update history JAIL --project PROJECT $ jailops image upgrade freebsd-15-1
data_loss_unconfirmed
data_loss_unconfirmed- Cause
The operation would discard data and was sent without confirmation.
- Fix
Check what would be lost (the CLI shows the bytes written since the snapshot), then type the jail name, or pass
--confirm-data-lossin a script.-ydoes not apply to data loss.
merge_conflict during an upgrade
merge_conflict during an upgrade- Cause
The
/etcmerge of the new generation needs a person. The jail stays on the old generation.- Fix
Resolve the listed files in the jail, then upgrade again.
$ jailops jail upgrade JAIL
image_in_use
image_in_use- Cause
A jail still uses the image or generation. The error names the jails; an alias alone never blocks.
- Fix
Upgrade or delete those jails, then retry.
chain_broken
chain_broken- Cause
A backup chain has a missing or corrupt link.
- Fix
Verify the backup, then take a new full backup.
$ jailops backup verify BACKUP
key_not_escrowed
key_not_escrowed- Cause
A raw backup was refused because the backup key has no escrow.
- Fix
Escrow the key to a recipient you control.
$ jailops backup key escrow --recipient age1...
jailops doctor fails zfs.compat
jailops doctor fails zfs.compat- Cause
The pool has features beyond
zfs.compat_file, so replication to an older host would fail.- Fix
Check the pool features on both ends. JailOps never runs
zpool upgrade; do not run it on a pool that replicates to an older host.
jailops doctor fails db.integrity
jailops doctor fails db.integrity- Cause
The daemon's SQLite database is damaged.
- Fix
Restore from the latest copy in
<data_dir>/backups/, which the daemon writes withVACUUM INTO.
jailops doctor fails operations.journal
jailops doctor fails operations.journal- Cause
A rollback did not complete.
- Fix
Inspect the datasets of the jails it names, then remove the journal record.
Not here? Ask.
Open an issue with the command, its output and what jailops doctor says. The troubleshooting guide in the repository has every doctor check.