JailOps on a netcup server, with IPv6 that stays up
Running FreeBSD jails on a netcup server with a public IPv6 address for every jail, and the cause and fix of the intermittent "Host is unreachable": one FreeBSD sysctl, how to check it on your own server, and what jailops doctor says.
The network
A /64 on the link, two routers, one virtual gateway.
A netcup server comes with a /64 of IPv6, and that is all a jail host needs: one address for the host, and a public address for every jail out of the same prefix. No NAT, no port juggling, every service reachable at an address of its own.
The prefix is delivered on the link. netcup's routers treat every address in the /64 as a neighbour on the segment and find it with neighbour discovery: who has this address? They come as a pair that shares one virtual gateway address with VRRP, so either can carry your traffic, and each keeps its own view of who is on the segment.
The host answers for its own address. The jails sit behind it, each on a link of its own, and never hear the question, so in proxy mode JailOps answers for them: for each running jail's address it replies with the host's own MAC address, the routers send the traffic to the host, and the host routes it to the jail through its firewall.
Addresses from the documentation range. The jails take a /80 inside the /64, approved once on the host.
The symptom
Minutes of silence, every few hours.
Everything works, and then an uptime monitor reports Host is unreachable over IPv6. A few minutes later it is back on its own. It happens again a few hours later, and the jails go dark together with the host. There is nothing in the host's logs, because nothing on the host failed.
The message is the router's: it tried to deliver a packet to your address and could not find you on the segment. So the question is why a router that found you a minute ago stops finding you.
The cause
We found it, and we tested the fix.
A router does not ask once and trust the answer for ever. When its entry for your address gets old, it checks that you are still there with a unicast neighbour solicitation sent straight to your address. netcup's routers send those probes from their global addresses.
FreeBSD, by default, does not answer a neighbour solicitation from a sender it does not consider on-link. The host holds its address as a /128 and no router advertisement describes the /64, so for FreeBSD a router's global address is not a neighbour. With the default net.inet6.icmp6.nd6_onlink_ns_rfc4861=0 the probe is dropped and counted as a bad neighbor solicitation.
- The router's entry for your address goes stale and it probes from its global address.
- The host drops every probe. The router hears nothing and marks your address failed.
- Packets for the host and its jails now come back as unreachable.
- Later the router asks again the other way, with a multicast solicitation from its link-local address. FreeBSD answers that one, and everything works again until the next round.
That is why the outage is short, regular and quiet: the host never sees anything wrong, and the router is only following the protocol.
The fix
One sysctl.
Tell FreeBSD to answer solicitations from senders off the prefix, as RFC 4861 describes. The packet must still arrive with hop limit 255, so it can only come from the local link: nothing from further away gets in.
# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0 -> 1 # echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf
The first line takes effect at once. The second makes it survive a reboot: /etc/sysctl.conf is read at boot by the base system's sysctl rc script, so there is nothing else to enable.
After the change every probe from the routers was answered in well under a millisecond, the counter of bad solicitations stopped growing, and in more than a day of monitoring there was not one incident. Set it on any FreeBSD host on a switched IPv6 segment, with or without JailOps.
Check your own server
Four commands, two minutes.
The setting. 0 is the default and the problem; 1 is fixed.
$ sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0The counter. Run it twice, some minutes apart. If it grows on a host with the setting at 0, the routers' probes are being dropped.
$ netstat -s -p icmp6 | grep 'bad neighbor solicitation'The neighbours. ndp -an prints the host's neighbour cache without name lookups. Your default router is there; note its link-local address and MAC, they are the other half of the conversation you are about to watch.
$ ndp -anThe probes. Watch neighbour solicitations (ICMPv6 type 135) and advertisements (type 136) for the host's own address on the uplink. Replace vtnet0 and the address with yours.
# tcpdump -n -e -i vtnet0 'icmp6 and (ip6[40] == 135 or ip6[40] == 136) and host 2001:db8:1:2::2'The filter reads the ICMPv6 type at offset 40, right after the fixed IPv6 header, which is where it sits in neighbour discovery packets. What to look for: a solicitation from a router's global address for 2001:db8:1:2::2. With the setting at 0, no advertisement from your host follows it. With 1, the advertisement comes right after it, every time. Solicitations from fe80:: addresses are answered either way; they are not the problem.
In jailops doctor
It warns before the monitor does.
JailOps never sets the sysctl itself: it is a decision about your host. It tells you. On a host with a network in proxy mode, jailops doctor warns while the setting is 0, and warns again while it is not in /etc/sysctl.conf:
$ jailops doctor CHECK STATUS DETAIL server: sysctl.net.inet6.icmp6.nd6_onlink_ns_rfc4861 WARN net.inet6.icmp6.nd6_onlink_ns_rfc4861 is 0, JailOps recommends 1 (the routers of a shared IPv6 segment probe the host from off-link addresses) (the routers of a shared IPv6 segment probe the host's address from their global addresses and FreeBSD drops those solicitations as off-link, so the host (and its jails) is unreachable over IPv6 for minutes every few hours; run `sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1` and add `net.inet6.icmp6.nd6_onlink_ns_rfc4861=1` to /etc/sysctl.conf (JailOps never sets it)) server: sysctl.persistent WARN net.inet6.icmp6.nd6_onlink_ns_rfc4861 is not set to 1 in /etc/sysctl.conf or /etc/rc.conf; after a reboot the host stops answering the probes of the routers of its shared IPv6 segment (add net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 to /etc/sysctl.conf (JailOps never edits it))
Creating a proxy-mode network says the same in its answer, so you hear about it the moment it starts to matter. With both lines in place the two checks pass:
CHECK STATUS DETAIL
server: sysctl.net.inet6.icmp6.nd6_onlink_ns_rfc4861 PASS net.inet6.icmp6.nd6_onlink_ns_rfc4861=1
server: sysctl.persistent PASS forwarding and net.inet6.icmp6.nd6_onlink_ns_rfc4861 are persistent (sysctl.conf or rc.conf)The whole setup
From a FreeBSD host to jails with public IPv6.
On a netcup server with FreeBSD 15.1 and JailOps installed (the quick start), the IPv6 part is three steps.
These commands run as root on the host.
# 1. answer the routers' probes # sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 # echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf # 2. a network whose jails take addresses from the /80 # jailops network create web --ipv4 10.94.0.1/24 --nat --uplink vtnet0 \ --ipv6 2001:db8:1:2:100::/80 --ipv6-mode proxy # 3. a jail on it # jailops create site --image freebsd-15-1 --network web --storage default --start --autostart # jailops network show web
The network's NDPProxy condition says whether the proxy is answering; right after the daemon starts it is Starting while the capture of the uplink opens. To let the internet in, add a rule for the jails' prefix and the ports you want to your own pf.conf, for example pass in quick on vtnet0 inet6 proto tcp from any to 2001:db8:1:2:100::/80 port 443. Without the optional host firewall module JailOps does not edit that file, and each jail's own anchor still decides what reaches it.
More on both IPv6 modes is on the IPv6 page, and the short version of this story is in one small cloud server, many services. If something else goes wrong, the help page has the networking answers.
Why we care about this
The details are the product.
A setting like this is why we build JailOps for FreeBSD and nothing else. Jails, ZFS, pf and IPv6 are not checkboxes to us: we read the packets, find the line in the stack that decides, and then make the tool tell you about it before your monitor does. If you run FreeBSD on a small cloud server, you should not have to find this yourself.
More articles
Jail updates without cron
Replace the nightly script that upgrades every jail: automatic jail upgrades per image, with a health check, a rollback and a record of every run, what they leave alone, and how a new FreeBSD release comes in.
Read the article: Jail updates without cronJail search and shell
Ctrl+K, a few letters, Shift+Enter: find a jail by name, label or app and open its shell. Labels that filter the jail list, bash in new jails, and the jailops command next to every step.
Read the article: Jail search and shellGetting started with the JailOps CLI
Root on the host, the command line and nothing else: install JailOps, build an image, create a network and a jail, reach it by name, snapshot it, open a port and install an app.
Read the article: Getting started with the JailOps CLIOne small cloud server, many services, solid IPv6
Run your database, web tier and workers on one low-cost server, each jail with its own IPv6 address, and fix the alerts a shared /64 causes with one sysctl.
Read the article: One small cloud server, many services, solid IPv6Write your own app: memcached, from package to catalog
Turn a small FreeBSD package into a JailOps app: the definition line by line, the offline check, a local install, the five traps, and the pull request that puts it in the catalog.
Read the article: Write your own app: memcached, from package to catalogFrom catalog to running service in one command
Install nginx from the app catalog with one command, see each of its steps, and find the data, the configuration, the logs and the snapshot afterwards.
Read the article: From catalog to running service in one commandA firewall you can read
Policies, address sets and jails by name (jail:web), each with a comment, so the firewall can be reviewed by someone who did not write it.
Read the article: A firewall you can readPostgreSQL and git, closed except for who needs them
Close a database and a git server, then let in exactly the web jail and the office, with a comment on every entry.
Read the article: PostgreSQL and git, closed except for who needs themProjects are boundaries
A project is a boundary. Another project is an external host, so only addresses cross it, written out on both sides. Why that is simpler to review, and what a project does not isolate.
Read the article: Projects are boundariesThe life of a jail
A jail starts at boot only when you declared it with --autostart. Why JailOps does it this way, how the order is decided, and how to see it with jailops boot.
Read the article: The life of a jailTab completion for jailops
Press Tab and get the names of your jails, networks and firewall policies, even on another host. How to install and turn on completion in bash, zsh and fish.
Read the article: Tab completion for jailopsTest the firewall before you trust it
Ask the firewall whether a connection would pass, and why, before you try it. Read the answer, put it in a script, and know what it cannot tell you.
Read the article: Test the firewall before you trust itSee a connection before it happens
Pick a From and a To, a protocol and a port, press Run, and see the connection flow through the firewall: what passes, what stops it, and which rule decides.
Read the article: See a connection before it happensLimit what a jail can use
Give a jail a ceiling for memory, processes, CPU, disk, firewall states and bandwidth, one command each, and know what happens when it reaches it.
Read the article: Limit what a jail can useStop every jail cleanly when the host shuts down
A host that reboots should stop its databases before it stops the disks. How JailOps stops the jails in order, and how to start them in order again.
Read the article: Stop every jail cleanly when the host shuts downOne line in pf.conf
The host's firewall stays yours. JailOps asks for one line, and everything for the jails lives below it.
Read the article: One line in pf.confWhat the host tells you by itself
A pool that reports an error, an uplink that goes down, a jail that reaches a limit: they show up in the event stream by themselves.
Read the article: What the host tells you by itselfNetworks the host controls
No bridge, no DHCP. The host hands out the addresses, gives each jail a link of its own and is the router, so jails cannot see or impersonate each other.
Read the article: Networks the host controlsVolumes: data that outlives a jail
Keep the data in a volume and the jail disposable. One volume can be attached to several jails, has its own snapshots and its own backup.
Read the article: Volumes: data that outlives a jailBack up volumes once
One copy of a volume, however many jails use it. Leave volumes out when you want to, bring them back with the jail.
Read the article: Back up volumes onceA private Docker registry
A registry of your own with users in a file: one jail, docker-registry and docker-auth, htpasswd for the passwords, the images on a volume.
Read the article: A private Docker registryNames for jails
A jail is reached by its name inside its project: db.shop.internal, or just db. Create a jail and its name is there; delete it and the name is gone.
Read the article: Names for jails
Try it on one server.
Try the console from the screenshots, install on a clean FreeBSD 15.1 machine, or add JailOps to a host you already run.