Articles

JailOps on a netcup server, with IPv6 that stays up

Running FreeBSD jails on a netcup server with a public IPv6 address for every jail, and the cause and fix of the intermittent "Host is unreachable": one FreeBSD sysctl, how to check it on your own server, and what jailops doctor says.

The network

A /64 on the link, two routers, one virtual gateway.

A netcup server comes with a /64 of IPv6, and that is all a jail host needs: one address for the host, and a public address for every jail out of the same prefix. No NAT, no port juggling, every service reachable at an address of its own.

The prefix is delivered on the link. netcup's routers treat every address in the /64 as a neighbour on the segment and find it with neighbour discovery: who has this address? They come as a pair that shares one virtual gateway address with VRRP, so either can carry your traffic, and each keeps its own view of who is on the segment.

The host answers for its own address. The jails sit behind it, each on a link of its own, and never hear the question, so in proxy mode JailOps answers for them: for each running jail's address it replies with the host's own MAC address, the routers send the traffic to the host, and the host routes it to the jail through its firewall.

Addresses from the documentation range. The jails take a /80 inside the /64, approved once on the host.

The symptom

Minutes of silence, every few hours.

Everything works, and then an uptime monitor reports Host is unreachable over IPv6. A few minutes later it is back on its own. It happens again a few hours later, and the jails go dark together with the host. There is nothing in the host's logs, because nothing on the host failed.

The message is the router's: it tried to deliver a packet to your address and could not find you on the segment. So the question is why a router that found you a minute ago stops finding you.

The cause

We found it, and we tested the fix.

A router does not ask once and trust the answer for ever. When its entry for your address gets old, it checks that you are still there with a unicast neighbour solicitation sent straight to your address. netcup's routers send those probes from their global addresses.

FreeBSD, by default, does not answer a neighbour solicitation from a sender it does not consider on-link. The host holds its address as a /128 and no router advertisement describes the /64, so for FreeBSD a router's global address is not a neighbour. With the default net.inet6.icmp6.nd6_onlink_ns_rfc4861=0 the probe is dropped and counted as a bad neighbor solicitation.

  1. The router's entry for your address goes stale and it probes from its global address.
  2. The host drops every probe. The router hears nothing and marks your address failed.
  3. Packets for the host and its jails now come back as unreachable.
  4. Later the router asks again the other way, with a multicast solicitation from its link-local address. FreeBSD answers that one, and everything works again until the next round.

That is why the outage is short, regular and quiet: the host never sees anything wrong, and the router is only following the protocol.

The fix

One sysctl.

Tell FreeBSD to answer solicitations from senders off the prefix, as RFC 4861 describes. The packet must still arrive with hop limit 255, so it can only come from the local link: nothing from further away gets in.

now, and at every boot
# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0 -> 1
# echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf

The first line takes effect at once. The second makes it survive a reboot: /etc/sysctl.conf is read at boot by the base system's sysctl rc script, so there is nothing else to enable.

After the change every probe from the routers was answered in well under a millisecond, the counter of bad solicitations stopped growing, and in more than a day of monitoring there was not one incident. Set it on any FreeBSD host on a switched IPv6 segment, with or without JailOps.

Check your own server

Four commands, two minutes.

The setting. 0 is the default and the problem; 1 is fixed.

$ sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0

The counter. Run it twice, some minutes apart. If it grows on a host with the setting at 0, the routers' probes are being dropped.

$ netstat -s -p icmp6 | grep 'bad neighbor solicitation'

The neighbours. ndp -an prints the host's neighbour cache without name lookups. Your default router is there; note its link-local address and MAC, they are the other half of the conversation you are about to watch.

$ ndp -an

The probes. Watch neighbour solicitations (ICMPv6 type 135) and advertisements (type 136) for the host's own address on the uplink. Replace vtnet0 and the address with yours.

as root, Ctrl-C to stop
# tcpdump -n -e -i vtnet0 'icmp6 and (ip6[40] == 135 or ip6[40] == 136) and host 2001:db8:1:2::2'

The filter reads the ICMPv6 type at offset 40, right after the fixed IPv6 header, which is where it sits in neighbour discovery packets. What to look for: a solicitation from a router's global address for 2001:db8:1:2::2. With the setting at 0, no advertisement from your host follows it. With 1, the advertisement comes right after it, every time. Solicitations from fe80:: addresses are answered either way; they are not the problem.

In jailops doctor

It warns before the monitor does.

JailOps never sets the sysctl itself: it is a decision about your host. It tells you. On a host with a network in proxy mode, jailops doctor warns while the setting is 0, and warns again while it is not in /etc/sysctl.conf:

jailops doctor, the two lines that matter here
$ jailops doctor
CHECK                                                  STATUS   DETAIL
server: sysctl.net.inet6.icmp6.nd6_onlink_ns_rfc4861   WARN     net.inet6.icmp6.nd6_onlink_ns_rfc4861 is 0, JailOps recommends 1 (the routers of a shared IPv6 segment probe the host from off-link addresses) (the routers of a shared IPv6 segment probe the host's address from their global addresses and FreeBSD drops those solicitations as off-link, so the host (and its jails) is unreachable over IPv6 for minutes every few hours; run `sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1` and add `net.inet6.icmp6.nd6_onlink_ns_rfc4861=1` to /etc/sysctl.conf (JailOps never sets it))
server: sysctl.persistent                              WARN     net.inet6.icmp6.nd6_onlink_ns_rfc4861 is not set to 1 in /etc/sysctl.conf or /etc/rc.conf; after a reboot the host stops answering the probes of the routers of its shared IPv6 segment (add net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 to /etc/sysctl.conf (JailOps never edits it))

Creating a proxy-mode network says the same in its answer, so you hear about it the moment it starts to matter. With both lines in place the two checks pass:

CHECK                                                  STATUS   DETAIL
server: sysctl.net.inet6.icmp6.nd6_onlink_ns_rfc4861   PASS     net.inet6.icmp6.nd6_onlink_ns_rfc4861=1
server: sysctl.persistent                              PASS     forwarding and net.inet6.icmp6.nd6_onlink_ns_rfc4861 are persistent (sysctl.conf or rc.conf)

The whole setup

From a FreeBSD host to jails with public IPv6.

On a netcup server with FreeBSD 15.1 and JailOps installed (the quick start), the IPv6 part is three steps.

These commands run as root on the host.

a proxy-mode network and a jail on it
# 1. answer the routers' probes
# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1
# echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf

# 2. a network whose jails take addresses from the /80
# jailops network create web --ipv4 10.94.0.1/24 --nat --uplink vtnet0 \
    --ipv6 2001:db8:1:2:100::/80 --ipv6-mode proxy

# 3. a jail on it
# jailops create site --image freebsd-15-1 --network web --storage default --start --autostart
# jailops network show web

The network's NDPProxy condition says whether the proxy is answering; right after the daemon starts it is Starting while the capture of the uplink opens. To let the internet in, add a rule for the jails' prefix and the ports you want to your own pf.conf, for example pass in quick on vtnet0 inet6 proto tcp from any to 2001:db8:1:2:100::/80 port 443. Without the optional host firewall module JailOps does not edit that file, and each jail's own anchor still decides what reaches it.

More on both IPv6 modes is on the IPv6 page, and the short version of this story is in one small cloud server, many services. If something else goes wrong, the help page has the networking answers.

Why we care about this

The details are the product.

A setting like this is why we build JailOps for FreeBSD and nothing else. Jails, ZFS, pf and IPv6 are not checkboxes to us: we read the packets, find the line in the stack that decides, and then make the tool tell you about it before your monitor does. If you run FreeBSD on a small cloud server, you should not have to find this yourself.

More articles

Try it on one server.

Try the console from the screenshots, install on a clean FreeBSD 15.1 machine, or add JailOps to a host you already run.