Articles

One small cloud server, many services, solid IPv6

Hardware is expensive; a small cloud server running JailOps is not. How to run many services on a netcup-style server with real IPv6 for every jail, and the one FreeBSD setting that stops the random unreachable alerts.

Why one small server

Pay for what you use.

A staging environment, an internal tool, a small production service: each one usually gets its own machine because that is the easy way to keep them apart. Jails keep them apart on one machine without the second and third bill: they share the host's kernel, so for services you run yourself, not for tenants who must be protected from each other. Every jail has its own filesystem, its own firewall rules, its own addresses, and ZFS snapshots and backups per jail. One server with a few cores and some RAM holds a dozen of them, and JailOps manages them from one place: a console, a command line, and a Git repository if you want the whole fleet declared.

Cloud providers like netcup sell exactly the right machine for this: a generous /64 of IPv6, fast local storage, a price you stop thinking about. The one thing to know is how that /64 is delivered, because it decides how the jails get their addresses.

The setup

A /80 inside the shared /64.

On a netcup-style network the /64 sits on a segment shared with other customers, and the provider's routers ask who has each address. JailOps answers for every jail address itself, so each jail gets a real, routable IPv6 address with no NAT and no extra daemon. Create a network in proxy mode and start a jail:

a switched IPv6 network
$ jailops network create web --ipv4 10.94.0.1/24 --nat --uplink vtnet0 \
    --ipv6 2001:db8:1:2:100::/80 --ipv6-mode proxy
$ jailops create site --image freebsd-15-1 --network web --storage default --start
$ jailops list

The full walk-through, including the inbound firewall rule, is on the IPv6 page.

The random unreachable alerts

Short, regular, and not your fault.

On a switched segment a stock FreeBSD host can show up as unreachable over IPv6 for a few minutes every few hours, and an uptime monitor reports it. The server is fine. “Host is unreachable” comes from the provider's router: it could not find your server on the segment.

The reason is neighbor discovery. When its entry for your address goes stale, the router checks it with a unicast neighbor solicitation sent from its own global address. FreeBSD, by default, does not answer a solicitation from a sender it does not consider on-link. Your address is a /128 and no router advertisement describes the prefix, so the probe is dropped and counted as a bad neighbor solicitation. The router's multicast question from its link-local address is answered, so everything works again until the next check.

The whole story, with the captures to take on your own server and what jailops doctor says, is in JailOps on a netcup server, with IPv6 that stays up.

The fix

One sysctl.

accept neighbor solicitations from off-link senders
# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1
# echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf

With the setting on, FreeBSD answers the routers' probes, and it still requires the packet to have come from the local link (hop limit 255), so nothing from further away gets in. Every probe is then answered within a fraction of a millisecond and the counter stops growing. Set it on any FreeBSD host that sits on a switched IPv6 segment, with or without JailOps; on a host with a proxy-mode network, jailops doctor warns until you have.

Next

Start small.

Install JailOps on one server, move one service into a jail, and see the console. The quick start takes a fresh FreeBSD 15.1 install to a running jail, and the fleet examples show what a few servers look like when they are declared in Git.

More articles

Try it on one server.

Try the console from the screenshots, install on a clean FreeBSD 15.1 machine, or add JailOps to a host you already run.