One small cloud server, many services, solid IPv6
Hardware is expensive; a small cloud server running JailOps is not. How to run many services on a netcup-style server with real IPv6 for every jail, and the one FreeBSD setting that stops the random unreachable alerts.
Why one small server
Pay for what you use.
A staging environment, an internal tool, a small production service: each one usually gets its own machine because that is the easy way to keep them apart. Jails keep them apart on one machine without the second and third bill: they share the host's kernel, so for services you run yourself, not for tenants who must be protected from each other. Every jail has its own filesystem, its own firewall rules, its own addresses, and ZFS snapshots and backups per jail. One server with a few cores and some RAM holds a dozen of them, and JailOps manages them from one place: a console, a command line, and a Git repository if you want the whole fleet declared.
Cloud providers like netcup sell exactly the right machine for this: a generous /64 of IPv6, fast local storage, a price you stop thinking about. The one thing to know is how that /64 is delivered, because it decides how the jails get their addresses.
The setup
A /80 inside the shared /64.
On a netcup-style network the /64 sits on a segment shared with other customers, and the provider's routers ask who has each address. JailOps answers for every jail address itself, so each jail gets a real, routable IPv6 address with no NAT and no extra daemon. Create a network in proxy mode and start a jail:
$ jailops network create web --ipv4 10.94.0.1/24 --nat --uplink vtnet0 \ --ipv6 2001:db8:1:2:100::/80 --ipv6-mode proxy $ jailops create site --image freebsd-15-1 --network web --storage default --start $ jailops list
The full walk-through, including the inbound firewall rule, is on the IPv6 page.
The random unreachable alerts
Short, regular, and not your fault.
On a switched segment a stock FreeBSD host can show up as unreachable over IPv6 for a few minutes every few hours, and an uptime monitor reports it. The server is fine. “Host is unreachable” comes from the provider's router: it could not find your server on the segment.
The reason is neighbor discovery. When its entry for your address goes stale, the router checks it with a unicast neighbor solicitation sent from its own global address. FreeBSD, by default, does not answer a solicitation from a sender it does not consider on-link. Your address is a /128 and no router advertisement describes the prefix, so the probe is dropped and counted as a bad neighbor solicitation. The router's multicast question from its link-local address is answered, so everything works again until the next check.
The whole story, with the captures to take on your own server and what jailops doctor says, is in JailOps on a netcup server, with IPv6 that stays up.
The fix
One sysctl.
# sysctl net.inet6.icmp6.nd6_onlink_ns_rfc4861=1 # echo 'net.inet6.icmp6.nd6_onlink_ns_rfc4861=1' >> /etc/sysctl.conf
With the setting on, FreeBSD answers the routers' probes, and it still requires the packet to have come from the local link (hop limit 255), so nothing from further away gets in. Every probe is then answered within a fraction of a millisecond and the counter stops growing. Set it on any FreeBSD host that sits on a switched IPv6 segment, with or without JailOps; on a host with a proxy-mode network, jailops doctor warns until you have.
Next
Start small.
Install JailOps on one server, move one service into a jail, and see the console. The quick start takes a fresh FreeBSD 15.1 install to a running jail, and the fleet examples show what a few servers look like when they are declared in Git.
More articles
Jail updates without cron
Replace the nightly script that upgrades every jail: automatic jail upgrades per image, with a health check, a rollback and a record of every run, what they leave alone, and how a new FreeBSD release comes in.
Read the article: Jail updates without cronJail search and shell
Ctrl+K, a few letters, Shift+Enter: find a jail by name, label or app and open its shell. Labels that filter the jail list, bash in new jails, and the jailops command next to every step.
Read the article: Jail search and shellGetting started with the JailOps CLI
Root on the host, the command line and nothing else: install JailOps, build an image, create a network and a jail, reach it by name, snapshot it, open a port and install an app.
Read the article: Getting started with the JailOps CLIJailOps on a netcup server, with IPv6 that stays up
A /64, a router pair and a FreeBSD default that drops their probes. How every jail gets its own IPv6 address on netcup, and the one sysctl that ends the random unreachable alerts.
Read the article: JailOps on a netcup server, with IPv6 that stays upWrite your own app: memcached, from package to catalog
Turn a small FreeBSD package into a JailOps app: the definition line by line, the offline check, a local install, the five traps, and the pull request that puts it in the catalog.
Read the article: Write your own app: memcached, from package to catalogFrom catalog to running service in one command
Install nginx from the app catalog with one command, see each of its steps, and find the data, the configuration, the logs and the snapshot afterwards.
Read the article: From catalog to running service in one commandA firewall you can read
Policies, address sets and jails by name (jail:web), each with a comment, so the firewall can be reviewed by someone who did not write it.
Read the article: A firewall you can readPostgreSQL and git, closed except for who needs them
Close a database and a git server, then let in exactly the web jail and the office, with a comment on every entry.
Read the article: PostgreSQL and git, closed except for who needs themProjects are boundaries
A project is a boundary. Another project is an external host, so only addresses cross it, written out on both sides. Why that is simpler to review, and what a project does not isolate.
Read the article: Projects are boundariesThe life of a jail
A jail starts at boot only when you declared it with --autostart. Why JailOps does it this way, how the order is decided, and how to see it with jailops boot.
Read the article: The life of a jailTab completion for jailops
Press Tab and get the names of your jails, networks and firewall policies, even on another host. How to install and turn on completion in bash, zsh and fish.
Read the article: Tab completion for jailopsTest the firewall before you trust it
Ask the firewall whether a connection would pass, and why, before you try it. Read the answer, put it in a script, and know what it cannot tell you.
Read the article: Test the firewall before you trust itSee a connection before it happens
Pick a From and a To, a protocol and a port, press Run, and see the connection flow through the firewall: what passes, what stops it, and which rule decides.
Read the article: See a connection before it happensLimit what a jail can use
Give a jail a ceiling for memory, processes, CPU, disk, firewall states and bandwidth, one command each, and know what happens when it reaches it.
Read the article: Limit what a jail can useStop every jail cleanly when the host shuts down
A host that reboots should stop its databases before it stops the disks. How JailOps stops the jails in order, and how to start them in order again.
Read the article: Stop every jail cleanly when the host shuts downOne line in pf.conf
The host's firewall stays yours. JailOps asks for one line, and everything for the jails lives below it.
Read the article: One line in pf.confWhat the host tells you by itself
A pool that reports an error, an uplink that goes down, a jail that reaches a limit: they show up in the event stream by themselves.
Read the article: What the host tells you by itselfNetworks the host controls
No bridge, no DHCP. The host hands out the addresses, gives each jail a link of its own and is the router, so jails cannot see or impersonate each other.
Read the article: Networks the host controlsVolumes: data that outlives a jail
Keep the data in a volume and the jail disposable. One volume can be attached to several jails, has its own snapshots and its own backup.
Read the article: Volumes: data that outlives a jailBack up volumes once
One copy of a volume, however many jails use it. Leave volumes out when you want to, bring them back with the jail.
Read the article: Back up volumes onceA private Docker registry
A registry of your own with users in a file: one jail, docker-registry and docker-auth, htpasswd for the passwords, the images on a volume.
Read the article: A private Docker registryNames for jails
A jail is reached by its name inside its project: db.shop.internal, or just db. Create a jail and its name is there; delete it and the name is gone.
Read the article: Names for jails
Try it on one server.
Try the console from the screenshots, install on a clean FreeBSD 15.1 machine, or add JailOps to a host you already run.