Articles

Write your own app: memcached, from package to catalog

A worked tutorial: turn the FreeBSD memcached package into a JailOps app definition, validate it, install it on your own host, avoid the traps of installing from the host, and send it to the catalog.

What we build

A small, real package.

memcached is the right first app: one FreeBSD package (databases/memcached), one rc.d service, one port, 11211/tcp, and no state on disk. It keeps its cache in memory, so a restart starts empty and the app needs no data volume. Everything the definition says can be checked against the package.

Start with the facts, from the package and not from memory. On a FreeBSD host:

$ pkg rquery '%n %v: %c' memcached
# then read the rc script it installs, /usr/local/etc/rc.d/memcached

The rc script answers most questions: the service is memcached, it runs the server as nobody (a user every FreeBSD system has), it creates /var/run/memcached for its pid file at every start, and it reads extra options from memcached_flags. memcached listens on port 11211 and logs nothing to a file.

You need a host with JailOps, a project, a network and a storage, as in the quick start; below they are web, lan and default.

Start

A file to fill in.

jailops app new writes a commented starter definition, memcached.yaml, and an icon placeholder, memcached.svg, in the current directory (--dir DIR writes them elsewhere). The flags fill in what you already know:

$ jailops app new memcached --package memcached --service memcached --port 11211/tcp --category database

Every value you still have to decide is marked TODO, and jailops app validate fails until none is left, so a half-finished starter can never be installed by accident. --package takes several names separated by commas.

The definition

Format version 1, the format of the current release.

One YAML document, strict: a key the format does not know is an error, and the file name is the app name. This is the whole definition; jailops app validate accepts it as it stands.

memcached.yaml
# memcached from the FreeBSD packages (databases/memcached). Format: docs/apps.md, version 1.
# From the package's rc.d script: the server runs as nobody, with its pid file in
# /var/run/memcached, which the script creates at every start.
version: 1
name: memcached
title: memcached
summary: Distributed memory object cache, from the FreeBSD packages.
description: |
  memcached keeps key/value pairs in memory: a fast cache that several
  applications can share in front of a database.

  It writes nothing to disk, and a restart starts with an empty cache, so this
  app has no data volume. The FreeBSD rc script runs the server as nobody.
category: database
homepage: https://memcached.org/
license: BSD-3-Clause
image:
  os: freebsd
  release_min: "15.1"
packages:
  - name: memcached
    repo: ports
services:
  - name: memcached
    rc:
      memcached_flags: "-m 256"
    start: true
ports:
  - name: memcache
    port: 11211
    proto: tcp
    description: The memcached protocol, without authentication.
    expose: false
health:
  tcp: 11211
  wait_seconds: 30
resources:
  memory: 512M
  disk: 1G
  cpu: "1"
notes: |
  memcached answers on port 11211 of the jail's address, without a password:
  whoever reaches the port can read and change the cache. The port is
  reachable from the jail's network only; allow just the jails that need it
  with a firewall policy.

  The cache has 256 MB (-m 256). To change it:
  jailops exec NAME -- sysrc memcached_flags="-m 512"
  jailops exec NAME -- service memcached restart

What each part does, and why it says what it says:

  • version, name, title, summary, description. The name is a jail name and must match the file name. The summary is one line of at most 140 characters; the description is plain text.
  • category is one of a fixed list: database, web, tools, devops, storage, monitoring, communication, network, security, other.
  • image says FreeBSD, 15.1 or later. Unless you name one, the installer takes the FreeBSD image of the newest release the host runs.
  • packages: repo: ports is the FreeBSD repository the host already uses. Add version: with what pkg rquery '%v' memcached printed when you checked: it records what you wrote the definition against and is not a pin.
  • services: the rc.d name, and rc.conf knobs other than memcached_enable, which the installer writes itself. -m 256 gives the cache 256 MB.
  • ports: recorded, not forwarded. expose: false, because a cache without a password has no business on the outside.
  • No data, config, logs or setup. memcached keeps nothing on disk, has no configuration file, writes no log file, and its rc script prepares its own pid directory. A definition says only what is true.
  • health: a TCP connection from the host to port 11211 on the jail's address, retried for up to 30 seconds. The app counts as installed only once it passes.
  • resources are hints that become the jail's limits unless the installer is given others; 512 MB of memory leaves room above the 256 MB cache.
  • notes are printed after the installation: what the person has to know next.

Validate

Offline, no root, no server.

jailops app validate runs the daemon's own parser and validator on the files you name. It needs no server, no network and no root, so run it on your laptop as often as you like.

$ jailops app validate memcached.yaml
valid: 1 definition(s), 0 icon(s)

A mistake is printed as FILE: reason, and the exit status is 2. Writing memcached_enable yourself, for example:

$ jailops app validate memcached.yaml
memcached.yaml: services[0].rc: memcached_enable is written by the installer; leave it out
invalid: 1 problem(s) in 1 definition(s) checked
hint: fix the files listed and run 'jailops app validate' again; docs/apps.md has the format

Install it on your host

Your definition, a real jail.

Drop the file into the local apps directory, /usr/local/etc/jailops/apps (apps.local_dir in jailopsd.yaml). It is listed and installable at once, marked local, and a local definition with the name of a packaged one takes its place, so you can also try a fix to a catalog app this way.

These commands run as root on the host.

# cp memcached.yaml /usr/local/etc/jailops/apps/
# jailops app list memcached
# jailops app install memcached --network lan --storage default --dry-run
# jailops app install memcached --network lan --storage default

The dry run lists the steps and creates nothing. The installation then creates a thin jail, installs the package from the host, enables and starts the service, waits for the health check and takes the snapshot app-installed. Then talk to it from the host, with nc from the base system and the address jailops app installed shows:

$ jailops app installed memcached
$ printf 'stats\r\nquit\r\n' | nc 10.77.0.5 11211

The server answers with its STAT lines. If the installation fails after the jail first started, the jail is stopped and kept: jailops start memcached and jailops shell memcached to look inside, jailops delete memcached before you try again.

The traps

Five things that bite when packages go in from the host.

The installer runs pkg -r on the host, against the jail's root. That is safe and fast, and it is not the same as running pkg install inside the jail. Five differences decide whether a definition works the first time.

  1. Post-install steps do not always take effect. A package can leave work to its install steps that does not have the same result when it is installed from the host. nginx is the example: its default site is linked into place by such a step, and the catalog's nginx definition copies it in its setup instead. Read the package's list and install message, then check the result in the jail rather than assuming it. memcached has nothing of the kind.
  2. Ownership. A package that creates its own user gets its directories root-owned in the jail, because the user exists only in the jail's password file, and pkg's database records no owners that could be restored. Give them back in setup, as the catalog's Valkey does: [/bin/mkdir, -p, /var/log/valkey, /var/run/valkey], then [/usr/sbin/chown, "valkey:valkey", /var/db/valkey, /var/log/valkey, /var/run/valkey]. Data volumes start root-owned too. memcached runs as nobody and its rc script makes its own pid directory, so it needs no setup.
  3. The library cache. pkg -r does not run the jail's library trigger, so the running jail would not know a new library such as memcached's libevent. The installer runs /sbin/ldconfig -m /usr/local/lib in the jail for you. After a pkg -r of your own, do the same with jailops exec.
  4. System V IPC. A jail has no System V shared memory, semaphores or message queues unless it asks. A server that needs them fails, PostgreSQL's initdb at shmget; sysvipc: true gives the jail its own. memcached does not use them, so the key stays out.
  5. daemon(8) services. An rc script that starts its daemon through daemon(8) without -f leaves a supervisor holding the descriptors it was given. JailOps starts services through a wrapper that keeps the output in a file and sets exec.consolelog for every jail, so this no longer hangs an installation; still, the health check is what proves the service runs, so make it a real one. memcached daemonizes itself with -d.

The help page has each of these as symptom, cause and fix.

From a jail you tuned by hand

The other way round.

Often the app already runs: a jail where you installed the packages, set the rc.conf knobs and opened a port until it worked. jailops app export turns that jail into a draft definition: its installed packages, enabled services, listening ports, data volumes, the names of configuration files that differ from the package's, and its recent log files. It never copies secrets.

$ jailops app export cache --project web --file memcached.yaml
$ jailops app validate memcached.yaml

Read the draft line by line. Name it, write the summary and the notes, decide what the health check proves, and remove what was only there while you experimented. Then validate it and install it from the local apps directory on a fresh jail: an export is a draft, and only a clean installation shows that the definition is complete.

Share it

One pull request, no JailOps release.

The catalog lives in the packages repository, in main/jailops-apps/apps/, and ships as the signed package jailops-apps with a version of its own. A new app is one pull request there:

  1. Add icon: memcached.svg to the definition and put the icon next to it.
  2. Copy both into main/jailops-apps/apps/.
  3. Raise version: in main/jailops-apps/pkgops.yaml: the next minor for a new app, the next patch for a fix. A published version is never rebuilt, so a change without a new version is refused.
  4. Validate the whole directory, then open the pull request against master.
# in your fork of the packages repository
$ cp memcached.yaml memcached.svg main/jailops-apps/apps/
$ sh scripts/bump-version.sh 0.5.0 jailops-apps   # the next minor of the version in pkgops.yaml
$ jailops app validate main/jailops-apps/apps
$ git switch -c app/memcached
$ git add main/jailops-apps && git commit -m "Apps: memcached"

CI builds the JailOps client at the release the repository packages and runs the same jailops app validate on the directory, so the definition must be valid for the daemon hosts actually run. A person then reviews it against the package. Merging publishes the new, signed jailops-apps, and every host gets memcached with its next pkg upgrade.

The packages repository's CONTRIBUTING.md has the details for contributors.

What reviewers look for, and what makes an app featured, is on the community page.

More articles

Try it on one server.

Try the console from the screenshots, install on a clean FreeBSD 15.1 machine, or add JailOps to a host you already run.