Edit and Write Quadlets Easily

About Quadlets

People usually mention that one of Podman’s main features is that it runs rootless by default, which provides better security. Which is true I don’t doubt it. But for me, there are several other reason why I use Podman, especially via Quadlet.

Here are a few reason why I prefer Quadlet over Docker compose and moved my services.

• Systemd unit syntax, is much more readable for me than YAML. Some people find it painful to switch between files, but I’d rather jump between short files than scroll through a 200–300 line compose file (besides I can just simple jump between files using my editor’s ‘go definition’ or ‘go references’ action).
• Because Quadlet becomes a regular service, I can trigger them by systemd timer or path unit
• No need to install extra monitoring software. On a server, in a good situation, systemd is monitored anyway, (even you are using docker), e.g.: by systemd_exporter. In case of Quadlet, systemd exporter is fine for me no need extra exporter.
• I can use socket activation and having a basic out of the box serverless functionality. I use this on my dev machine: if any connection coming to specific port, e.g.: 5432, then postgres container is automatically started and traffic routed there. If no connection for 30 sec it just simply stop. I don’t have to install any extra proxy software to achieve it, but pure systemd. See for more details.
• Because Quadlet becomes a regular service, I can have dependency between containerized and non-containerized unit.
• Via Requires, After, BindsTo, etc I have better control when I want to start or stop my services/containers.
• It uses journal for logging, on server, at a server journal logs are already aggregated, no need to install extra log aggregator.
• Limit resources via systemd cgroup limits.

Most of these points, can also be solved via Docker (functionality-wise), but that requires to install additional products. With Quadlet, all things above are possible with pure systemd.

File format

Systemd unit files using a syntax which is very similar like the ini format. But it has some difference, for example line continuation sign (\) and some key can be repeated.

[Container]
HealthCmd=/bin/curlcurl -k --fail --connect-timeout 5 \
  https://127.0.0.1:3000/api/healthz

Problems with the syntax

From the readability, I prefer the systemd syntax comparing with YAML files. But everything comes at a price – in this case, complexity. What am I talking about? Advantage of compose file that it is compact: container, volume, network, all definitions are in the same file. With Quadlet they are split onto multiple files.

So this gives me some problem to solve:

• Navigation between files should be easy.
• Quadlet files, from the naming perspective, more verbose and less compact than compose file.
• The documentation of Podman systemd unit is great, but opening it every time when I need info, is just too much time.

And as solution, I’ve come up with an idea of a language server implementation for Podman Quadlets. Problems above can be solved as:

• Using the built-in editor’s go to definition and go to reference commands to navigate between files.
• About document, I’ve implemented hints when a hover action is done.
• All features are documented and showed in the documentation.
• Use the completion feature for the following actions:
  • Create new basic files from scratch
  • Complete the name of the properties
  • Give dynamic completions (e.g.: list exposed ports, users)

Common errors that usually pop up lately

My other problem was related for the problem detection. We can refresh the Quadlets, and generate the systemd services, by using systemctl --user daemon-reload command. My issues with this command:

• It does not tell me (on my console) if it failed to generate service.
• The generator itself, does not make too much static checking. Lot of problem occur during runtime of the container.

These points are painful for me, because I usually does not edit the Quadlet file on the machine where I’ll run them. I’ve often edited Quadlet files locally, thought everything was fine, but then they failed to start on the target server.

To solve this problem, I’ve created a collection, and updating still, that are checking the Quadlet file itself for syntax errors. I call them: QUadlet Syntax Rules and they can be read in the QSR document.

I also made the run the language server run as binary file to check the syntax. This comes handy to check syntax in a GitHub action or in other pipelines.

In this post, I talk about how do I organize and manage my Quadlet files. I’m using Neovim editor, so all the screenshots and records are from Neovim, but same actions can be done using VS Code or Zed.

The demo below shows how easy to create a new service with the language server. It is just a few moment and it create a container and corresponding volume.

For the full features, please check the reference.

Organize files

Do we really need directories

With compose files, we usually put the different project into different directory. Something like this file structure.

haproxy
  |- docker-compose.yaml
  |- .env
  |- volumes
       |- ...
web
  |- docker-compose.yaml
  |- .env
  |- volumes
       |- ...

If somebody is coming from Docker, it would make sense to organize the Quadlet files same way like:

haproxy
  |- haproxy.container
  |- haproxy-cfg.volume
  |- haproxy-log.volume
  |- .env
web
  |- web.container
  |- web-data.volume
  |- web-log.volume
  |- web-db.container
  |- web-db-data.volume
  |- .env

But does it make sense? What do we gain from such a layout? In my view, nothing – in fact, we lose some safety features.

I recommend using flat structure

To understand why directories are unnecessary, first let’s understand why do we do it with Docker. If I execute docker compose up -d command, without specified project name, then it uses the parent directory name as project name. Which means all created container get that name as prefix. It guarantee if we had an app service in two different compose file, they won’t have naming mismatch.

What about Quadlet? When we make a systemctl --user daemon-reload command, it does not matter what is the parent directory name. It won’t be in the unit naming!

Using directories can actually cause unexpected errors. Since files are in separate directories, you can accidentally create files with the same name multiple times. But due to naming convention, only file name matter, so only just one of these files will be generated. Let me explain in practice, we have the following directory structure:

test1
  |- test-app.volume
test2
  |- test-app.volume

What happens after refresh? Do we get test1-test-app-volume.service and test2-test-app-volume.service? No! We only got one test-app-volume.service. And here you can accidentally attach wrong volume to another container where you did not want. It can cause damage to data.

I recommend to use flat structure and give the files prefix which replaces the functionality of parent directory in case of compose. Besides, if you have already given unique prefix, what would be the practical meaning of putting them to directory?

My naming convention

Here is an example from one of my server if you need an idea to organize Quadlet files:

• Every file start with a prefix.
• Network files are just prefix.network.
• If the file is a pod, then name is prefix.pod. For example: 1pw.pod, gitea.pod
• If the file is a container, then name is prefix-component.container. For example: gitea-app.container, gitea-db.container.
• Volume files:
  • If there is only one I just use the same name then container but with different extension: gitea-db.volume.
  • If there are more of them, I append something at the end that identify what is that volume used for: gitea-app-data.volume, gitea-app-etc.volume.
  • If a volume is on pod level, then prefix.volume, like: 1pw.volume.

With these of my rules, I can easily identify, just by name, which file belongs to where, it also guarantee unique file name.

1pw-api.container
1pw.pod
1pw-sync.container
1pw.volume
gitea-app.container
gitea-app-data.volume
gitea-app-etc.volume
gitea-db.container
gitea-db.volume
gitea.pod
keycloak-app.container
keycloak-db.container
keycloak-db.volume
keycloak.pod
renovate@.container

How to organize a file

Environment variables

In the compose world, we usually use external .env (and similar) files. With Quadlet, we can still use them (via EnvironmentFile options), but I can’t see any benefit using it. In my understanding, environment variables belongs to container. Let’s keep them together with the container’s other settings!

Quadlet is not YAML. Much easier to read, even with a lot of environment variables.

What about when environment variable store something sensitive (e.g.: password)? Don’t use environment variable, we have secrets!

Organize the options in groups

Quadlet file can be chaotic and difficult to read if properties are not in order. To prevent this, I always organize my Quadlet’s content based on the following:

• Place them to functional groups which are divided by one line comment.
• Within the group, I order lines in alphabetical order.
• I don’t let that lines become too long (over 80 character).

Sample Quadlet file:

[Unit]
Description=Gitea application
Wants=gitea-db.service

[Container]
# Base options
AutoUpdate=registry
Image=docker.io/gitea/gitea:latest-rootless
Pod=gitea.pod

# Storage options
Volume=/etc/localtime:/etc/localtime:ro
Volume=/etc/timezone:/etc/timezone:ro

# Environment options
Environment=GITEA__database__DB_TYPE=postgres
Environment=GITEA__database__HOST=127.0.0.1
Environment=GITEA__database__NAME=gitea
Environment=GITEA__database__USER=gitea

# Secret options
Secret=gitea-db-password,type=env,target=GITEA__database__PASSWD
Secret=gitea-smtp-password,type=env,target=GITEA__mailer__PASSWD

# Healthcheck options
HealthCmd=/bin/curlcurl -k --fail --connect-timeout 5 \
  https://127.0.0.1:3000/api/healthz
HealthRetries=10
HealthStartPeriod=15s
HealthTimeout=15s

# Other options
LogDriver=journald
UserNS=keep-id

[Service]
Restart=on-failure
RestartSec=5
StartLimitBurst=5

[Install]
WantedBy=default.target

Drop-In directories

We can override any properties in the Quadlet files, without modify them. Quote from Podman systemd documentation:

The source files also support drop-ins in the same way systemd does. For a given source file (foo.container), the corresponding .d directory (foo.container.d) will be scanned for files with a .conf extension, which are then merged into the base file in alphabetical order. Top-level type drop-ins (container.d) will also be included. If the unit contains dashes (“-“) in the name (foo-bar-baz.container), then the drop-in directories generated by truncating the name after the dash are searched as well (foo-.container.d and foo-bar-.container.d). Drop-in files with the same name further down the hierarchy override those further up (foo-bar-baz.container.d/10-override.conf overrides foo-bar-.container.d/10-override.conf, which overrides foo-.service.d/10-override.conf, which overrides container.d/10-override.conf). The format of these drop-in files is the same as the base file. This is useful to alter or add configuration settings for a unit, without having to modify unit files.

When is it useful in practice

Imagine the following situation:

• You have a repository where you store Quadlet file of specific application.
• You make a git pull on target system where you want to run it.
• You deploy the same application on multiple system, but there are system specific settings.

How do you change the Quadlet file for those system specific changes without cause trouble with git? The answer is: using drop-in directories. Because with drop-in files, you can override any settings from the Quadlet file! So your Quadlet file can be still sync with the git remote repository, meanwhile you can make your system wide customizations.

Final words

In this post, I’ve shared some of my methods and approaches about Quadlet file handling. It requires a different mindset than using compose files to make it easy to read and modify. For this, I also made my Quadlet language server, to simply make navigation and editing easier and smoother.

Of course, this post was an opinion, not rules that must be followed, but I find my opinions and defaults as reasonable. If you’re struggling, give this language server a try and apply some of my approaches – they might work for you as well.