A systemd service has no $HOME, and a Go CLI will fail on it

System services run without HOME unless the unit sets User=, DynamicUser=, PAMName= or SetLoginEnvironment=. Go tools that look up the home directory then fail instantly, and a health check that throws its output away turns that into an abort with no reason attached.

I had a script that worked every time I ran it by hand and failed every time systemd ran it. When that happens it’s nearly always the environment, and the gap that caught me was one I didn’t expect: there was no $HOME.

What systemd actually sets

A system service starts with very little in its environment: PATH, USER and a few variables of systemd’s own. Whether HOME, LOGNAME and SHELL get set depends on SetLoginEnvironment=, and systemd.exec(5) spells out the default:

If not set, this defaults to true if User=, DynamicUser= or PAMName= are set, false otherwise. If set to true, the variables will always be set for system services, i.e. even when the default user root is used.

So a unit that just runs as root, which is the default, gets no HOME. That covers everything you start with a bare systemd-run, and most one-off timers too. You can check it in a few seconds:

systemd-run --quiet --wait --pipe env | grep -E '^(HOME|USER)='
# USER=root                  <- no HOME line

systemd-run --quiet --wait --pipe -p SetLoginEnvironment=yes env | grep HOME
# HOME=/root

SetLoginEnvironment= arrived in systemd v255. On anything older, -p User=root does the same for HOME.

Why Go tools care

On Unix, Go’s os.UserHomeDir() returns $HOME, or an error if it isn’t set. Plenty of CLIs call it at startup to find their config directory, and some do it even when you’ve handed them a config file explicitly. talosctl is one of them:

env -u HOME TALOSCONFIG=/path/to/talosconfig talosctl -n <node> service etcd
# failed to open config file "": $HOME is not defined

It exits 1 straight away, before it makes a single network call. kubectl in the same script carried on working fine, which is what threw me: it looked like one tool was broken, when really the environment was.

How it hid

The script was an unattended maintenance job. It did a rolling restart of three Kubernetes VMs on one hypervisor, with a health check after each step. That check was a chain of small probes, and I’d written each one for its exit code only:

etcd_ok() { talosctl -n "$1" service etcd 2>/dev/null | grep -q "HEALTH *OK" && … }

It retried for fifteen minutes and then gave up with:

ABORT: cluster not healthy before <node>:

The reason after the colon was empty, and the cluster had been healthy the whole time. Every probe I ran afterwards passed, because I ran them from a login shell, where HOME is set. The job failed the same way the next night.

What finally got me to a one-line diagnosis was three changes to the script. The first was logging the raw output of every check that failed. A probe that sends its output to /dev/null can only tell you that it failed, never why. On the rerun the log showed talosctl=[] next to healthy values for everything else, and that pointed at the tool within seconds.

The second was to stop filtering diagnostics for the word “error”. My first attempt grepped for rror and printed nothing, because $HOME is not defined doesn’t contain it. Now I log the first few lines as they are.

The third was a pre-flight check at startup, before the first destructive step, that runs every probe once against every node. If a tool can’t work in this environment, the job now stops immediately and says so, instead of retrying for fifteen minutes.

Fixes

Any one of these is enough:

export HOME=/root                              # at the top of the script
systemd-run -p SetLoginEnvironment=yes …       # systemd v255+
systemd-run -p User=root …                     # sets the login variables

After that, I test the script the way it will actually run, with systemd-run --wait --pipe or env -i PATH=/usr/bin:/bin. The shell I wrote it in has an environment the service will never see.