One unreachable Docker host leaves the Homepage dashboard empty

Homepage asks every server in docker.yaml on each /api/services call. One host that times out delays the whole response by minutes, and the page renders no tiles. Time the API from inside the pod to find it.

My cluster’s start page is Homepage, and it had two problems in a row. The first was a page that didn’t change after I changed its config. The second showed up as soon as I fixed the first: the page loaded and had no tiles at all.

A config that never arrived

I had removed a tile in git, Flux had applied it, and the page still showed the tile. Flux was fine (all Kustomizations ready on the latest revision). The pod wasn’t:

kubectl -n homepage exec deploy/homepage -- grep -c <removed-service> /app/config/services.yaml   # 3
kubectl -n homepage get configmap homepage -o yaml | grep -c <removed-service>                    # 0

The ConfigMap is mounted with subPath, and those mounts never update in a running pod. A plain ConfigMap edit doesn’t restart the Deployment either. I’d known that and restarted by hand before. This time I made it permanent: a kustomize configMapGenerator builds the ConfigMap from files and adds a hash of the content to its name. Any change gives a new name, the Deployment’s reference changes with it, and that rolls the pod. Flux prunes the old ConfigMap.

Then the page went blank

After the first real restart in a while, the page came up empty. The browser was waiting on one request, /api/services. From inside the pod:

kubectl -n homepage exec deploy/homepage -- \
  sh -c 'time wget -qO- localhost:3000/api/services > /dev/null'

It hung for more than two minutes. The pod log said why:

ETIMEDOUT <address>:2375
ECONNREFUSED <address>:2375

Homepage’s docker.yaml lists Docker hosts whose containers it can show status for. On every /api/services call, it asks each of them. Two of my hosts no longer ran Docker and refused at once, which is cheap. A third no longer existed, and its connection attempt hung until the timeout. Nothing renders until the slowest host answers, so one dead host in that file is enough to empty the page.

I removed the three hosts, and a tile that depended on one of them. The API now answers in about 0.2 seconds.

The part I find worth remembering is the order. The dead hosts had been in the file for a while. The old pod had started with an older config and never re-read it, so the problem only surfaced once the config fix made the pod restart. A stale pod can hide a newer problem.