My Pi-hole answered every query and logged none of them

DNS lookups to the Pi-hole address succeeded, but its query log stayed empty. The answers came from the router, whose ad-block feature redirects every port-53 packet on the LAN to its own resolver. How I found out, and the one-line test that would have saved me the afternoon.

Checked against Pi-hole 6.4, UniFi Network 10.

I wanted a boring confirmation. My Windows machine has the Pi-hole’s address as its first DNS server, so I ran a hundred lookups from it and expected to see them in the Pi-hole’s query log. All hundred resolved. The log had nothing, apart from the pod’s own liveness probe asking about pi.hole from localhost.

That’s a strange result, because DNS isn’t something you can half-do. The lookups worked, so somebody answered.

First suspects: the DNS settings and the logging

The Windows adapter really did list the Pi-hole first, with Cloudflare and Google behind it as fallbacks. That setup hides a dead primary completely: every lookup still succeeds, so “all resolved” proves very little. I switched to unique marker names, probe-<random>.example.com, and grepped the pod’s log for them. Zero hits.

pi.hole made it weirder. Asked through the Pi-hole’s address, it came back with the LAN address of a different machine. Pi-hole itself answers that name with its own address, so I was talking to something else.

I spent a while on the wrong layers. The router has no firewall rule, port forward or DNS setting that mentions the Pi-hole. The pi.hole oddity turned out to be a harmless detail: UniFi publishes every client’s name as a local DNS record, and one of my clients is literally named pi.hole. Inside the pod everything was in order, with the right listener, the right upstream and query logging switched on. MetalLB’s speaker was answering ARP for the Pi-hole’s address like it should.

Capturing packets

If an answer arrives and the only pod behind the service never saw a query, packets have to tell the story. I attached an ephemeral debug container to the Pi-hole pod and ran tcpdump on port 53 while sending marker queries.

kubectl debug pihole-0 -n pihole --image=nicolaka/netshoot --profile=baseline \
  -- timeout 40 tcpdump -nn -i any 'port 53 and not net 127.0.0.0/8'

Two small gotchas there. The default debug profile asks for SYS_PTRACE, which PodSecurity’s baseline level rejects, hence --profile=baseline. And the capture shows up in the debugger container’s logs, not in the attach stream.

Zero packets, while three queries from my machine got answers. That settled it: the Pi-hole wasn’t involved at all.

The test I should have started with

Then I did what I should have done in the first five minutes. I asked for an address that nobody owns: 192.0.2.1, from the range reserved for documentation. A server that exists would answer. Nothing should.

It answered. So did a pod inside the cluster. Whatever intercepted my DNS did it by source, wherever the packet was headed. (My earlier WSL tests were worthless for a second reason: WSL2’s DNS tunnelling does exactly the same thing inside the VM, answering for any destination.)

What was really happening

The router has a feature called content filtering, and a policy called “LAN: Ad Block” was switched on for the LAN network. What I hadn’t connected is that it works by redirecting DNS. On the UDM you can see it in the NAT table:

iptables-save -t nat | grep -w 53
# -A DNSFILTER -p udp -m set --match-set dnsfilter src --dport 53 \
#    -j DNAT --to-destination <loopback>:1053

Every port-53 packet from the LAN’s address range gets rewritten to the router’s own resolver, which blocks the ad category and forwards everything else. The connection tracking table confirms it: a flow to my “nobody’s” address has a reply that comes from the loopback address, port 1053. Only traffic that stays inside one layer-2 segment, like the cluster VLAN, is spared, because it never goes through the router.

My nodes are dual-homed, so even pods in the cluster got caught. DHCP handed out the Pi-hole as the LAN’s DNS server, and the router quietly overrode it for every client that asked.

The fix, and what I changed in how I test

I disabled the content-filter policy. Within seconds the NAT chain was empty, the address set was gone, and my marker queries showed up in the Pi-hole’s log with the Windows machine as the client. The router’s ad blocking is off now; the Pi-hole does that job.

Two habits came out of it. First, a DNS server is only proven when a unique name shows up in its log, so I always send a marker and grep for it. Second, ask for the address nobody owns: if it answers, something on the path is pretending to be your server.