Split-Horizon DNS with Pi-hole and Unbound

DNS is the piece of a homelab nobody photographs and everybody depends on. Serve several isolated networks — separate segments for different device classes — and the first thing that breaks is names: half your services are published to the internet through a CDN, half exist only inside, and clients have no idea which is which. Split-horizon DNS is the fix — the same name resolves to an internal address on the inside and a public address on the outside, from one resolver you control. This post documents a real one: Pi-hole in front for local records and ad-blocking, Unbound behind it for DNSSEC validation over DNS-over-TLS, on a single dedicated DNS VM.

Step 1: Understand what split-horizon actually buys you

#

Three problems, one mechanism.

The hairpin. A service published through a CDN resolves to a public anycast address. A laptop sitting two metres from the server that hosts it will resolve that public address, leave the LAN, cross the WAN, hit the CDN, come back through your firewall’s NAT and reverse proxy, and finally reach a box on the same switch. That path burns WAN bandwidth, adds tens of milliseconds, and depends on your firewall supporting NAT reflection at all. Many do it badly. Split-horizon answers the same name with the internal address, so the client goes direct.

Namespace leakage. Internal-only names should not exist on the public internet. If monitoring.home.lab is only ever answered by your resolver, a public resolver returns NXDOMAIN and your service inventory is not a DNS enumeration away.

One hop for two policies. Every client query already goes to your resolver. That is the natural place to drop advertising and telemetry domains, and the natural place to enforce DNSSEC validation — without a per-client agent.

The whole design in one command. Same name, two resolvers, two answers:

# Internal resolver — the local override
dig +short workflows.example.com @192.0.2.53
192.0.2.20

# Public resolver — the real, CDN-fronted answer
dig +short workflows.example.com @8.8.8.8
203.0.113.41

And a name that should exist only on the inside:

dig +short monitoring.home.lab @192.0.2.53
192.0.2.40

dig +short monitoring.home.lab @8.8.8.8
# (empty — NXDOMAIN)

That asymmetry is the entire point. Everything below is how to build it.

Step 2: The architecture

#

One VM, two daemons, strict layering:

• Pi-hole owns port 53 on the VM’s LAN address. It holds the blocklists, the local A records, the CNAME overrides and the PTR records. It is the only thing clients talk to.
• Unbound listens on 127.0.0.1:5335 and nowhere else. It is Pi-hole’s only upstream. It validates DNSSEC and forwards over TLS to Cloudflare on port 853.
• The firewall owns DHCP and hands out exactly one DNS server: the DNS VM. It also permits outbound TCP/853 from the DNS VM only, and blocks external DNS (53, 853, DoH endpoints) from everything else, so no client can bypass the resolver.

Pi-hole does not serve DHCP here. The router already does, and running two DHCP authorities on one broadcast domain is a support ticket waiting to happen.

Why forward-over-TLS instead of full recursion? Full recursion (Unbound talking directly to the root and TLD servers) is the purist answer and removes the third-party dependency. It also emits plaintext DNS on port 53 to hundreds of authoritative servers, which is trivially observable by anyone on the path — including your ISP. This lab chose encrypted forwarding to two upstreams with local DNSSEC validation: the validation is still yours, the query path is encrypted, and the trade is that Cloudflare sees the queries. Pick the trade deliberately; Step 10 shows the switch to recursion if you want the other one.

Query path:

client ──53──▶ Pi-hole (blocklist, local records, cache)
                  │  cache miss, not local
                  ▼
              Unbound 127.0.0.1:5335 (DNSSEC validation, cache)
                  │
                  ▼
              1.1.1.1:853 / 1.0.0.1:853 over TLS

Nothing else on the VM should listen on 53 — check before you start, because systemd-resolved frequently does:

ss -lunp | grep ':53'
systemctl status systemd-resolved

If systemd-resolved holds the port, disable its stub listener before installing Pi-hole:

mkdir -p /etc/systemd/resolved.conf.d
printf '[Resolve]\nDNSStubListener=no\n' > /etc/systemd/resolved.conf.d/no-stub.conf
systemctl restart systemd-resolved

Give the VM a static address. A resolver on DHCP is a resolver that eventually moves.

Step 3: Install Pi-hole

#

curl -sSL https://install.pi-hole.net | bash

The installer is interactive. Choices that matter:

• Upstream DNS provider: pick anything — you will replace it with Unbound in Step 5.
• Web admin interface: yes. It is the fastest way to manage local records and read the query log.
• Query logging: on, at least while you build. Privacy level can be tightened later.

Set the admin password immediately — the installer prints a random one and you will lose it:

pihole -a -p

Confirm Pi-hole is answering on the LAN address, not just loopback:

ss -lunp | grep ':53'
dig +short example.com @192.0.2.53

Interface listening behaviour matters in a routed network. Pi-hole’s default is to answer only queries that arrive from the directly attached subnet. In a routed network every query from another subnet arrives via the router’s interface — a different subnet — and gets silently dropped. Under Settings → DNS → Interface settings, choose Permit all origins and then constrain access at the firewall, where it belongs. The symptom of getting this wrong is unmistakable: the DNS VM’s own subnet resolves fine, every other subnet times out.

Step 4: Install and configure Unbound

#

apt update
apt install -y unbound

Write /etc/unbound/unbound.conf.d/pi-hole.conf:

server:
    verbosity: 0
    interface: 127.0.0.1
    port: 5335
    do-ip4: yes
    do-udp: yes
    do-tcp: yes
    do-ip6: no
    prefer-ip6: no

    # Only the local Pi-hole may query this resolver.
    access-control: 127.0.0.0/8 allow
    access-control: 0.0.0.0/0 refuse

    # Hardening
    harden-glue: yes
    harden-dnssec-stripped: yes
    harden-below-nxdomain: yes
    harden-referral-path: yes
    use-caps-for-id: no

    # 1232 bytes: fits inside the smallest common path MTU without IP
    # fragmentation, which DNSSEC responses will otherwise trigger.
    edns-buffer-size: 1232

    prefetch: yes
    cache-min-ttl: 300
    cache-max-ttl: 86400
    rrset-roundrobin: yes

    # Required to validate the upstream TLS certificates.
    tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt

    # Do not leak RFC1918 reverse lookups upstream.
    private-address: 192.168.0.0/16
    private-address: 172.16.0.0/12
    private-address: 10.0.0.0/8

forward-zone:
    name: "."
    forward-tls-upstream: yes
    forward-addr: 1.1.1.1@853#cloudflare-dns.com
    forward-addr: 1.0.0.1@853#cloudflare-dns.com

Two syntax notes. forward-tls-upstream is the current directive; older guides and older Unbound versions write forward-ssl-upstream — they are the same knob, and this lab’s config still carries the legacy spelling. And the #hostname suffix on each forward-addr is not a comment: it is the name Unbound verifies against the upstream’s certificate. Drop it and you get encryption without authentication.

use-caps-for-id: no is deliberate. DNS-0x20 randomises the case of query names as an anti-spoofing measure, but a meaningful number of authoritative servers mishandle it, and it buys little on top of an authenticated TLS channel. Leave it off here.

Validate the config, restart, verify:

unbound-checkconf
systemctl restart unbound
systemctl enable unbound

dig +short example.com @127.0.0.1 -p 5335

Now prove three separate things about Unbound before you put it in the path.

It validates DNSSEC. A correctly signed name must come back with the ad flag; a deliberately broken one must fail:

dig sigok.verteiltesysteme.net @127.0.0.1 -p 5335 | grep -E 'flags:|status:'
# ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, ...

dig dnssec-failed.org @127.0.0.1 -p 5335 | grep 'status:'
# ;; status: SERVFAIL, id: 41927

SERVFAIL on dnssec-failed.org is the pass condition, not a bug. It means Unbound refused to hand you an answer it could not verify.

It is not reachable from anywhere else. From another host on the lab network:

dig +short example.com @192.0.2.53 -p 5335
# ;; communications error to 192.0.2.53#5335: connection refused

Connection refused is correct. A resolver bound to 0.0.0.0 on a port nobody guards is an open resolver and an amplification source.

Its upstream actually works. If the TLS handshake to Cloudflare fails, every query dies with SERVFAIL and the cause is invisible in Pi-hole. Test the channel directly:

openssl s_client -connect 1.1.1.1:853 -servername cloudflare-dns.com </dev/null 2>&1 | head -5
kdig -d @1.1.1.1 +tls-ca +tls-host=cloudflare-dns.com example.com

Keep that command; Step 10 explains why it may fail everywhere except the DNS host.

Step 5: Point Pi-hole at Unbound

#

Web UI: Settings → DNS. Untick every upstream under Upstream DNS Servers, then under Custom 1 (IPv4) enter:

127.0.0.1#5335

Leave Use DNSSEC unticked. Unbound already validates; enabling it in Pi-hole means validating twice, which adds latency and produces confusing SERVFAILs. Same file, from the CLI:

grep PIHOLE_DNS /etc/pihole/setupVars.conf
# PIHOLE_DNS_1=127.0.0.1#5335

pihole restartdns

Verify that Pi-hole is genuinely using it. Query Pi-hole and watch Unbound’s counters move:

dig +short example.com @192.0.2.53
unbound-control stats_noreset | grep -E 'total.num.queries|num.query.tls'

If unbound-control is not set up, unbound-control-setup once is enough; failing that, the Pi-hole query log at http://192.0.2.53/admin shows the upstream as 127.0.0.1#5335 per query, which is proof enough.

While you are there, confirm the EDNS behaviour matches on both sides — Pi-hole should advertise a 1232-byte buffer with the DO bit set, matching edns-buffer-size in Unbound:

dig +dnssec +bufsize=1232 example.com @192.0.2.53 | grep -A1 'OPT PSEUDOSECTION'
# ; EDNS: version: 0, flags: do; udp: 1232

Mismatched buffer sizes are the classic cause of “DNSSEC works for small answers and breaks for big ones”.

Step 6: Build the internal namespace

#

Now the split-horizon part. Two kinds of record, and it is worth being clear about which is which.

Internal-only names are services that have no public presence at all: dashboards, the media server, the hypervisor UIs, home automation. Give them a name under a domain you control and that never appears in public DNS. This lab uses home.lab, with a short role prefix so the name itself tells you what the service is:

<role>.home.lab

Web UI: Local DNS → DNS Records. Add one A record per service:

The UI writes them to /etc/pihole/custom.list, one IP name pair per line, so you can bulk-edit and reload:

cat /etc/pihole/custom.list
# 192.0.2.40 monitoring.home.lab
# 192.0.2.20 proxy.home.lab
# 192.0.2.30 stream.home.lab

pihole restartdns

Use a real domain suffix you own (lab.example.com) rather than an invented TLD if you ever want internal TLS certificates from a public CA — ACME DNS-01 can issue for a name that only resolves internally, but only if the parent domain is real. .home.lab and .local will never get a public certificate.

Step 7: Kill the hairpin with CNAME overrides

#

The second kind of record is for services that are public — behind a CDN, with a real certificate and a real external address — but should be reached directly when the client is inside. Pointing the public name at the internal reverse proxy does exactly that.

Web UI: Local DNS → CNAME Records. Target the internal proxy’s name, not a raw IP:

These land in /etc/dnsmasq.d/05-pihole-custom-cname.conf:

cat /etc/dnsmasq.d/05-pihole-custom-cname.conf
# cname=workflows.example.com,proxy.home.lab
# cname=auth.example.com,proxy.home.lab

pihole restartdns

Two constraints that bite people:

• The CNAME target must itself be resolvable by Pi-hole — a local A record from Step 6, or a real public name. A CNAME to a name Pi-hole cannot answer produces a silent SERVFAIL.
• The internal reverse proxy must present a valid certificate for the public name. The client is still requesting https://workflows.example.com; only the address changed. Use the same certificate the public endpoint uses, or issue one via DNS-01 for that name.

Deliberately do not override public names you want to keep going through the CDN — anything relying on the CDN’s WAF, rate limiting or access policies. Overriding those internally bypasses the control.

Step 8: Verify the split

#

Do this from a normal LAN client, not from the DNS VM. Five checks, five expected outcomes.

# 1. Internal-only name → internal address
dig +short stream.home.lab @192.0.2.53
# 192.0.2.30

# 2. Same name via a public resolver → nothing. No leakage.
dig +short stream.home.lab @8.8.8.8
# (empty)

# 3. Overridden public name → internal proxy, no hairpin
dig +short workflows.example.com @192.0.2.53
# proxy.home.lab.
# 192.0.2.20

# 4. Non-overridden public name → same public answer inside and out
dig +short www.example.com @192.0.2.53
dig +short www.example.com @8.8.8.8
# 203.0.113.41   (identical both ways)

# 5. Advertising domain → blackholed
dig +short doubleclick.net @192.0.2.53
# 0.0.0.0
dig +short doubleclick.net @8.8.8.8
# 203.0.113.99   (a real address)

Then DNSSEC, end to end through Pi-hole:

dig dnssec-failed.org @192.0.2.53 | grep 'status:'
# ;; status: SERVFAIL, id: 22104

One behaviour to expect and not chase: Pi-hole does not set the AD bit on its responses, even though Unbound validated the answer. Query a public resolver and you see flags: qr rd ra ad; query Pi-hole and the ad is missing. Validation happened — the SERVFAIL on dnssec-failed.org proves it — but dnsmasq does not propagate the flag from its upstream unless its own DNSSEC validation is enabled. If a client application insists on seeing AD, it must validate for itself; the security property is enforced at the resolver either way.

Step 9: Reverse DNS, so your logs are readable

#

Firewall logs, netstat, traceroute, monitoring dashboards, tcpdump — all of them are three times more useful when addresses render as names. Pi-hole generates a PTR record automatically for every A record in /etc/pihole/custom.list, so if you did Step 6 you already have reverse DNS for those hosts. Confirm:

dig +short -x 192.0.2.30 @192.0.2.53
# stream.home.lab.

dig +short -x 192.0.2.53 @192.0.2.53
# pi.hole.

The DNS VM answers to pi.hole for itself — that name is Pi-hole’s own default and is fine to keep.

For addresses that have no service record — gateways, switches, access points, printers — add them explicitly rather than leaving them anonymous in your logs. A dnsmasq drop-in is the cleanest place:

cat > /etc/dnsmasq.d/06-local-ptr.conf <<'EOF'
ptr-record=1.2.0.192.in-addr.arpa,firewall.home.lab
ptr-record=2.2.0.192.in-addr.arpa,switch.home.lab
EOF

pihole restartdns
dig +short -x 192.0.2.1 @192.0.2.53
# firewall.home.lab.

Note the reversed octet order in the in-addr.arpa name — 192.0.2.1 becomes 1.2.0.192.in-addr.arpa. Getting it backwards is the most common reason a hand-written PTR silently does nothing.

Also make sure Unbound is not asked to resolve your private ranges upstream. The private-address lines in Step 4 already prevent the leak; Pi-hole answering PTR locally means the query never gets that far in the first place.

Step 10: Firewall rules that make the design enforceable

#

DNS policy is only policy if clients cannot route around it. Three rules:

1. DHCP hands out one DNS server — the DNS VM — on every VLAN. No secondary public resolver in the lease, ever. A secondary means half your queries silently bypass every rule below, intermittently, which is the worst failure mode there is.
2. Block outbound 53 and 853 from all client machines — phones, laptops, visitors’ devices — and NAT-redirect any stray port 53 to the DNS VM so hardcoded devices (smart TVs, smart-home gear) get transparently corrected instead of failing.
3. Permit outbound TCP/853 from the DNS VM only. This is the exception that makes Unbound work.

That last rule matters more than it looks. If your firewall does L7 inspection on outbound TLS, it can break DNS-over-TLS everywhere except the explicitly allowed DNS host. From a normal client the failure is exact and unhelpful:

kdig -d @1.1.1.1 +tls-ca +tls-host=cloudflare-dns.com example.com
# ;; DEBUG: TLS, handshake failed: no peer certificate

From the DNS VM the same command completes. If you deploy Unbound with DoT and every query SERVFAILs while unbound-checkconf is clean, test the 853 path from that host before touching the config — the resolver is almost certainly fine and the firewall is eating the handshake.

If you would rather do full recursion and drop the upstream dependency entirely, replace the whole forward-zone block with root hints and a trust anchor:

server:
    root-hints: "/var/lib/unbound/root.hints"
    auto-trust-anchor-file: "/var/lib/unbound/root.key"

curl -s -o /var/lib/unbound/root.hints https://www.internic.net/domain/named.root
chown unbound:unbound /var/lib/unbound/root.hints
unbound-anchor -a /var/lib/unbound/root.key
unbound-checkconf && systemctl restart unbound

Then your firewall needs outbound UDP/TCP 53 from the DNS VM instead of 853, and you should schedule a monthly refresh of root.hints. Everything else in this post is unchanged — that is the benefit of keeping Pi-hole and the resolver as separate layers.

Step 11: Pitfalls and ops

#

Don’t run DHCP on Pi-hole if the router already does. Two authorities, one broadcast domain, intermittent lease chaos. Pick one. The router usually wins because it survives DNS VM maintenance.

Keep 5335 on loopback. The moment Unbound listens on the LAN address, you own an open recursive resolver that can be used for DNS amplification against a third party. interface: 127.0.0.1 plus access-control: 0.0.0.0/0 refuse is not paranoia, it is the minimum.

The DNS VM is now a single point of failure. Everything on the network fails simultaneously and confusingly when it goes down — services do not “break”, they become unreachable by name, which users report as “the internet is down”. Either run a second Pi-hole with synchronised records and hand out both in DHCP, or accept the risk knowingly and snapshot the VM before every change. Do not accidentally end up in the middle: a secondary that does not have the same local records will answer NXDOMAIN for half your namespace, at random.

Watch log growth. A verbose resolver logs every query from every device on the network. This lab once filled the root filesystem with an over-15 GB /var/log/syslog from exactly this. Fix it before it happens:

cat > /etc/logrotate.d/pihole-dns <<'EOF'
/var/log/pihole.log /var/log/pihole-FTL.log {
    daily
    rotate 5
    copytruncate
    compress
    delaycompress
    missingok
    notifempty
}
EOF

logrotate -d /etc/logrotate.d/pihole-dns
du -sh /var/log/*

Set verbosity: 0 in Unbound in steady state and raise it only while debugging. Pi-hole’s own retention lives under Settings → Privacy; a few days is plenty for a lab.

EDNS buffer 1232 on both sides. DNSSEC answers are large. A 4096-byte advertised buffer invites IP fragmentation, and fragmented UDP is dropped by a meaningful share of middleboxes — which presents as random, name-specific resolution failures that “fix themselves” on retry. 1232 is the number the DNS Flag Day 2020 consensus settled on; use it in Unbound and confirm Pi-hole matches, as in Step 5.

Blocklists are an availability dependency. An over-broad list will break a service you care about, weeks after you added it, in a way nobody connects to DNS. Learn the whitelist command before you need it:

pihole -w tracking.vendor-you-actually-use.com
pihole -q some-broken-domain.com     # which list blocked it
tail -f /var/log/pihole.log | grep -i blocked

Test after every firewall change. The five dig commands in Step 8 take fifteen seconds to run and catch nearly everything: a broken upstream, a lost local record, a rule that started intercepting 53, a blocklist that swallowed a real domain.

What the split looks like when it works

#

• Internal names resolve only internally. Public resolvers return NXDOMAIN — no service inventory leaked in DNS.
• Public names you host resolve to the internal reverse proxy for LAN clients and to the CDN for everyone else. No hairpin, no NAT reflection dependency, and the certificate still validates because the name never changed.
• Public names you don’t host resolve identically inside and out. Split-horizon should be surgical, not a parallel internet.
• Ads and telemetry are blackholed to 0.0.0.0 for every device on the network, including the ones that cannot run a blocker.
• DNSSEC is validated once, at the resolver, for every client. Broken signatures return SERVFAIL rather than a forged answer — accept that the AD bit does not survive the last hop.
• Reverse DNS covers the estate, so firewall and monitoring logs read as names.

None of this needs exotic hardware — it is one small VM, two packages, a table of local records, and three firewall rules. The discipline is in the rules, not the software: one DNS server in DHCP, one upstream in Pi-hole, one interface on Unbound. Break any of those three and you still have DNS, which is precisely what makes the failure so hard to find later.