DNS Security & Protective DNS — Control Point, Filtering & Tunneling Detection

① DNS as a security control point — the earliest interception layer

DNS is the phonebook every internet transaction consults first. Before a browser loads a page, before malware phones home, before ransomware contacts its key server — a DNS query goes out. That makes the recursive resolver the earliest possible interception point in your network, one that sits upstream of firewalls, proxies and endpoint agents.

Research consistently shows that a very high proportion of malware families — often cited at around ninety percent — use DNS at some stage, whether to locate a C2 server, resolve a DGA-generated domain, or tunnel exfiltrated data in query payloads. Because DNS is rarely blocked at perimeters, it is an attacker-favourite covert channel. Blocking at DNS costs almost nothing compared with deep-packet inspection and requires no endpoint agent.

The practical model: all outbound DNS traffic from your network is forced through a Protective DNS resolver you control. Every query is logged, checked against threat intelligence, and answered — or blocked — before the user's device ever forms a TCP connection to the destination.

② DNS filtering and RPZ — blocking C2 and phishing at the resolver

Response Policy Zones (RPZ) are the standard mechanism for DNS-layer filtering. An RPZ is a specially formatted DNS zone that your recursive resolver consults on every query: if the queried domain (or the IP it resolves to) appears in an RPZ feed, the resolver rewrites the answer — returning NXDOMAIN, a sinkhole IP, or a walled-garden page — rather than the real malicious address. RPZ feeds are subscribed from threat intelligence providers and updated continuously.

What RPZ can block

• C2 domains: domains registered by malware authors that beaconing agents look up for instructions or payloads.
• Phishing domains: lookalike or typosquatted domains used to harvest credentials.
• Malware distribution: domains serving exploit kits, drive-by downloads, or ransomware payloads.
• DGA domains: algorithmically generated names used by botnets to evade static blocklists.

The interview point: RPZ is a resolver-level control — it does not need to be inline on the data path. A query never reaches the attacker's server if it is answered with NXDOMAIN at your resolver first. Pair RPZ with DNSSEC validation so the feeds themselves cannot be tampered with in transit.

③ DNS tunneling and data exfiltration — spotting data hidden in queries

DNS tunneling is the technique of encoding arbitrary data (keylogger output, stolen files, C2 commands) inside DNS query labels and responses. Because port 53 is almost always allowed through firewalls, it is a reliable exfiltration channel. A tunneling tool encodes data as Base32 or Base64 in subdomains — for example d2hvYW1p.attacker.com — and the attacker-controlled authoritative server decodes and reassembles it.

Detection relies on behavioural anomalies, not just blocklists. Key indicators:

• High label entropy: legitimate hostnames are human-readable; tunneled labels look random and long (often over forty characters per label).
• Excessive TXT record use: TXT queries/responses are a favourite payload carrier because they can hold arbitrary text.
• Abnormal query volume: tunneling tools generate far more queries per minute to a single authoritative domain than normal browsing.
• High NXDOMAIN rate: misconfigured tunnel tools generate many non-existent name lookups as side effects.
• Large response payloads: DNS responses rarely exceed a few hundred bytes; tunneled responses are often at the 512-byte or 4096-byte (EDNS0) ceiling.

Modern Protective DNS platforms use machine learning on these features to flag tunneling in near-real-time. Manual response is impractical given query volumes — automatic sinkholing of flagged authoritative domains is the operational response.

④ DoH, DoT and resolver hardening — closing the remaining gaps

Encrypted DNS protects query privacy and prevents eavesdropping or on-path tampering. Two main protocols: DNS over TLS (DoT) runs on TCP port 853 and uses a distinct port that is easy to monitor or restrict; DNS over HTTPS (DoH) runs on TCP/UDP port 443, blending into normal HTTPS traffic and making it much harder to block or selectively inspect. A third option, DNS over QUIC (DoQ), uses UDP port 853 and combines the low-latency of QUIC with TLS 1.3 encryption.

The security trade-off: encrypted DNS stops passive eavesdroppers, but it can also let devices bypass your corporate Protective DNS resolver if users or malware configure their own DoH endpoint. The hardening response is to block unauthorised encrypted DNS — firewall TCP 853 (DoT/DoQ) to external servers, and use URL filtering or proxy interception to redirect unauthorised DoH to your internal resolver.

Recursive resolver hardening checklist

• DNSSEC validation: verify the cryptographic chain of trust so forged records are rejected.
• Response rate limiting (RRL): throttle repeated identical answers to stop the resolver being used as a DDoS amplifier.
• Minimal query logging retention: log enough for forensics, not enough to create a privacy liability.
• RPZ subscription feeds: automated threat-intel-driven blocklists refreshed continuously.
• Split-horizon DNS: internal names answer internally; external queries never leak internal topology.