Security of the PowerDNS Recursor

For Security Advisories, see the dedicated page.

PowerDNS Security Policy

If you have a security problem to report, please email us at both peter.van.dijk@powerdns.com and remi.gacogne@powerdns.com. In case you want to encrypt your report using PGP, please use: https://www.powerdns.com/powerdns-keyblock.asc

Please do not mail security issues to public lists, nor file a ticket, unless we do not get back to you in a timely manner. We fully credit reporters of security issues, and respond quickly, but please allow us a reasonable timeframe to coordinate a response.

We remind PowerDNS and dnsdist users that under the terms of the GNU General Public License, PowerDNS and dnsdist come with ABSOLUTELY NO WARRANTY. This license is included in this documentation.

If you believe you have found a security vulnerability that applies to DNS implementations generally, and you want to report this responsibly to a number of implementers, you might consider also using the Open Source DNS Vulnerability mailing list, managed by DNS-OARC.

HackerOne

Security issues can also be reported on our HackerOne page and might fetch a bounty. Do note that only the PowerDNS software is in scope for the HackerOne program, not our websites or other infrastructure.

Disclosure Policy

  • Let us know as soon as possible upon discovery of a potential security issue, and we’ll make every effort to quickly resolve the issue.
  • Provide us a reasonable amount of time to resolve the issue before any disclosure to the public or a third-party.
  • We will always credit researchers in our Security Advisories.

Anti-spoofing

The PowerDNS Recursor uses a fresh UDP source port for each outgoing query, making spoofing around 64000 times harder. This raises the bar from ‘easily doable given some time’ to ‘very hard’. Under some circumstances, ‘some time’ has been measured at 2 seconds. This technique was first used by dnscache by Dan J. Bernstein and is standardized in RFC 5452

In addition, PowerDNS detects when it is being sent too many unexpected answers, and mistrusts a proper answer if found within a clutch of unexpected ones.

This behaviour can be tuned using the spoof-nearmiss-max.

Throttling

PowerDNS implements a very simple but effective nameserver. Care has been taken not to overload remote servers in case of overly active clients.

This is implemented using the ‘throttle’. This accounts all recent traffic and prevents queries that have been sent out recently from going out again.

There are three levels of throttling.

  • If a remote server indicates that it is lame for a zone, the exact question won’t be repeated in the next 60 seconds.
  • After 4 ServFail responses in 60 seconds, the query gets throttled too.
  • 5 timeouts in 20 seconds also lead to query suppression.

Security Polling

PowerDNS products can poll the security status of their respective versions. This polling, naturally, happens over DNS. If the result is that a given version has a security problem, the software will report this at level ‘Error’ during startup, and repeatedly during operations.

By default, security polling happens on the domain ‘secpoll.powerdns.com’, but this can be changed with the security-poll-suffix. If this setting is made empty, no polling will take place. Organizations wanting to host their own security zones can do so by changing this setting to a domain name under their control.

To make this easier, the zone used to host secpoll.powerdns.com is available.

To enable distributors of PowerDNS to signal that they have backported versions, the PACKAGEVERSION compilation-time macro can be used to set a distributor suffix.

Details

PowerDNS software sadly sometimes has critical security bugs. Even though we send out notifications of these via all channels available, we find that not everybody actually find out about our security releases.

To solve this, PowerDNS software will start polling for security notifications, and log these periodically. Secondly, the security status of the software will be reported using the built-in metrics. This allows operators to poll for the PowerDNS security status and alert on it.

In the implementation of this idea, we have taken the unique role of operating system distributors into account. Specifically, we can deal with backported security fixes.

Finally, this feature can be disabled, or operators can have the automated queries point at their own status service.

Implementation

PowerDNS software periodically tries to resolve ‘auth-x.y.z.security-status.secpoll.powerdns.com|TXT’ or ‘recursor-x.y.z.security-status.secpoll.powerdns.com’.

The data returned is in one of the following forms:

  • NXDOMAIN or resolution failure -> 0
  • “1 Ok” -> 1
  • “2 Upgrade recommended for security reasons, see …” -> 2
  • “3 Upgrade mandatory for security reasons, see …” -> 3

In cases 2 or 3, periodic logging commences. The metric security-status is set to 2 or 3 respectively. If at a later date, resolution fails, the security-status is not reset to 1. It could be lowered however if we discover the security status is less urgent than we thought.

If resolution fails, and the previous security-status was 1, the new security-status becomes 0 (‘no data’). If the security-status was higher than 1, it will remain that way, and not get set to 0.

In this way, security-status of 0 really means ‘no data’, and can not mask a known problem.

Distributions

Distributions frequently backport security fixes to the PowerDNS versions they ship. This might lead to a version number that is known to us to be insecure to be secure in reality.

To solve this issue, PowerDNS can be compiled with a distribution setting which will move the security polls from: ‘auth-x.y.z.security-status.secpoll.powerdns.com’ to ‘auth-x.y.z-n.debian.security-status.secpoll.powerdns.com.

Note two things, one, there is a separate namespace for debian, and secondly, we use the package version of this release. This allows us to know that 4.0.1-1 (say) is insecure, but that 4.0.1-2 is not.

Configuration Details

The configuration setting security-poll-suffix is by default set to ‘secpoll.powerdns.com’. If empty, nothing is polled. This can be moved to ‘secpoll.yourorganization.com’.

If compiled with PACKAGEVERSION=3.1.6-abcde.debian, queries will be sent to “auth-3.1.6-abcde.debian.security-status.security-poll-suffix”.

Delegation

If a distribution wants to host its own file with version information, we can delegate dist.security-status.secpoll.powerdns.com to their nameservers directly.

Newly Observed Domain Tracking

A common security technique for detecting domains that may be suspicious or be associated with bad actors such as hosting malware, phishing or botnet command and control, is to investigate domains that haven’t been seen before, i.e. are newly observed.

Deciding whether a domain is truly a new domain would involve deterministic methods, such as maintaining a database of all domains ever seen, and comparing all domain lookups against that database. Such a mechanism would not be scalable in a recursor, and so is best suited to offline analysis. However, determining candidate domains for such an offline service is a problem that can be solved in the recursor, given that sending all domain lookups to such an offline service would still be prohibitively costly, and given that the true number of newly observed domains is likely to be relatively small in a given time period.

A simple method to determine a candidate domain would simply be to check if the domain was not in the recursor cache; indeed this is a method used by many security researchers. However, while that does produce a smaller list of candidate domains, cache misses are still relatively common, particularly in deployments where techniques such as EDNS client-subnet are used.

Therefore, a feature has been developed for the recursor which uses probabilistic data structures (specifically a Stable Bloom Filter (SBF): [http://webdocs.cs.ualberta.ca/~drafiei/papers/DupDet06Sigmod.pdf]). This recursor feature is named “Newly Observed Domain” or “NOD” for short.

The use of a probabilistic data structure means that the memory and CPU usage for the NOD feature is minimal, however it does mean that there can be false positives (a domain flagged as new when it is not), and false negatives (a domain that is new is not detected). The size of the SBF data structure can be tuned to reduce the FP/FN rate, although it is created with a default size (67108864 cells) that should provide a reasonably low FP/FN rate. To configure a different size use the new-domain-db-size setting to specify a higher or lower cell count. Each cell consumes 1-bit of RAM (per recursor thread) and 1-byte of disk space.

NOD is disabled by default, and must be enabled through the use of the following setting in recursor.conf:

new-domain-tracking=yes

Once enabled the recursor will keep track of previously seen domains using the SBF data structure, which is periodically persisted to the directory specified in the new-domain-history-dir, which defaults to /var/lib/pdns-recursor/nod.

Administrators may wish to prevent certain domains or subdomains from ever triggering the NOD algorithm, in which case those domains must be added to the new-domain-ignore-list setting as a comma separated list. No domain (or subdomain of a domain) listed will be considered a newly observed domain.

There are several ways to receive the information about newly observed domains:

Logging

The setting new-domain-log is enabled by default once the NOD feature is enabled, and will log the newly observed domain to the recursor logfile.

DNS Lookup

The setting new-domain-lookup=<base domain> will cause the recursor to issue a DNS A record lookup to <newly observed domain>.<base domain>. This can be a suitable method to send NOD data to an offsite or remote partner, however care should be taken to ensure that data is not leaked inadvertently.

Protobuf Logging

If both NOD and protobuf logging are enabled, then the newlyObservedDomain field of the protobuf message emitted by the recursor will be set to true. Additionally newly observed domains will be tagged in the protobuf stream using the tag pdns-nod by default. The setting new-domain-pb-tag=<tag> can be used to alter the tag.

Unique Domain Response

A similar feature to NOD is Unique Domain Response (UDR). This feature uses the same probabilistic data structures as NOD to store information about unique responses for a given lookup domain. Determining if a particular response is unique for a given lookup domain is extremely useful for determining potential security issues such as:

  • Fast-Flux Domain Names
  • Cache-Poisoning Attacks
  • Botnet Command and Control Servers etc.

This is because well-behaved domains tend to return fairly stable results to DNS record lookups, and thus domains which don’t exhibit this behaviour may be suspicious or may indicate a domain under attack.

UDR is disabled by default - to enable it, set unique-response-tracking=yes in recursor.conf.

The data is persisted to /var/lib/pdns-recursor/udr by default, which can be changed with the setting unique-response-history-dir=<new directory>.

The SBF (which is maintained separately per recursor thread) cell size defaults to 67108864, which can be changed using the setting unique-response-db-size. The same caveats regarding FPs/FNs apply as for NOD.

Similarly to NOD, unique domain responses can be tracked using several mechanisms:

Logging

The setting unique-response-log is enabled by default once the NOD feature is enabled, and will log the newly observed domain to the recursor logfile.

Protobuf Logging

If both UDR and protobuf logging are enabled, then unique domain responses will be tagged in the protobuf stream using the tag pdns-udr by default. The setting unique-response-pb-tag=<tag> can be used to alter the tag.