Previous topic

PowerDNS Security Advisory 2024-04: Crafted responses can lead to a denial of service due to cache inefficiencies in the Recursor

Next topic

Changelogs

This Page

Upgrade Guide

Before upgrading, it is advised to read the Changelogs. When upgrading several versions, please read all notes applying to the upgrade.

5.1.0 to master

Changed behaviour

Parsing of old-style settings is no longer enabled by default. Convert your settings file to YAML (see Conversion of old-style settings to YAML format) or pass --enable-old-settings on the command line.

The way incoming.max_tcp_clients is enforced has changed. If there are too many incoming TCP connections, new connections will be accepted but then closed immediately. Previously, excess connections would linger in the OS listen queue until timeout or until processing of incoming TCP connections resumed due to the number of connections being processed dropping below the limit. There is a new metric tcp-overflow that counts the connections closed immediately.

The outqueries-per-query value reported in the log by the periodic statistics function is now reported as outqueries-per-query-perc as it is a percentage. A value of 1 means that on average each 100 incoming queries lead to a single query to an authoritative server.

A new rec_control reload-yaml command has been introduced as an alias for reload-lua-config. Both commands will (if YAML settings are active), reload the runtime reloadable parts of the YAML settings. These are the YAML settings that correspond to Lua configuration items, plus a few new settings that have no Lua equivalent. The documentation has been updated to state more clearly which settings can be modified at runtime.

Changed settings

New Settings

5.1.2 to 5.1.3

New Settings

  • The serve-rfc6303 settings has been introduced to implement RFC 6303. By default this setting is enabled so this potentially changes behaviour for names inside the ip6.arpa domain.

5.1.1 to 5.1.2, 5.0.8 to 5.0.9 and 4.9.8 to 4.9.9

New settings

5.0.6 to 5.1.0

The recursor.conf configuration file may contain YAML configuration syntax and new installs using our packages from repo.powerdns.com will install a configuration file using YAML syntax. Note to third-party package maintainers: please start doing the same.

Warning

If you are using the default unmodified recursor.conf from a previous release, it will be overwritten by an equivalent recursor.conf in YAML format by most packaging tools. If you also have local setting files in the include directory, these are now expected to be in YAML format as well, because the format of the included files must be the same as the format of the main recursor.conf. This has the consequence that the previously included files will not be processed after the upgrade. To work around this issue, either:

  • modify the recursor.conf before upgrading so it does not get overwritten by the upgrade,
  • copy back the original old-style recursor.conf after upgrading,
  • or change the format of the existing included files into YAML and make sure their names have the .yml suffix.

A modified recursor.conf file will not be overwritten by an upgrade.

New settings

Changed settings

  • The max-qperq default value has been lowered to 50, and the qname-minimization special case has been removed.
  • Disabling structured-logging is no longer supported.
  • The structured-logging-backend setting has gained the possibility to request JSON formatted output of structured logging information.

5.0.5 to 5.0.6

Changed settings

  • The max-mthreads setting will be adjusted to a lower value if the value of sysctl vm.max_map_count is too low to support the maximum number of mthread stacks. In this case Recursor logs an error message including the suggested value of vm.max_map_count to not cause lowering of max-mthreads.

5.0.4 to 5.0.5

Changed settings

5.0.2 to 5.0.3, 4.9.3 to 4.9.4 and 4.8.6 to 4.8.7

Known issue solved

The DNSSEC validation issue with the zoneToCache() function has been resolved and workarounds can be removed.

5.0.1 to 5.0.2, 4.9.2 to 4.9.3 and 4.8.5 to 4.8.6

Known issues

The zoneToCache() function fails to perform DNSSEC validation if the zone has more than max-rrsigs-per-record RRSIG records at its apex. There are two workarounds: either increase the max-rrsigs-per-record to the number of RRSIGs in the zone’s apex, or tell zoneToCache() to skip DNSSEC validation. by adding dnssec="ignore", e.g.:

zoneToCache(".", "url", "https://www.internic.net/domain/root.zone", {dnssec="ignore"})

New settings

4.9.0 to 5.0.0

YAML settings

Starting with version 5.0.0-alpha1 the settings file(s) can be specified using YAML syntax. The old-style settings files are still accepted but will be unsupported in a future release. When a recursor.yml settings file is encountered it will be processed instead of a recursor.conf file. Refer to PowerDNS Recursor New Style (YAML) Settings for details and the Conversion of old-style settings to YAML format guide for how to convert old-style settings to the new YAML format.

Rust

Some parts of the Recursor code are now written in Rust. This has impact if you do local builds or are a third-party package maintainer. According to cargo msrv the minimum version to compile the Rust code and its dependencies is 1.64. Some distributions ship with an older Rust compiler, see Rustup for a way to install a more recent one. For our package builds, we install a Rust compiler from the Standalone section of Other Rust Installation Methods.

New settings

  • The bypass-server-throttling-probability setting has been introduced to try throttled servers once in a while.
  • The tcp-threads setting has been introduced to set the number of threads dedicated to processing incoming queries over TCP. Previously either the distributor thread(s) or the general worker threads would process TCP queries.
  • The qname-max-minimize-count and qname-minimize-one-label have been introduced to allow tuning of the parameters specified in RFC 9156.
  • The allow-no-rd has been introduced, default disabled, disallowing queries that do not have the Recursion Desired (RD) flag set. This is a change in behavior compared to previous releases.
  • The setting ignoreDuplicates was added to the RPZ loading Lua functions rpzPrimary() and rpzFile(). If set, duplicate records in RPZs will be allowed but ignored. The default is to fail loading an RPZ with duplicate records.

Changed settings

4.8.0 to 4.9.0

Metrics

The way metrics are collected has been changed to increase performance, especially when many thread are used. This allows for solving a long-standing issue that some statistics were not updated on packet cache hits. This is now resolved, but has the consequence that some metrics (in particular response related ones) changed behaviour as they now also reflect packet cache hits, while they did not before. This affects the results shown by rec_control get-qtypelist and the response-by-qtype, response-sizes and response-by-rcode items returned by the /api/v1/servers/localhost/statistics API endpoint. Additionally, most RCodes and QTypes that are marked Unassigned, Reserved or Obsolete by IANA are not accounted, to reduce the memory consumed by these metrics.

New settings

  • The packetcache-negative-ttl settings to control the TTL of negative (NxDomain or NoData) answers in the packet cache has been introduced.
  • The stack-cache-size setting to control the number of allocated mthread stacks has been introduced.
  • The packetcache-shards settings to control the number of shards in the packet cache has been introduced.
  • The aggressive-cache-min-nsec3-hit-ratio setting to control which NSEC3 records are stored in the aggressive NSEC cache has been introduced. This setting can be used to switch off aggressive caching for NSEC3 only.
  • The dnssec-disabled-algorithms has been introduced to not use DNSSEC algorithms disabled by the platform’s security policy. This applies specifically to Red Hat Enterprise Linux 9 and derivatives. The default value (automatically determine the algorithms that are disabled) should work for many cases.
  • The setting includeSOA was added to the rpzPrimary() and rpzFile() Lua functions to include the SOA of the RPZ the responses modified by the RPZ.

Changed settings

The first two settings below have effect on the way the recursor distributes queries over threads. In some cases, this can lead to imbalance of the number of queries process per thread. See Performance Guide, in particular the Imbalance section.

  • The pdns-distributes-queries default has been changed to no.
  • The reuseport default has been changed to yes.
  • The packetcache-ttl default has been changed to 24 hours.
  • The max-recursion-depth default has been changed to 16. Before it was, 40, but effectively the CNAME length chain limit (fixed at 16) took precedence. If you increase max-recursion-depth, you also have to increase stack-size. A starting point of 5k per recursion depth is suggested. Add some extra safety margin to avoid running out of stack.
  • The hint-file setting gained a new special value to disable refreshing of root hints completely. See Handling of root hints.

rec_control

The trace_regex subcommand has been changed to take a file argument. Refer to rec_control trace-regex and Tracing Queries for details and example use.

4.8.1 to 4.8.2

Cache eviction policy

The cache eviction policy for the record and the negative caches has been improved to reduce imbalance between shards. The maximum size of the negative cache is now 1/8th of the size of the record cache and its number of shards is 1/8th of the record-cache-shards setting. Previously the size was 1/10th of the record cache size and the number of shards was equal to the number of shards of the record cache. The rec_control dump-cache command now prints more information about shards.

4.7.0 to 4.8.0

Structured logging

All logging (except query tracing) has been converted to structured logging. Switch to old style logging by setting the structured-logging setting to no. When using systemd, structured logging information will be sent to journald using formatted text strings that list the key-value pairs and are human readable. Switch to native key-value pair logging (more suitable for automated log processing) by setting structured-logging-backend on the command line to systemd-journal.

New settings

pdns_recursor changes

THe --config command line option now implements the check, default and diff keywords.

rec_control changes

The dump-throttle and dump-edns subcommands no longer produces a table per thread, as the corresponding tables are now shared by all threads. Additionally, the dump-edns command now only lists IPs that have a not OK status. The dump-nsspeeds command has changed format to make it more readable and lists the last round trip time recorded for each address. The get-proxymapping-stats and get-remotelogger-stats subcommands have been added.

4.7.2 to 4.7.3

New settings

  • The max-ns-per-resolve setting to limit the number of NS records processed to resolve a name has been introduced.

4.6.2 to 4.7.0

Zone to Cache Changes

The Zone to Cache feature now validates ZONEMD records. This means that zones containing invalid ZONEMD records will be rejected by default, while previously the ZONEMD records would be ignored. For more detail, refer to Zone to Cache.

Asynchronous retrieval of AAAA records for nameservers

If IPv6 is enabled for outgoing queries using query-local-address, the Recursor will schedule an asynchronous task to resolve IPv6 addresses of nameservers it did not otherwise learn. These addresses will then be used (in addition to IPv4 addresses) for future queries to authoritative nameservers. This has the consequence that authoritative nameservers will be contacted over IPv6 in more case than before.

New Lua Configuration Functions

  • The addAllowedAdditionalQType() Lua configuration function was added to make the Recursor add additional records to answers for specific query types.
  • The addProxyMapping() Lua configuration function was added to map source addresses to alternative addresses.

Post Resolve FFI Function

A new postresolve_ffi() Lua callback function has been introduced.

New settings

  • The save-parent-ns-set setting has been introduced, enabling fallback cases if the parent NS set contains names not in the child NS set.
  • The max-busy-dot-probes settings has been introduced, enabling the Recursor probe for DoT support of authoritative servers. This is an experimental function, use with care.

rec_control changes

The dump-nsspeeds, dump-failedservers and dump-non-resolving subcommands no longer produce a table per thread, as the corresponding tables are now shared by all threads. They also use a better readable and sortable timestamp format.

4.6.3 to 4.6.4

New settings

  • The max-ns-per-resolve setting to limit the number of NS records processed to resolve a name has been introduced.

4.6.1 to 4.6.2

Deprecated and changed settings

  • The hint-file gained a special value no to indicate that no hint file should be processed. The hint processing code is also made less verbose.

4.5.x to 4.6.1

Offensive language

Using the settings mentioned in Offensive language now generates a warning. Please start using the new names.

File descriptor usage

The number of file descriptors used by the Recursor has increased because the Recursor now keeps idle outgoing TCP/DoT connections open for a while. The extra file descriptors used in comparison to previous versions of the Recursor is tcp-out-max-idle-per-thread times the number of worker threads (threads).

New settings

Deprecated and changed settings

  • The api-key and webserver-password settings now accept a hashed and salted version (if the support is available in the openssl library used).

Privileged port binding in Docker

In our Docker image, our binaries are no longer granted the net_bind_service capability, as this is unnecessary in many deployments. For more information, see the section “Privileged ports” in Docker-README.

4.5.10 to 4.5.11

New settings

  • The max-ns-per-resolve setting to limit the number of NS records processed to resolve a name has been introduced.

4.5.1 to 4.5.2

Deprecated and changed settings

4.4.x to 4.5.1

Offensive language

Synonyms for various settings names containing master, slave, whitelist and blacklist have been introduced.

Currently, the older setting names are also accepted and used. The next release will start deprecating them. Users are advised to start using the new names to avoid future trouble.

Special domains

Queries for all names in the .localhost domain will answer in accordance with RFC 6761 section 6.3 point 4. That means that they will be answered with 127.0.0.1, ::1 or a negative response.

rec_control command writing to a file

For the commands that write to a file, the file to be dumped to is now opened by the rec_control command itself using the credentials and the current working directory of the user running rec_control. A single minus - can be used as a filename to write the data to the standard output stream. Previously the file was opened by the recursor, possibly in its chroot environment.

New settings

Deprecated and changed settings

Removed settings

  • The query-local-address6 setting has been removed. It already was deprecated.

4.3.x to 4.4.0

Response Policy Zones (RPZ)

To conform better to the standard, RPZ processing has been modified. This has consequences for the points in the resolving process where matches are checked and callbacks are called. See Response Policy Zones (RPZ) for details. Additionally a new type of callback has been introduced: policyEventFilter().

Dropping queries from Lua callbacks

The method to drop a query from a Lua callback has been changed. Previously, you could set rcode to pdns.DROP. See Callback Semantics for the new method.

Parsing of unknown record types

The parsing (from zone files) of unknown records types (of the form \# <length> <hex data>) has been made more strict. Previously, invalid formatted records could produce inconsistent results.

Deprecated and changed settings

  • The query-local-address setting has been modified to be able to include both IPv4 and IPv6 addresses.
  • The query-local-address6 setting is now deprecated.

New settings

  • The dns64-prefix setting has been added, enabling common cases of DNS64 handling without having to write Lua code.
  • The proxy-protocol-from and proxy-protocol-maximum-size settings have been added to allow for passing of Proxy Protocol Version 2 headers between a client and the recursor.
  • The record-cache-shards setting has been added, enabling the administrator to change the number of shards in the records cache. The value of the metric record-cache-contended divided by record-cache-acquired indicates if the record cache locks are contended. If so, increasing the number of shards can help reducing the contention.

4.2.x to 4.3.0

Lua Netmask class methods changed

  • Netmask class methods isIpv4 and isIpv6 have been deprecated in Lua, use Netmask.isIPv4() and Netmask.isIPv6() instead. In C++ API these methods have been removed.

socket-dir changed

The default socket-dir has changed to include pdns-recursor in the path. For non-chrooted setups, it is now whatever is passed to --with-socketdir during configure (/var/run by default) plus pdns-recursor. The systemd unit-file is updated to reflect this change and systemd will automatically create the directory with the proper permissions. The packaged sysV init-script also creates this directory. For other operating systems, update your init-scripts accordingly.

Systemd service and permissions

The systemd service-file that is installed no longer uses the root user to start. It uses the user and group set with the --with-service-user and --with-service-group switches during configuration, “pdns” on Debian and “pdns-recursor” on CentOS by default. This could mean that PowerDNS Recursor cannot read its configuration, lua scripts, auth-zones or other data. It is recommended to recursively chown directories used by PowerDNS Recursor:

# For Debian-based systems
chown -R root:pdns /etc/powerdns

# For CentOS and RHEL based systems
chown -R root:pdns-recursor /etc/pdns-recursor

Packages provided on the PowerDNS Repository will chown directories created by them accordingly in the post-installation steps.

New settings

  • The allow-trust-anchor-query setting has been added. This setting controls if negative trust anchors can be queried. The default is no.
  • The max-concurrent-requests-per-tcp-connection has been added. This setting controls how many requests are handled concurrently per incoming TCP connection. The default is 10.
  • The max-generate-steps setting has been added. This sets the maximum number of steps that will be performed when loading a BIND zone with the $GENERATE directive. The default is 0, which is unlimited.
  • The nothing-below-nxdomain setting has been added. This setting controls the way cached NXDOMAIN replies imply non-existence of a whole subtree. The default is dnssec which means that only DNSSEC validated NXDOMAINS results are used.
  • The qname-minimization setting has been added. This options controls if QName Minimization is used. The default is yes.

4.1.x to 4.2.0

Two new settings have been added:

  • xpf-allow-from can contain a list of IP addresses ranges from which XPF (X-Proxied-For) records will be trusted.
  • setting-xpf-rr-code should list the number of the XPF record to use (in lieu of an assigned code).

4.0.x to 4.1.0

loglevel defaulted to 4 but was always overridden to 6 during the startup. The issue has been fixed and the default value set to 6 to keep the behavior consistent.

The --with-libsodium configure flag has changed from ‘no’ to ‘auto’. This means that if libsodium and its development header are installed, it will be linked in.

4.0.3 to 4.0.4

One setting has been added to limit the risk of overflowing the stack:

4.0.0 to 4.0.1

Two settings have changed defaults, these new defaults decrease CPU usage: