Logging DNS messages with Protocol Buffers

The PowerDNS Recursor has the ability to emit a stream of protocol buffers messages over TCP, containing information about queries, answers and policy decisions.

Messages contain the IP address of the client initiating the query, the one on which the message was received, whether it was received over UDP or TCP, a timestamp and the qname, qtype and qclass of the question. In addition, messages related to responses contain the name, type, class and rdata of A, AAAA and CNAME records present in the response, as well as the response code.

Finally, if a RPZ or custom Lua policy has been applied, response messages also contain the applied policy name and some tags. This is particularly useful to detect and act on infected hosts.

Configuring Protocol Buffer logs

Protobuf export to a server is enabled using the protobufServer() directive:

protobufServer(servers [, options]))

New in version 4.2.0.

Send protocol buffer messages to one or more servers for incoming queries and/or outgoing responses. The client address may be masked using setProtobufMasks(), for anonymization purposes.

Parameters:
  • servers (string or list of strings) – The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  • options (table) – A table with key=value pairs with options.

Options:

  • timeout=2: int - Time in seconds to wait when sending a message
  • maxQueuedEntries=100: int - How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime=1: int - How long to wait, in seconds, between two reconnection attempts
  • taggedOnly=false: bool - Only entries with a policy or a policy tag set will be sent
  • asyncConnect: bool - When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  • logQueries=true: bool - Whether to export queries
  • logResponses=true: bool - Whether to export responses
  • exportTypes={'A', 'AAAA', 'CNAME'}: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
protobufServer(server[[[[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], maskV4=32], maskV6=128], asyncConnect=false], taggedOnly=false])

Deprecated since version 4.2.0.

Parameters:
  • server (string) – The IP and port to connect to
  • timeout (int) – Time in seconds to wait when sending a message
  • maxQueuedEntries (int) – How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime (int) – How long to wait, in seconds, between two reconnection attempts
  • maskV4 (int) – network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  • maskV6 (int) – Same as maskV4, but for IPv6. Defaults to 128.
  • taggedOnly (bool) – Only entries with a policy or a policy tag set will be sent.
  • asyncConnect (bool) – When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued..
setProtobufMasks(maskv4, maskV6)

New in version 4.2.0.

Parameters:
  • maskV4 (int) – network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  • maskV6 (int) – Same as maskV4, but for IPv6. Defaults to 128.

Logging outgoing queries and responses

While protobufServer() only exports the queries sent to the recursor from clients, with the corresponding responses, outgoingProtobufServer() can be used to export outgoing queries sent by the recursor to authoritative servers, along with the corresponding responses.

outgoingProtobufServer(servers[, options])

New in version 4.2.0.

Send protocol buffer messages to one or more servers for outgoing queries and/or incoming responses.

Parameters:
  • servers (string or list of strings) – The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  • options (table) – A table with key=value pairs with options.

Options:

  • timeout=2: int - Time in seconds to wait when sending a message
  • maxQueuedEntries=100: int - How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime=1: int - How long to wait, in seconds, between two reconnection attempts
  • taggedOnly=false: bool - Only entries with a policy or a policy tag set will be sent
  • asyncConnect: bool - When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  • logQueries=true: bool - Whether to export queries
  • logResponses=true: bool - Whether to export responses
  • exportTypes={'A', 'AAAA', 'CNAME'}: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
outgoingProtobufServer(server[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], asyncConnect=false])

Deprecated since version 4.2.0.

Parameters:
  • server (string) – The IP and port to connect to
  • timeout (int) – Time in seconds to wait when sending a message
  • maxQueuedEntries (int) – How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime (int) – How long to wait, in seconds, between two reconnection attempts
  • asyncConnect (bool) – When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued..

Protocol Buffers Definition

The protocol buffers message types can be found in the dnsmessage.proto file and is included here:

/*
 * This file describes the message format used by the protobuf logging feature in PowerDNS and dnsdist.
 *
 * MIT License
 * 
 * Copyright (c) 2016-now PowerDNS.COM B.V. and its contributors.
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in all
 * copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */
syntax = "proto2";

message PBDNSMessage {
  enum Type {
    DNSQueryType = 1;                           // Query received by the service
    DNSResponseType = 2;                        // Response returned by the service
    DNSOutgoingQueryType = 3;                   // Query sent out by the service to a remote server
    DNSIncomingResponseType = 4;                // Response returned by the remote server
  }
  enum SocketFamily {
    INET = 1;                                   // IPv4 (RFC 791)
    INET6 = 2;                                  // IPv6 (RFC 2460)
  }
  enum SocketProtocol {
    UDP = 1;                                    // User Datagram Protocol (RFC 768)
    TCP = 2;                                    // Transmission Control Protocol (RFC 793)
    DOT = 3;                                    // DNS over TLS (RFC 7858)
    DOH = 4;                                    // DNS over HTTPS (RFC 8484)
    DNSCryptUDP = 5;                            // DNSCrypt over UDP (https://dnscrypt.info/protocol)
    DNSCryptTCP = 6;                            // DNSCrypt over TCP (https://dnscrypt.info/protocol)
  }
  enum PolicyType {
    UNKNOWN = 1;                                // No RPZ policy applied, or unknown type
    QNAME = 2;                                  // Policy matched on the QName
    CLIENTIP = 3;                               // Policy matched on the client IP
    RESPONSEIP = 4;                             // Policy matched on one of the IPs contained in the answer
    NSDNAME = 5;                                // Policy matched on the name of one nameserver involved
    NSIP = 6;                                   // Policy matched on the IP of one nameserver involved
  }
  enum PolicyKind {
    NoAction = 1;                               // No action taken
    Drop = 2;                                   // https://tools.ietf.org/html/draft-vixie-dns-rpz-04 3.4
    NXDOMAIN = 3;                               // https://tools.ietf.org/html/draft-vixie-dns-rpz-04 3.1
    NODATA = 4;                                 // https://tools.ietf.org/html/draft-vixie-dns-rpz-04 3.2
    Truncate= 5;                                // https://tools.ietf.org/html/draft-vixie-dns-rpz-04 3.5
    Custom = 6;                                 // https://tools.ietf.org/html/draft-vixie-dns-rpz-04 3.6
  }
  enum VState {
    Indeterminate = 1;
    Insecure = 2;
    Secure = 3;
    BogusNoValidDNSKEY = 4;
    BogusInvalidDenial = 5;
    BogusUnableToGetDSs = 6;
    BogusUnableToGetDNSKEYs = 7;
    BogusSelfSignedDS = 8;
    BogusNoRRSIG = 9;
    BogusNoValidRRSIG = 10;
    BogusMissingNegativeIndication = 11;
    BogusSignatureNotYetValid = 12;
    BogusSignatureExpired = 13;
    BogusUnsupportedDNSKEYAlgo = 14;
    BogusUnsupportedDSDigestType = 15;
    BogusNoZoneKeyBitSet = 16;
    BogusRevokedDNSKEY = 17;
    BogusInvalidDNSKEYProtocol = 18;
  }
  required Type type = 1;                       // Type of event
  optional bytes messageId = 2;                 // UUID, shared by the query and the response
  optional bytes serverIdentity = 3;            // ID of the server emitting the protobuf message
  optional SocketFamily socketFamily = 4;
  optional SocketProtocol socketProtocol = 5;
  optional bytes from = 6;                      // DNS requestor (client) as 4 (IPv4) or 16 (IPv6) raw bytes in network byte order
  optional bytes to = 7;                        // DNS responder (server) as 4 (IPv4) or 16 (IPv6) raw bytes in network byte order
  optional uint64 inBytes = 8;                  // Size of the query or response on the wire
  optional uint32 timeSec = 9;                  // Time of message reception (seconds since epoch)
  optional uint32 timeUsec = 10;                // Time of message reception (additional micro-seconds)
  optional uint32 id = 11;                      // ID of the query/response as found in the DNS header

  message DNSQuestion {
    optional string qName = 1;                  // Fully qualified DNS name (with trailing dot)
    optional uint32 qType = 2;                  // https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-4
    optional uint32 qClass = 3;                 // Typically 1 (IN), see https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-2
  }
  optional DNSQuestion question = 12;           // DNS query received from client

  message DNSResponse {
    // See exportTypes in https://docs.powerdns.com/recursor/lua-config/protobuf.html#protobufServer
    // for the list of supported resource record types.
    message DNSRR {
      optional string name = 1;                 // Fully qualified DNS name (with trailing dot)
      optional uint32 type = 2;                 // https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-4
      optional uint32 class = 3;                // Typically 1 (IN), see https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-2
      optional uint32 ttl = 4;                  // TTL in seconds
      optional bytes rdata = 5;                 // raw address bytes in network byte order for A & AAAA; text representation for others, with fully qualified (trailing dot) domain names
      optional bool  udr = 6;                   // True if this is the first time this RR has been seen for this question
    }
    optional uint32 rcode = 1;                  // DNS Response code, or 65536 for a network error including a timeout
    repeated DNSRR rrs = 2;                     // DNS resource records in response
    optional string appliedPolicy = 3;          // Filtering policy (RPZ or Lua) applied
    repeated string tags = 4;                   // Additional tags applied
    optional uint32 queryTimeSec = 5;           // Time of the corresponding query reception (seconds since epoch)
    optional uint32 queryTimeUsec = 6;          // Time of the corresponding query reception (additional micro-seconds)
    optional PolicyType appliedPolicyType = 7;  // Type of the filtering policy (RPZ or Lua) applied
    optional string appliedPolicyTrigger = 8;   // The RPZ trigger
    optional string appliedPolicyHit = 9;       // The value (qname or IP) that caused the hit
    optional PolicyKind appliedPolicyKind = 10; // The Kind (RPZ action) applied by the hit
    optional VState validationState = 11;       // The DNSSEC Validation State
  }

  optional DNSResponse response = 13;
  optional bytes originalRequestorSubnet = 14;  // EDNS Client Subnet value (4 or 16 raw bytes in network byte order)
  optional string requestorId = 15;             // Username of the requestor
  optional bytes initialRequestId = 16;         // UUID of the incoming query that initiated this outgoing query or incoming response
  optional bytes deviceId = 17;                 // Device ID of the requestor (could be mac address IP address or e.g. IMEI, format implementation dependent)
  optional bool  newlyObservedDomain = 18;      // True if the domain has not been seen before
  optional string deviceName = 19;              // Device name of the requestor
  optional uint32 fromPort = 20;                // Source port of the DNS query (client)
  optional uint32 toPort = 21;                  // Destination port of the DNS query (server)

  message MetaValue {
    repeated string stringVal = 1;
    repeated int64 intVal = 2;
  }

  message Meta {
    required string key = 1;                    // MUST be unique, so if you have multiple values they must be aggregated into on Meta
    required MetaValue value = 2;
  }
  repeated Meta meta = 22;                      // Arbitrary meta-data - to be used in future rather than adding new fields all the time

  // The well known EventTrace event numbers
  enum EventType {
                                                // Range 0..99: Generic events
    CustomEvent = 0;                            // A custom event
    ReqRecv = 1;                                // A request was received
    PCacheCheck = 2;                            // A packet cache check was initiated or completed; value: bool cacheHit
    AnswerSent = 3;                             // An answer was sent to the client

                                                // Range 100: Recursor events
    SyncRes = 100;                              // Recursor Syncres main function has started or completed; value: int rcode
    LuaGetTag = 101;                            // Events below mark start or end of Lua hook calls; value: return value of hook
    LuaGetTagFFI = 102;
    LuaIPFilter = 103;
    LuaPreRPZ = 104;
    LuaPreResolve = 105;
    LuaPreOutQuery = 106;
    LuaPostResolve = 107;
    LuaNoData = 108;
    LuaNXDomain = 109;
  }

  message Event {
    required int64 ts = 1;                      // Timestamp in ns relative to time of creation of event trace data structure
    required EventType event = 2;               // Type of event
    required bool start = 3;                    // true for "start" events, false for "completed" events
    optional bool boolVal = 4;                  // Below are optional values associated with events
    optional int64 intVal = 5;
    optional string stringVal = 6;
    optional bytes bytesVal = 7;
    optional string custom = 8;                 // The name of the event for custom events
  }
  repeated Event trace = 23;
}

message PBDNSMessageList {
  repeated PBDNSMessage msg = 1;
}

Logging in dnstap format using framestreams

Define the following function to enable logging of outgoing queries and/or responses in dnstap format. The recursor must have been built with configure --enable-dnstap to make this feature available.

dnstapFrameStreamServer(servers[, options])

New in version 4.3.0.

Send dnstap formatted message to one or more framestream servers for outgoing queries and/or incoming responses.

Parameters:
  • servers (string or list of strings) – Either a pathname of a unix domain socket starting with a slash or the IP:port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  • options (table) – A table with key=value pairs with options.

Options:

  • logQueries=true: bool - log outgoing queries
  • logResponses=true: bool - log incoming responses

The following options apply to the settings of the framestream library. Refer to the documentation of that library for the default values, exact description and allowable values for these options. For all these options, absence or a zero value has the effect of using the library-provided default value.

  • bufferHint=0: unsigned
  • flushTimeout=0: unsigned
  • inputQueueSize=0: unsigned
  • outputQueueSize=0: unsigned
  • queueNotifyThreshold=0: unsigned
  • reopenInterval=0: unsigned