Changelog

2.1.1

Released: Aug 13th, 2024

• dstore-dist: Fix issue with storage destination when no batch_buffer_size is configured

  References: pull request 339

• Add supervisord to dstore images for control by Cloud Control agent

  References: pull request 340

• topn-reporter: Fix logging issues in topn reporter

  References: pull request 338

• eventforwarder: Fix elastic wait loop issue and add option to append date to elastic index name

  References: pull request 335

• dstore-dist: Detect half-closed connections immediately for pdns destinations

  References: pull request 336

2.1.0

Released: Jul 8th, 2024

• dstore-dist: configurable instance name for kafka destinations to allow source dstore-dist instance to be identified

  References: pull request 332

• eventforwarder: new daemon to send filtered events to Elasticsearch for reporting purposes

  References: pull request 326

• dstore-dist: sasl auth support for kafka destinations (plain, scram-256, scram-512)

  References: pull request 329

• dstore-dist: add new prometheus metrics to count filter hits and accepted/rejected/received rates/sec

  References: pull request 325

• dstore-dist: new IPset filter to filter based on source IP/prefix

  References: pull request 324

• dstore-dist: new storage destination supporting local files or S3, and a variety of encodings including JSON, protobuf or Bind query logging

  References: pull request 327

• security: SLSA Level 3 for images (provenance, SBOM) and image signing

  References: pull request 323

• dstore-dist: support different framing options for outbound messages (pdns destinations only)

  References: pull request 322

2.0.0

Released: Dec 5th, 2023

• packaging: Enterprise Linux 9 Support and many python fixes packaging: Production image for dstore-dist and top-n-reporter uploaded to OX registry in CI

  References: pull request 389

• tests: Add and update dstore-dist tests in CI

  References: pull request 311

• cleanup: Remove legacy dstore code leaving only dstore-dist and related tools

  References: pull request 318

• security: Add features outstanding from Nixu code-audit

  References: pull request 313

• logging: Add structured logging to dstore-dist

  References: pull request 316

• api: Add history API to dstore-dist for debugging

  References: pull request 319

1.5.0

Released: May 3rd, 2023

• python: update requirements.txt to eliminate setuptools failures

  References: pull request 384

• docs: upload docs to docs.powerdns.com as part of build

  References: pull request 382, pull request 383, pull request 380

• dstore-dist: compression support for Kafka

  References: pull request 378

• go: Update Go to v1.20.2

  References: pull request 378

• dstore-dist: connect and write timeouts for PDNS destinations

  References: pull request 381

1.4.4

Released: October 20nd, 2022

• dnspb2json:

  • Include the dnspb2json tool as dstore-dist-dnspb2json in the dstore-dist RPM package.
  • -b option to record raw protobuf streams instead of outputting JSON
  • -L option to output one JSON object per line for easy grep and processing

  References: pull request 371

• dstore-dist: null Kafka key for better load balancing

  References: pull request 374

• ev_aggregator: fix elasticsearch webhook username/password config flag parsing

  References: pull request 370

• dstore-dist: add filters and JSON output for new protobuf fields

  New filters include:

  • tag_prefix
  • socket_protocol
  • is_newly_observed_domain
  • policy_type
  • policy_kind
  • validation_state and is_validation_state_bogus
  • trace_event and trace_match (see event-trace-enabled recursor setting)
  • meta_key, meta_key_string and meta_key_int

  The example config has been updated with examples for the new filters.

  dnspb2json now also supports a -d option that will output JSON in the same format as dstore-dist uses to write to Kafka queues. This format is different from the default format, e.g. it uses snake_case instead of camelCase.

  References: pull request 373

1.4.3

Released: April 22nd, 2022

1.4.3-rc1

Released: March 18th, 2022

• dstore-dist: Support appending arbitrary tags to messages (route specific)

  References: pull request 366

• dstore-dist: Build and test on Oracle Linux 8 instead of Centos 8

  References: pull request 367

• dstore-dist: Add TLS support for both inbound and outbound traffic topn: Add TLS support for inbount traffic

  References: pull request 365

• report: Support reporting not just on a per-user basis, but also per-device.

  References: pull request 363

• dstore-ev-aggregator: add support for (now mandatory) “cat:” and “rule:” prefixes while reading tag list

  References: pull request 361

• dstore-dist, topn: Add them configured to the development enviroment

  References: pull request 369, pull request 364

• Fix and update regression tests

  References: pull request 362

1.4.2

Released: October 15, 2021

1.4.2-rc1

Released: October 1, 2021

• top-n: support reporting by source IP

  References: pull request 359

• top-n: improve templates and provide example configuration files for dstore-dist, topn-reporter and kibana

  References: pull request 349

• ev_aggregator: performance improvements

  References: pull request 343

• dstore-dist: add is_incoming_response and is_outgoing_query filters

  References: pull request 357

• dstore-dist: exit with non-zero code upon error at startup

  References: pull request 348

• golang: upgrade protobuf dependency to a more recent version

  References: pull request 353

• egateway: document HTTP API

  References: pull request 352

• Add a documentation target to the CI

  References: pull request 350

1.4.1

Released: March 12, 2021

1.4.1-beta2

Released: February 23, 2021

1.4.1-beta1

Released: February 23, 2021

• dstore-dist: add file and syslog destinations support for logging

  References: pull request 346

• dstore-dist: improve unit tests for protobuf/kafka integration

  References: pull request 345

1.4.0

Released: January 14, 2021

1.4.0-beta3

Released: December 18, 2020

• build: update alpine linux base images and centos fixes

  References: pull request 344

1.4.0-beta2

Released: December 16, 2020

• dstore-dist: add support and tooling to measure and report Top N domains

  References: pull request 342

1.4.0-beta1

Released: November 27, 2020

• dstore-dist: support writing dns messages as JSON for Kafka destinations

  References: pull request 339

• all: support new Device ID format while keeping backward compatibility

  References: pull request 340

1.4.0-alpha2

Released: November 13, 2020

• dstore-dist: add TLS support for Kafka destination

  References: pull request 332

• dstore-dist: add support for blackhole destination

  References: pull request 329

• doc: fix pdf generation

  References: pull request 337

• python-dist: upgrade Twisted package

  References: pull request 327

• dstore-dist: kafka: allow multiple dnsmessage per kafka message

  References: pull request 333

1.4.0-alpha1

Released: October 27, 2020

• dstore-dist: add Kafka support as destination for outgoing messages

• dstore-dist: add sampling and rate limiting support

  References: pull request 314

• dstore-dist: allow filtering for qname and subdomains of qname

  References: pull request 323

1.3.3

Released: July 27, 2020

• Use unique build IDs for debug files.

  References: pull request 324

1.3.2

Released: July 22, 2020

1.3.2-beta2

Released: July 17, 2020

• Also build release packages for CentOS 8.

1.3.2-beta1

Released: July 17, 2020

• Add support for centos 8

  References: pull request 322

1.3.2-alpha1

Released: May 14, 2020

• Fix compilation issues with recent versions of the build chain

  References: pull request 311, pull request 310

• Add override file to the dstore-report-api gunicorn config

  References: pull request 316

• Update Event Aggregator to handle protobuf for non-filtered DNS queries

  References: pull request 318

1.3.1

Released: February 11, 2020

• Build and use our own python distribution.

  References: pull request 306

• Sharding support in dstore-dist and other improvements.

  References: pull request 305

• Aggregation feature for ev_aggregator.

  References: pull request 307

1.3.0

Released: November 19, 2019

1.3.0-beta3

Released: November 6, 2019

• report-api: Make sure a default value PROM_STATS_DIR is defined.

  References: pull request 302

• report-api: Fix dstore install on RHEL 7.

  References: pull request 300

1.3.0-beta2

Released: October 23, 2019

• event-aggregator: Add support to Prometheus metrics.

  References: pull request 278

• report-api: Add support to Prometheus metrics.

  References: pull request 279

1.3.0-beta1

Released: July 30, 2019

• egateway: Allow searching by device name.

  References: pull request 288

• text2tcp: Close the connection gracefully in order to avoid issues on the server end.

  References: pull request 264

• Fix several issues that came up deploying the dstore-1.3.0 alphas:

  • dstore-ev-aggregator: fix an issue preventing Redis authentication to work correctly when the password is specified in the config file instead of command line,
  • dstore-report-api: handle API queries correctly when usernames (and possibly other fields) can be tokenised,
  • dstore-ev-aggregator: add a retry mechanism to gracefully handle situations where Redis connections are broken.

  References: pull request 289

• dstore-report-api: Fix OpenAPI spec error where user_id was specified instead of username.

  References: pull request 287

1.3.0-alpha6

Released: June 21, 2019

• Fix a couple of nits for event aggregator and dstore_alert

  References: pull request 285

• UI: remove queries without response stat

  References: pull request 284

• Improves code readability

  References: pull request 265

• egateway: upgrade HTTP handling to libh2o

  References: pull request 250

1.3.0-alpha5

Released: June 14, 2019

• Fix in protobuf split function and batch pool handling.

  References: pull request 266

• Web UI: export results as CSV

  References: pull request 262

• dgrep: Allow to lookup for outgoing queries.

  References: pull request 260

• Show versions in UI.

  References: pull request 258

• dnspbgen: Add real-time flag to throttle message generation.

  References: pull request 253

• dgrep: Add timestamp range specifier option.

  References: pull request 246

1.3.0-alpha4

Released: May 29, 2019

Note: versions 1.3.0-alpha1, -alpha2 and -alpha3 are internal only.

• Reporting API

• Event Aggregator

• Make user grouping interval configurable for scan_malware.

1.2.7

Released: February 8, 2019

DEPLOYMENT NOTES

Django Prometheus Metrics

To monitor database query metrics, you must replace the ENGINE property of your database, replacing django.db.backends with django_prometheus.db.backends:

DATABASES = {
    'default': {
        'ENGINE': 'django_prometheus.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    },
}

This has already been applied to the default sqlite database.

Add API-key Option to Setup API Authentication

This PR adds support for egateway API authentication.

This is done by configuring egateway with the desired secret (api-key option). The secret must be provided through the HTTP x-api-key header.

To match that need, dstore-web EGATEWAY_URLS has been updated to support URIs like https://egateway.local:1234#s3cr3t.

Changelog

• egateway: limit the number of results scanned by egateway

• ui: malware scan ‘no results found’ message

• tcpdistro: prevent FD leak when opening a corrupted file

• egateway: fix a nullptr deref on a query w/ only additional filters

• egateway: add api-key option to setup api authentication

• egateway: allow multiple search terms in query

• dstore-web: django prometheus metrics

• malware_scan cmd fixes

• enable hardening measures (SSP, PIE, full RELRO, fortify)

• crow: properly stop metrics webserver

• ui: adjust search results fields and add tcp field

• dgrep: add support for customer and device query params

• prometheus: only display metric header for distinct metrics

• just as for the new dcat utility below, this adds --raw modifier to dgrep output. Combined with --quiet, you get a stream of raw protobuf messages instead of json output

• this adds a dcat utility that pretty-prints a dstore data file

• tcpdistro: add compression-level parameter

1.2.6

Released: January 14, 2019

• tcpdistro: Move bucket list to a LRU list with regular cleaning

1.2.5

Released: November 14, 2018

NOTE: DStore 1.2.5 needs to be reinstalled but after that upgrading will work again.

• Update dnsmessage.proto to sync with PDNS

• dstore-web: Fix accidental removal of /usr/share/dstore-web when upgrading dstore

• egateway: Add an ‘ecs-override-requestor’ option, export more values via the API

  • Clean up the protobuf bytes to ComboAddress conversions
  • Export ‘ecs’, ‘serverId’, a non-overridden ‘from’ and ‘tcp’ via the API
  • Add an ‘ecs-override-requestor’ option to be able to configure whether an ECS value should override the ‘from’ value when present (default, existing behaviour)

1.2.4

Released: October 5, 2018

• Grafana dashboard for dstore

• dstore-web: change default search range from 12 months to 1 hour

• dstore-web: fix error on packages upgrade

• pbscanner: fix the error message when reading from a corrupted file

• dirwalker: explictly remove copy constructor

1.2.3

Released: September 27, 2018

• Define systemd SyslogIdentifier

• metrics: Add Prometheus support to tcpdistro and egateway

• dgrep: Add username and device ID to output

• pbscanner: Close the directory file descriptor in DirWalker

1.2.2

Released: June 1, 2018

• Sharding

• Support for outgoing protobuf logs

• program:`dnspbgen

• Stability improvements

• Accept all 2xx HTTP codes as positive

1.2.1

Released: June 1, 2018

• Support New Notification Centre API

1.2.0

Released: May 28, 2018

• The dstore-web packages are now built against Python 3.6.

• Drop requirement for Protobuf 3.

• Add on-disk compression based on zstd. This is enabled by default.

• Add dstore-dist, a service that can duplicate and shard data over multiple dstore nodes.

• Add support for storing, searching and extracting per-device information.