Removing DNS as a Single Point of Failure

A few months ago, Dyn's DNS service suffered an outage[1] because of a massive DDOS attack by the Mirai Botnet[2].

In the Dyn case, it was estimated that the attack involved 100,000 malicious endpoints. The botnet sent around 1 TB of traffic per second to the company’s servers, meaning legitimate requests were denied.

What do we need to do to survive the next outage?

We can add multiple NS records pointing to multiple DNS providers. So, even if the DNS queries are getting timed-out on one provider the recursive DNS server can fetch the details from the other provider.

In our case we will use Dyn as the primary and Route53 as the secondary. Here primary means all the record changes will be done on Dyn and it will be synced to Route53 afterwards.

How can we keep the primary and secondary DNS services in sync?

The DNS system provides the Notify[3] feature. With this feature a primary DNS provider can notify the secondary providers that the records have changed. After receiving the Notify message, secondary servers can use AXFR[4] or IXFR[5] query type to fetch the zone records.

What is the difference between AXFR and IXFR query types?

AXFR query initiates a full zone transfer. In other words, it tells the other DNS server to send all the records in the specified zone.

$ dig AXFR example.com @primary-dns-server

IXFR query, on the other hand, is an incremental one. It asks the other DNS server to send across the changes done after a specified point.

$ dig ixfr=serial-number example.com @primary-dns-server

Here the serial-number is the version number maintained by the primary DNS sever.

If you dig with query type SOA, you can check the current serial number of any domain.
Example:

$ dig SOA +multiline shreysinha.in;

Output:

ANSWER SECTION:
shreysinha.in.		3600 IN	SOA ns1.p24.dynect.net. admin.shrey.io. (
				8          ; serial
				3600       ; refresh (1 hour)
				600        ; retry (10 minutes)
				604800     ; expire (1 week)
				1800       ; minimum (30 minutes)
				)

Here the serial is the current version number. If any change is done in the zone, then the serial number will increase.

Demo

Whitelist an IP so that you can get notified on this IP and also send AXFR queries from this IP.

  • Installing nsnotifyd to receive Notify messages from Dyn.
$ apt-get install libbind-dev gcc curl dnsutils make build-essential; wget http://dotat.at/prog/nsnotifyd/nsnotifyd-1.6.tar.gz; tar -xvf nsnotifyd-1.6.tar.gz; cd nsnotifyd-1.6; ./configure; make all; make prefix=/usr/local install

Whitelist Dyn's IP address so that the nsnotifiy daemon can receive Notify messages on port 53.

  • Installing the command which nsnotifyd will use to fetch DNS records
$ wget https://raw.githubusercontent.com/shreysinha/DNSSyncScripts/master/recordfetcher | chmod +x recordfetcher | sudo mv recordfetcher /usr/local/bin/
  • Installing cli53 to sync the zone file to Route53
$ wget https://github.com/barnybug/cli53/releases/download/0.8.7/cli53-linux-amd64;sudo mv cli53-linux-amd64 /usr/local/bin/cli53;sudo chmod +x /usr/local/bin/cli53
  • Running the nsnotifyd command as a daemon
$ nsnotifyd -a 0.0.0.0 recordfetcher shreysinha.in

This daemon will run in the background and will keep the secondary DNS server in sync with the primary.


  1. https://www.dynstatus.com/incidents/nlr4yrr162t8 ↩︎

  2. https://www.infosecurity-magazine.com/opinions/standards-security-great-ddos/ ↩︎

  3. https://www.ietf.org/rfc/rfc1996.txt ↩︎

  4. https://tools.ietf.org/html/rfc5936 ↩︎

  5. https://tools.ietf.org/html/rfc1995 ↩︎