As part of a POC I deployed a pair of HA F5 LTM/GTM at home to use for all things DNS based. It is an indulgent over the top DNS solution for a 1 bed flat, but hey-ho we are in a pandemic….. This guide does not go through the HA F5 or GTM (still cant stop calling it that) configuration, it is focussed around using ZoneRunner for DNS (bind) with these zones transferred into DNS express and serviced by a listener.
The process for building this is as follows:
- Create the DNS Cache to do external recursive DNS lookups and cache the responses
- Create the DNS profile that holds the DNS configuration and bind to a listener that answers DNS queries
- Create a local BIND DNS instance (ZoneRunner) that the DNS records will be added into, this could be an external DNS server rather than ZoneRunner. It is only used to manage the DNS entries, it is not what serves the responses
- Create the DNS zone that DNS express will populate from ZoneRunner and use to serve up the answers from
Table Of Contents
DNS Cache
When you configure the DNS cache feature, the BIG-IP system resolves DNS requests and responds to repeated DNS queries by serving answers from the cache optimizing responses to DNS queries. There are three cache types, in all types the LTM uses the TTL (time-to-live) setting as received from the authoritative DNS server.
- Transparent: Sends the query to a DNS server (through a DNS server pool) for resolution
- Resolver: Performs recursive DNS lookups through the DNS root nameservers (by default uses those published by InterNIC) or to other primary servers that you choose (using Root Hints)
- Validating Resolver: Resolves the query, verifies the response using a DNSSEC key and stores the response in the DNS cache
The configuration for all of these types follows the same steps:
- Configure the cache type
- Configure how the cache is to be populated
- Configure how the cache will answer DNS queries
I used a resolver cache with the default settings, the configuration is auto-synced between the LTM HA pair.
tmsh
create ltm dns cache resolver dns_cache
list ltm dns cache
show ltm dns cache
DNS Profile and listener
The listener listens for DNS packets on a specified IP address and either handles the request directly or forwards the request to the appropriate resource. A DNS profile is bound to the listener and holds all the system wide DNS feature settings. If you are doing external and internal resolution best practice dictates that you to have separate DNS profiles for each so that you can disable features (such as BIND) that you would not have open to the Internet.
By default DNS express and Use BIND Server on BIG-IP are already enabled, I also enabled caching and associated the previously created cache. If you want logging of the DNS traffic you would create a logging profile (dns-local-syslog for local syslog) and associate it in the DNS profile.
create ltm profile dns internal_dns cache dns_cache enable-cache yes
list ltm profile dns
show ltm profile dns
A listener is created for TCP and UDP and looks very much like a normal LTM VIP, just with a lot less options. Can use a Self-IP or a floating-IP for the listener, only self-IPs can direct name resolution requests to BIND. As I am using DNS express rather than BIND I used a floating IP shared between HA LTMs. This only needs configuring on one device as both the profile and listener are synced as part of the LTM HA.
tmsh create gtm listener dns_tcp address 10.10.10.254 port 53 ip-protocol udp profiles add { internal_dns } vlans add { hme_compute_vl10 } vlans-enabled
tmsh create gtm listener dns_udp address 10.10.10.254 port 53 ip-protocol tcp profiles add { internal_dns } vlans add { hme_compute_vl10 } vlans-enabled
show gtm listener
list gtm listener
If you wanted to do load balancing to other DNS servers you could add a load balancing pool to the listener with DNS servers as the nodes and a DNS monitor health check (queries an A record domain name and expects a certain answer).
ZoneRunner
If you want to use ZoneRunner (Big-IP on-box BIND) you should always use DNS Express (high-speed authoritative DNS server) to serve those zones as the local BIND instance is not optimised to answer large numbers of DNS resolution requests nor include any of the Big-IP DoS features.
I configured ZoneRunner as the authoritative DNS server for the zone stesworld.com with a PTR of 10.0.0.0/8 to cover all my internal networks. The configuration is standard DNS, below are a few of the main settings.
- Zone name: This is your domain name, it must have a proceeding . (for example stesworld.com.). There are 5 types of zones:
- Primary (Master): Authoritative zone that responds to DNS queries for the domain or sub-domain. At minimum must have SOA and NS resource records for the zone
- Secondary (Slave): Copy of the principal zone (files) which responds authoritatively for the zone. SOA sets interval to query primary zone
- Stub: Similar to secondary zones, except that stub zones contain only the NS records for the zone
- Forward: The zone file for a forwarding zone contains only information to forward DNS queries to another name server on a per-zone (or per-domain) basis
- Hint: A hint zone file specifies an initial set of root name servers for the zone that local name server query for the most recent list of them
- Records Creation Method: Can import records from a file or transfer them from a server if you already have a DNS server
- Create Reverse Zone: I manually defined 10.in-addr.arpa. 16.172.in-addr.arpa. and 168.192.in-addr.arpa. to it keep it neat, guess may not want to create something so broad in production
- Options: By default has
allow-update { localhost; };meaning only local host can make changes to the zone - Records Created: Must create an SOA and NS record
- SOA (Start of authority): Only created at the time the master zone is created to indicate that a nameserver is the best authoritative source of information for a particular zone
- TTL: The default time-to-live (TTL) for the SOA record
- Master Server: The server (ns1.stesworld.com) that stores the original (master) copies of all zone records and that other DNS servers can automatically transfer zone files from
- NS (nameserver): The authoritative nameserver for a given domain, i.e. which server contains the actual DNS records. I used ns1.stesworld.com with the local loopback address of 127.0.0.1 as DNS express locally on the active F5 will be talking to this
- SOA (Start of authority): Only created at the time the master zone is created to indicate that a nameserver is the best authoritative source of information for a particular zone
By default DNS recursion is disabled, there are several options to allow it to provide non-authoritative responses (Internet address lookups):
- Pool of load balanced DNS servers behind the listener that do the recursive lookups
- Root hints or Forwarders, both are lists of servers that are used to resolve queries that the local DNS server can’t resolve on its own
- DNS Cache performs this same function of recursive DNS lookups through the root name servers (by default uses the DNS root nameservers published by InterNIC) but with the added benefit of caching the responses to optimize future responses
The BIND DNS server zone files (ZoneRunner) are not by default copied between LTM HA members. The only way to replicate this between members is by enabling Synchronize DNS Zone File within the GTM (DNS) synchronization group. Everytime a change is made to the zone file the zones serial number is incremented by one, all synchronized zones should have the exact same serial number.
DNS Express
The idea behind DNS Express is for it to serve as the authoritative DNS server answering for records from zones populated using a zone transfer from other DNS servers (local BIND server on BIG-IP or any other DNS server). The BIG-IP system acts as a high-speed, authoritative secondary DNS server allowing it to improve DNS performance and mitigate DDoS attacks (as it doesn’t run full BIND is not as vulnerable).
- Performs zone transfers from multiple primary DNS servers that are responsible for different zones
- Serves DNS records faster than both the primary DNS servers and the local BIND server
- DNS express is only used to answer for records it has, any recursive ones (via DNS cache) wont be seen in DNS express statistics
DNS NOTIFY messages allow a primary DNS server to utilize a “push” mechanism for notifying secondary DNS servers (in this case DNS express) that it has been updated with records that need to be replicated. If the secondary DNS server does not have the most up-to-date record it requests an update using a full zone transfer (AXFR) or an Incremental Zone Transfer (IXFR) to “pull” zone changes from their primary servers. In a DNS Notify configuration the IP addresses for all secondary DNS servers must be entered into the notify list of the primary DNS servers.
1. Create a nameserver object: It is the authoritative DNS server (primary) that hosts the DNS zone, as I am using ZoneRunner will be 127.0.0.1.
tmsh create ltm dns nameserver local_zonerunner { address 127.0.0.1 tsig-key none }
2. Configure DNS server to send NOTIFY: ZoneRunner must send the NOTIFY message (also-notify) to DNS express (::1 represents the localhost 127.0.0.1) when resource records are changed (the master zone file is reloaded). This could be added in each zone but is easier to add it under options in the named Configuration file to apply to all zones.
To do so in CLI you need to add this to the options section of /var/named/config/named.conf and restart services.
also-notify {
::1 port 5353;
};
bigstart restart named
bigstart restart zrd
Is probably easier to do this task in GUI as you are just editing the same file and it will automatically check formatting and restart services for you.
3. Create a DNS zone: Need to create a DNS zone (is not a ZoneRunner zone) for each zone you want the DNS Express engine to answer DNS queries (including reverse lookups) specifying the nameserver (step 1) as the authoritative DNS server. This zone name MUST be exactly the same on both the BIG-IP system and the authoritative DNS server.
DNS Express can either Consume (seen only by DNS Express), Bypass (only goes to a back-end DNS server) or Repeat (goes to both DNS Express and any back-end DNS server) NOTIFY messages. I left this as the the default of consume. The IP address of the authoritative primary DNS Server (nameserver) is allowed by default, it does not need to be added to ‘Allow NOTIFY From’ in the zone configuration.
create ltm dns zone stesworld.com dns-express-server local_zonerunner
create ltm dns zone 10.in-addr.arpa dns-express-server local_zonerunner
create ltm dns zone 16.172.in-addr.arpa dns-express-server local_zonerunner
create ltm dns zone 168.192.in-addr.arpa dns-express-server local_zonerunner
list ltm dns zone
show ltm dns zone | grep -A 3 Status
show ltm dns zone stesworld.com
If the DNS express server is to respond to zone transfer requests from other DNS nameservers (clients) these need adding as nameserver objects under Zone Transfer Client as well as enabling Zone Transfer in the DNS profile.
If you are not using the local BIND server (ZoneRunner) for the primary DNS servers and it is a HA configuration I think you may need to add a LTM virtual server to ensure all BIG-IP systems in the HA receive and respond to DNS notify messages. The way it is explained in the documents isn’t completely clear, I would have thought this is replicated by ‘Synchronize DNS Zone File’ within the GTM (DNS) synchronization group.
If your BIG-IP systems are in an HA configuration, zone transfer updates will not be processed by standby BIG-IP systems when using only the DNS request virtual server (listener). As a result, newly active BIG-IP systems may respond to DNS requests using stale DNS zone information. A UDP local-only virtual server must be created on each BIG-IP system in the HA configuration to respond to DNS notify requests from an authoritative DNS server. The destination IP addresses used for the local-only virtual servers must be the notify IP addresses used by the authoritative DNS server. Bind DNS servers use also-notify to determine which secondary DNS servers receive zone update.
Verification
The bind files are stored in /var/named/config/namedb and can be viewed using cat, this is where the dynamic update journal file is stored.
ls -l /var/named/config/namedb/
cat /var/named/config/namedb/db.external.stesworld.com.
Can look at the statistics for a DNS Express Zone such as state, number of RRs, message count and serial number.
tmsh show ltm dns zone stesworld.com
To verify transfers for a particular zone are successful compare the authoritative DNS server zone file serial with that of DNS Express Zone.
tmsh show ltm dns zone stesworld.com | grep Serial
dnsxdump can be used to view the DNS Express database information, which includes zone information, all the resource records and statistics. F5 recommends that you run this diagnostic utility during a maintenance window or period of low traffic to reduce any possible impact to traffic.
When looking at the DNS cache you need to remember it is only going to have cached records for lookups from non-locally hosted zones (like the resources on the Internet), local zone records are in the DNS express zone.
tmsh show ltm dns cache See stats on queries and responses
tmsh show ltm dns cache records rrset cache dns_cache count-only The number of records, need to specify cache name (dns_cache)
tmsh show ltm dns cache records rrset cache dns_cache All records
Transaction Signatures (TSIG)
TSIG is a networking protocol that is defined in RFC2845 to provide a secure method for communicating from a primary to a secondary DNS server. When used a cryptographic signature generated using a shared key and is added to all DNS packets exchanged between the servers. This ensures that the DNS packets originate from an authorized name server and have not been altered on route. In addition to a key, the protocol includes a timestamp so that communications cannot be intercepted and used at a later time.
DNS Express can use TSIG keys to authenticate communications about zone transfers between the BIG-IP system and authoritative DNS servers, and between the BIG-IP system and DNS nameservers (clients). The key can be added in several locations:
- Authoritative DNS server (master) nameserver: When the DNS server sends a NOTIFY message DNS Express responds with a TSIG-signed zone transfer request and the DNS server returns a TSIG-signed zone transfer
- DNS nameserver (client): When the client sends a TSIG-signed zone transfer request DNS Express returns a TSIG-signed zone transfer
- DNS zone: The system uses this TSIG key when the zone on the BIG-IP system is a proxy for the zone on the server and either removes the key when client sends TSIG-signed zone transfer request or adds the key when a client sends unsigned zone transfer request
Is no real need for this as am running BIND locally, the commands to configure the key and assign it are as below:
create ltm dns tsig-key <key name> algorithm <hmac> secret <TSIG key>
create ltm dns nameserver <name> address <IP address> tsig-key <key name>
create ltm dns zone <zone domain> dns-express-server <DNS nameserver> server-tsig-key <key>
Adding Resource Records using nsupdate
F5 recommends using ZoneRunner to manage the DNS/BIND file rather than manually editing the file. It uses dynamic update (RFC2136) to make zone changes by storing them in the zone’s journal file and periodically flushing the complete contents of the zone file as well incorporating changes at startup. This means that the zone file is not the absolute source of truth when allowing dynamic updates so should not be manually updated as it is not guaranteed to contain the most recent dynamic changes.
Handling resource record (RR) entries in the GUI is not scalable so I needed a method to be able to do this programmatically. There is no tmsh access into ZoneRunner so the only option I found was to use the Linux utility nsupdate as it also follows RFC2136 in allowing resource records to be added or removed from a zone without manually editing the zone file. A single nsupdate update request can contain requests to add or remove more than one resource record.
By default this can only be run locally on the F5 against its localhost address (127.0.0.1). If you adding entries in more than one zone you need to leave a space between them or it will fail with the error message ‘update failed: NOTZONE’
[root@hme-bip-lb01:Active:In Sync] config # nsupdate
server 127.0.0.1
update add test1.stesworld.com 3600 A 10.10.10.250
update add test2.stesworld.com 3600 A 10.10.10.251
update add 250.10.10.10.in-addr.arpa. 3600 PTR test1.stesworld.com
update add 250.10.10.10.in-addr.arpa. 3600 PTR test1.stesworld.com
show
send
If you want to manage the entries remotely you need to add your your management addresses to the allow-update list of each zone. It can be done in the CLI but for this is probably easier in GUI as it handles formatting and service restarts automatically.
At the bottom of the Options box of the ZoneRunner named Configuration add new a named ACL of the networks or IPs that will be allowed to make the updates. In this case it is called internal-mgmt and allows access from 10.10.10.0/24 and 10.10.20.0/24.
Under each ZoneRunner zone Options add the name of the acl (internal-mgmt) to the allow-update list.
Anyone on the specified networks can update records so if using in the real world you could associate a TSIG key to the nameserver and that can be specified in nsupdate to give some added security.
The problem I have come across with this method is that the changes made by nsupdate are not automatically updated to the standby LTM device like it would if done in the GUI. When the change is made the active device DNS Express database is updated (so DNS requests are answered) and can be seen in ZoneRunner, however the BIND file is not updated for a further 13 minutes. The standby device does not learn about the new entry until either a GUI change is made (will automatically update all other records) or you wait until the primary device has updated the BIND file before using touch /var/named/config/named.conf to trigger a DNS zone file synchronization.
Rather than waiting the 13 minutes you can trigger the journal file to update the zone file immediately and then use touch to perform the sync. In a normal production environment I doubt you would be using BIND on the F5 for DNS so this wouldn’t be a problem you would come across.
rndc sync -clean
touch /var/named/config/named.conf
References
K02439348: Creating or transferring DNS zone files using the ZoneRunner utility
K45907236: Overview of BIG-IP DNS synchronization
K45411181: Configuring DNS Express using tmsh
Configuring DNS Express
K7032: Freezing zone files to allow manual update to ZoneRunner-managed zone