DNS for private services

There are different means through which you can manage DNS records for your private services. You can use a single domain name to serve both internal and public records, different domain names for each, a hybrid DNS configuration in which you control DNS resolution based upon the origin of DNS query, and various other options. Which pattern you should choose depends on your use case and goals.

At Rokt, we have settled on a DNS configuration method known as dual-home DNS (also known as split-view DNS or split-horizon DNS). This enables proper name resolution of infrastructure resources from both inside and outside the local network using a single domain name. In our AWS environment, most requests originating from within our VPCs (Virtual Private Cloud) need to resolve to internal endpoints using private IP ranges. Requests originating from outside our VPCs, i.e. the internet, need to resolve to public endpoints with public IP ranges. In this post we will explain the use case we had at Rokt, why we settled on this strategy, and provide an example which demonstrates the actual implementation and its benefits.

Rokt used to operate a simple DNS infrastructure with host names for both public and internal service records on the same domain names. Later we introduced custom internal domains for each system in order to keep traffic internal. And as we are quickly moving towards microservice architectures, we started seeing undesirable behaviours on our network:

1. Security concerns due to internal-purpose service endpoints that are resolvable and potentially available to the Internet.
2. Security concerns due to private range IP addresses being disclosed through DNS.
3. Inefficient routing of traffic between microservices. Traffic between services should remain within our VPC and have fewer network hops. We would often see network traffic leaving our VPC via the NAT, only to come back via our Internet Gateways (IGW) to internet-facing load balancers (ALB).
4. Complexity for application configuration and logic. There was some efforts involved at the application level to correctly handle incoming requests for both real and custom domains.
5. Microservices communicating over HTTPS require SSL certificates and validate certificate authenticity. Obtaining SSL certificates requires domain name ownership validation, which in turn implies that all domain names are properly registered and queryable in public DNS infrastructure.

In order to address these concerns, we decided to implement a dual-home DNS configuration.

Implementing Dual-home DNS

Let’s introduce some basics about the AWS DNS management service, Route 53; a global AWS service which has Hosted Zones, administrative spaces to contain the DNS resource records for a given domain name. These resource records define how to resolve DNS queries for the host names in your domain. Route 53 supports two types of Hosted Zones: public and private. As their name suggests, public Hosted Zones contain records which are accessible via public Internet and private Hosted Zones contains records which are accessible only locally or internally i.e only within associated VPCs.

We achieved dual-home DNS configuration in Route 53 by creating both public and private Hosted Zones with the same domain name. Let’s consider an example with two microservices, service A and service B, to demonstrate the resulting configuration.

• We created both public and private Hosted Zones for the domain example.com.
• Service A should be accessible only within the VPC, hence its DNS record (service-a.example.com) has been added only in the private Hosted Zone.
• Service B should be accessible both within the VPC and through the Internet, so its DNS record (service-b.example.com) has been added in both the public and private Hosted Zones.
• Our VPC has been associated with private Hosted Zone example.com. All DNS queries for this domain and its host subdomains originating from this VPC will be resolved by this private Hosted Zone.
• DNS resolution for service B will be served in two different ways depending upon origin of request:
  • DNS queries for service-b.example.com from the public Internet will be resolved by the public Hosted Zone, which will result in a public range IP for the “internet-facing” ALB of service B, e.g. 54.86.100.255
  • DNS queries for service-b.example.com from within the VPC, for example from the Service A servers, will be resolved by the private Hosted Zone, which will result in a private range IP for the “internal” ALB of service B, e.g. 10.4.10.4.
• Since service A is only accessible within VPC, DNS resolution for service-a.example.com is not possible from the Internet. Service B however can resolve service-a.example.com correctly since both services resides within the same VPC, associated with the same private Hosted Zone.

Additionally, we have added both “internal” and “internet-facing” ALBs for services which are meant to be accessible via both within VPC and through the Internet. Services which are accessible within a VPC only have an “internal” ALB. This ensures DNS queries for ALBs resolve to private and public IP range addresses respectively, which ensures that traffic is now routed more efficiently within the VPC.

Plenty of benefits

As a result of our new dual-home DNS architecture, we have achieved the following benefits:

• Reduced network latency
  • We have reduced the number of network hops and hence latency for service-to-service calls.
• Improved security
  • Traffic between microservices no longer leaves our VPC.
  • Stricter firewall ingress and egress rules can now be implemented for service-to-service traffic, as traffic is no longer routed through our NATs or proxies.
  • Private IP addresses are no longer disclosed in public DNS zones.
  • We don’t need to trust self-signed certificates for internal HTTPS traffic. We can easily verify domain ownership and use real SSL certificates without exposing private IPs.
• Easier and cheaper
  • All required infrastructure, SSL certificates, etc are available through AWS services at operational costs well below traditional equivalents.
  • We have reduced the effort involved in creating, installing, and renewing self-signed certificates.

Some things to be aware of

DNS resolution when you have overlapping domain names

Suppose that you have a DNS query for the following hostname: accounting.example.com. The following Hosted Zones would be searched:

• The accounting.example.com zone would be searched for a “root” record of the requested type.
• The example.com zone would be searched for a host named accounting.

All DNS resolution within a VPC uses Route 53 DNS servers by default. These servers attempt to resolve DNS queries by looking for a matching private AWS Hosted Zone, if such a zone exists, and then failing that, the query will be resolved using public DNS infrastructure including resolutions for public AWS Hosted Zones. If there is a matching private Hosted Zone but there is no record that matches the host name and type in the DNS query, Route 53 will simply respond with an NXDOMAIN (non-existent domain) response. It will never resolve the request from a public Hosted Zone and never forward the request to a public DNS resolver.

This means that if you have a private Hosted Zone for either accounting.example.com or example.com, DNS queries originating in your VPC will only respond to host names defined in the private Hosted Zone. Unless your host name is explicitly defined in the private Hosted Zone, it will not be resolvable.

Associating a VPC with more than one private Hosted Zone

You can associate a VPC with more than one private Hosted Zone, but the domain names must not overlap. For example, you cannot associate a VPC with hosted zones for both example.com and accounting.example.com because both domain names end with example.com.

DNS resolution for private Hosted Zones from outside the VPC

A potentially negative side effect is that you cannot resolve private zones from outside AWS VPCs. We have found for example that our Engineers or CI/CD automation expect to be able to call internal service APIs. While they can still do so, they can no longer resolve the DNS for these services. There are various strategies to deal with this including improving operational practices to reduce the need for these behaviours, using the Route 53 DNS resolvers over VPN connections, or implementing your own conditional DNS forwarders.

Routing Policies

AWS Route 53 supports various resource record types to dynamically change the resolution of DNS requests. You can continue to use the following resource records for private Hosted Zones:

• Simple - lookup of a single domain name to one or more targets using equal distribution of traffic between services
• Failover - route traffic to a backup service if the primary service is unavailable
• Weighted - send a percentage of traffic to each service to spread load unevenly, e.g. to facilitate A/B testing, different instance types, etc

Private Hosted Zones do not support resource record types that are designed to route between multiple regions:

• Geolocation and geo-proximity based records, as they are intended to send traffic to the AWS region with the closest physical location and typically within the same country.
• Latency-based records, as they are intended to send traffic to the AWS region with the smallest network latency from a client’s location.

Despite the above limitation, AWS does allow multiple VPCs to be attached to a single private Hosted Zone and those VPCs can reside in different regions.