Centralizing endpoint configuration with a private DNS

You have a lot of services, each deployed independently, which all connect to a single database instance. You're about to make a change that's going to change the database instance's connection endpoint, and you don't want to go around changing each service's environment variables, since that would delay the change for several additional minutes.

We're going to use the following AWS services:

• Route 53: A managed DNS service, which lets you create public or private hosted zones.

Configure a Route 53 Private Hosted Zone to register the database endpoint as a DNS entry

Here's the initial setup, and you can deploy it here:

Step by step instructions to set up a Private Hosted Zone in Route 53

Step 0: Test the connection without the DNS record

1. Open the CloudFormation console

2. Select the initial state stack

3. Click the Outputs tab

4. Copy the value for DbEndpoint

5. Open the EC2 console

6. Go to Instances

7. Select your Instance

8. Click the Connect button

9. Click the Session Manager tab

10. Click Connect

11. Run the following command to install psql:

    sudo dnf update & sudo dnf install -y postgresql15

12. Run the following command, replacing [REMOTE_HOST] with the value for DbEndpoint you copied in the previous steps:
    psql -h [REMOTE_HOST] -p 5432 -U SimpleAWS simpleaws

13. You'll be prompted for the password. Enter SimpleAWSDbPassword (or the value you changed it for, if you changed it when deploying the initial state)

14. Verify that it works and you can connect

15. Now run the same command, but now replacing [REMOTE_HOST] with db.simpleaws.dev:

    psql -h db.simpleaws.dev -p 5432 -U SimpleAWS simpleaws

16. Verify that it doesn't work. We haven't configured anything yet, and while the domain simpleaws.dev does exist, there's no database at db.simpleaws.dev

Step 1: Create a Private Hosted Zone in Route 53

1. Open the Route 53 console

2. In the navigation pane on the left, click "Hosted zones"

3. Click "Create hosted zone"

4. For "Domain Name", enter simpleaws.dev (yeah, really, enter that, even though you don't own the domain)

5. For "Type", choose "Private hosted zone"

6. In the "VPCs to associate with the hosted zone" section, select the region where you deployed the initial setup, and then select the initial setup's VPC. The name of the VPC looks like [stack name]-NetworkStack-[some numbers]-VPC.

7. Click "Create hosted zone"

Step 2: Create a DNS record in the Private Hosted Zone

1. Click "Create record"

2. Under "Record name" enter "db", so that the final value will be db.simpleaws.dev

3. For Record type, select CNAME

4. Under Value, enter the endpoint for your RDS instance that you copied in Step 0. If you didn't copy it, here are the steps:

   1. Open the CloudFormation console

   2. Select the initial state stack

   3. Click the Outputs tab

   4. Copy the value for DbEndpoint

5. For "TTL", set it to 60 (that means 60 seconds)

6. Click Create record

Step 3: Test the connection with the DNS record

1. Go back to the browser tab where you're connected through SSH to the instance

2. Run the same command as in Step 0:

   psql -h db.simpleaws.dev -p 5432 -U SimpleAWS simpleaws

3. You'll be prompted for the password. Enter SimpleAWSDbPassword (or the value you changed it for, if you changed it when deploying the initial state)

4. Verify that it works and you can connect

How the Route 53 Private Hosted Zone Works

Step 0: Test the connection without the DNS record

Not that big of a deal for this test (if you trust me), but this issue is based on something I did for a consulting client, and I made the mistake of not testing this and thought I had set up the DNS wrong. I just don't want you to make the same mistake (=

Step 1: Create a Private Hosted Zone in Route 53

Private Hosted Zones are like Public Hosted Zones, but they're Private. Huh, that was a lot of information! It means they can't be accessed publicly. Instead, they're associated with one or more VPCs, and they can only be accessed from those VPCs. These records aren't propagated to any public DNS servers. Private Hosted Zones are still highly available across regions, and they can be associated with VPCs in different regions and even AWS accounts.

Fun fact: You don't need to own the domain you're using! I told you to use simpleaws.dev in the step by step, and you obviously don't own it. Nobody's checking here.

Step 2: Create a DNS record in the Private Hosted Zone

This record works like a regular DNS record. In this case it's of type CNAME because it needs to point at another DNS record: The one for the RDS instance. Pay attention to the TTL, since that's how long you'll have to wait to view the changes if you change the value.

Step 3: Test the connection with the DNS record

Test it again, to verify that it works.

After this step you'd update your app code to depend on the DNS record we just created, and when we want to change the database endpoint we just change the value of the DNS record to point to the new endpoint and wait out the TTL.

Considerations About Route 53

Route 53 Resolver

The service that's actually resolving these queries is called Route 53 Resolver. You can query it at VPC CIDR +2 (that is, if the VPC CIDR is 10.10.0.0/16, the IP address for Route 53 Resolver is 10.10.0.2). Route 53 Resolver checks whether there are any Route 53 Private Hosted Zones associated with the VPC, and attempts to resolve DNS queries using those zones first, before querying the public DNS servers (if possible, if you configured internet access).

Domain Overlap in Route 53 Private Hosted Zones

You can use Route 53 to configure split-view DNS, also known as split-horizon DNS. In split-view DNS, you use the same domain name (simpleaws.dev) for internal uses (db.simpleaws.dev) and external uses, like our public website (www.simpleaws.dev). This opens up the possibility of using the same subdomain name internally and externally, but serve different content or require different authentication for internal and external users.

If you have private and public hosted zones that have overlapping namespaces, like simpleaws.dev and internal.simpleaws.dev, Route 53 Resolver routes traffic based on the most specific match. Here's what happens when we send a query from our EC2 instance (assume our Private Hosted Zone is for subdomain internal.simpleaws.dev):

1. Route 53 Resolver evaluates whether the name of the Private Hosted Zone matches the domain name in the request, like internal.simpleaws.dev. There are two types of matches:

   1. Identical match: The domain name in the request matches exactly the domain name in the hosted zone.

   2. Parent match: The name of the private hosted zone is a parent of the domain name in the request. For example, in a request to subscribers.internal.simpleaws.dev, both internal.simpleaws.dev and simpleaws.dev are a match.

2. If there's no match in Private Hosted Zones, the request is forwarded to public DNS servers.

3. If there's a match for the Private Hosted Zone, Route 53 Resolver searches the hosted zone for a record that matches the domain name and DNS type in the request. If you have two or more private hosted zones that have overlapping namespaces, Resolver routes traffic based on the most specific match.

   1. If a record that matches is found, Resolver returns the value.

   2. If there was a match for the Private Hosted Zone but there's no record in that hosted zone that matches the request, the request is not forwarded to public DNS servers, it just fails and returns NXDOMAIN to the requester.

Best Practices for Route 53 Private Hosted Zones

Operational Excellence

• Use Health Checks for Endpoint Monitoring: Route 53 health checks allow you to monitor the health of your RDS database endpoints. If an endpoint becomes unhealthy, Route 53 can route traffic away from the unhealthy endpoint to a healthy one, improving overall system reliability. Implement health checks by using the Route 53 console, API, or CLI.

• Automate DNS Record Management: Using AWS SDKs or CLI, automate the creation and deletion of DNS entries for your RDS instances. This reduces manual effort, minimizes human error, and ensures a reliable and repeatable process.

Security

• Implement Least Privilege Access Control: Make sure only authorized users and services have the necessary permissions to make changes to the Route 53 private hosted zone. It's effectively working as a configuration (like the old endpoint in env vars was), so an attacker modifying it can be very serious.

Reliability

• Use DNS for failover: You can set up a primary endpoint and a secondary endpoint in the same DNS record, configure it for failover, and Route 53 will respond with the primary endpoint while it's healthy, and if it fails the DNS healthchecks it'll automatically start responding to DNS queries with the secondary endpoint. RDS already does this behind the scenes when it uses failover replicas, but you can do the same for services in different regions for example.

Performance Efficiency

• Use Latency-Based Routing: Not really applicable to RDS, but if you're using private hosted zones to route to a service deployed in several regions, you can add all endpoints to the same DNS record and set up latency-based routing so your code always goes to the endpoint that's closer latency-wise.

Cost Optimization

This time I've got nothing for you here. Route 53 costs $0.50 per hosted zone, nothing we can optimize there, and not even worth thinking about it for that price.