Using Client VPN to securely connect to an RDS instance

Let's say you have an Amazon RDS instance that can't be accessed publicly (as it should be). For your dev environment, you've been using a jump host (a public EC2 instance through which you run an SSH tunnel). Now you want to give several customers access to the database, and the jump host solution isn't a good idea (we'll dive into why a couple of sections below).

Instead of that, we're going to use AWS Client VPN: A managed client-based VPN service that enables you to access private AWS resources securely.

How to Set up AWS Client VPN

To get started with this workshop, you'll need to deploy the initial setup. It consists of a VPC with 2 public subnets with NAT Gateways and 2 private subnets, and an RDS instance in one of the private subnets. Here's the CloudFormation Template.

This setup is production ready. If you're using this as a reference instead of just a learning resource, you won't need the initial setup (presumably you already have your own database). If that's your case, I left a few notes for you on each step.

Step 1: Generate server and client certificates and keys using EasyRSA

We'll configure our Client VPN endpoint to use mutual authentication, which means we sign a certificate for the server, we sign a certificate for the client, and any client who wants to connect needs to have that certificate. This step is about creating the certificate authority and signing those certificates.

Running EasyRSA on Windows

1. Open the EasyRSA website, download the zip file and extract it

2. Open a terminal on the location where you extracted the EasyRSA folder.

3. Open the EasyRSA shell
   C:\Program Files\EasyRSA-3.x> .\EasyRSA-Start.bat

Running EasyRSA on Linux or MacOS

1. Clone the OpenVPN easy-rsa repo
   git clone https://github.com/OpenVPN/easy-rsa.git

2. Go to the EasyRSA folder
   cd easy-rsa/easyrsa3

Generating and uploading the certificates and keys (same for all 3 OSs)

1. Initialize a new PKI environment
   ./easyrsa init-pki

2. Build a new certificate authority
   ./easyrsa build-ca nopass

3. Generate the server certificate and key (replace server with a unique name)
   ./easyrsa build-server-full server nopass

4. Generate the client certificate and key (replace simpleaws with a unique name). Repeat this step for every client if you have many.
   ./easyrsa build-client-full simpleaws.dev nopass

5. Create a separate folder and copy the cert and key
   mkdir ~/simple_aws_certs/
cp pki/ca.crt ~/simple_aws_certs/
cp pki/issued/server.crt ~/simple_aws_certs/

   cp pki/private/server.key ~/simple_aws_certs/

   cp pki/issued/simpleaws.dev.crt ~/simple_aws_certs/

   cp pki/private/simpleaws.dev.key ~/simple_aws_certs/

   cd ~/simple_aws_certs/

6. Upload the server cert and key to ACM. Use the following AWS CLI commands, or do it through the console. Remember to replace server for your value.
   aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crt --region us-east-1

You'll note that we're only uploading the server certificate, and in the next step we're assigning it as both server and client certificate. We do that so our Client VPN endpoint will accept any client certificate signed with the same certificate authority. That way, we can distribute different client certificates to different users, and if one is leaked we can just expire that certificate without having to re-distribute certificates to all other users. This is just a bonus, but it fits our scenario of granting access to several customers.

Step 2: Create an AWS Client VPN endpoint

1. Open the ACM console

2. Find the certificates you uploaded on Step 1, and take note of their ARNs (or leave this open in another tab)

3. Open the VPC console

4. In the navigation pane, click Client VPN Endpoints and then click Create Client VPN endpoint

5. Add a name and description for the Client VPN endpoint

6. For Client IPv4 CIDR, enter 10.1.0.0/16. Note: This range cannot overlap with the CIDR range of the VPC you're connecting to. The initial template created a VPC with range 10.0.0.0/16, but if you're planning on using another VPC, keep this in mind.

7. For Server certificate ARN, choose the ARN of the server certificate you uploaded in Step 1

8. Under Authentication options, enable Use mutual authentication, and for Client certificate ARN choose the ARN of the server (yes, server again, not client) certificate you uploaded in Step 1

9. For DNS server 1 IP address and DNS server 2 IP address, enter in both fields the value 10.0.0.2. Note: If you're connecting to a VPC with a different CIDR range, enter the VPC+2 address (the one ending in .2) for that VPC, for example 10.8.0.2 for a VPC of 10.8.0.0/16

10. Under VPC ID click the dropdown and select your VPC. If you used the initial template, the name starts with simpleaws-

11. Leave the rest as default and click Create Client VPN endpoint

Two important details here:

• Client IPv4 CIDR cannot overlap with the CIDR range of the VPC you're connecting to. I suggested 10.1.0.0/16 because the initial template created a VPC with range 10.0.0.0/16, but if you're planning on using another VPC, pay close attention to this

• If you don't set DNS servers, you will have connectivity to the private IP address of the RDS instance, but you won't have the ability to resolve the DNS name into a private IP address. You could use a public DNS server like Google's 8.8.8.8 and 8.8.4.4, but you'd need to grant your VPN access to the internet (which may be something you want to do, but it's not something we need for this solution). Instead, we're entering the IP address of Route 53 Resolver, the private DNS server of every VPC. It can't resolve any DNS name, but resources inside that VPC with a DNS name automatically register it to their corresponding Route 53 Resolver, so it will resolve the DNS name of our RDS instance.

By the way, in case you were curious, the DNS name of our RDS instance is public, even though we disabled public access and even though it resolves to a private IP address. You can verify this by opening a terminal and pinging the DNS name (without being on the VPN). You'll see the private IP address associated to that DNS name, and (obviously) the ping won't reach it.

Step 3: Associate a target network to the Client VPN endpoint

This step essentially tells AWS which subnets to use for the Client VPN endpoint. The endpoint will get one ENI in each subnet. We're associating two of them for high availability. The Client VPN endpoint itself is highly available, but if we were only using one subnet in one AZ, that's a single point of failure.

1. Open the VPC console and click Client VPN Endpoints on the left

2. Select the Client VPN endpoint that you created in Step 2, click the Target network associations tab and click Associate target network

3. For VPC, select your VPC (starting with simpleaws-)

4. Under Choose a subnet to associate, select one of the public subnets

5. Click Associate target network

6. Repeat the process for the other public subnet

Note: You're going to have to wait around 20 minutes until the Client VPN Endpoint goes from status Associating to status Available. Meanwhile, you can proceed up until Step 8.

Step 4: Add an Authorization Rule to access the VPC

Up until this point, the Client VPN endpoint has an ENI in a subnet, so it has network visibility. However, that doesn't mean anyone connected to the VPN can access it. Here's where we grant those permissions.

1. Open the VPC console and click Client VPN Endpoints on the left

2. Select the Client VPN endpoint that you created in Step 2, click the Authorization rules tab and click Add authorization rule

3. For Destination network to enable access enter 10.0.0.0/16 or the CIDR range of the VPC you're using

4. Click Add authorization rule

With those steps we just gave every user permissions on the whole VPC because we're not really managing VPN users, but if we were using user-based authentication, this is where we'd tighten permissions.

Step 5: Associate a security group to your Client VPN endpoint

In this step we're associating to the Client VPN endpoint a security group created by the initial template. Its permissions are rather loose: Ingress from 0.0.0.0/0 over protocol UDP and port 443 (which are what we configured for our VPN endpoint). The main point of this is that the security group associated with our RDS instance allows ingress from this security group, over protocol TCP and port 5432. This way, the database can only be accessed from the Client VPN endpoint (and anything else you need, obviously).

1. Open the VPC console and click Client VPN Endpoints on the left

2. Select the Client VPN endpoint that you created in Step 2, click Actions and click Modify client VPN endpoint

3. Under Security group IDs, add the security group called ClientVPNEndpointSecurityGroup created by the initial template

4. Click Modify client VPN endpoint

In case you're curious, our connection from our computer to the Client VPN endpoint goes over protocol UDP and reaches port 443, it's terminated there, and a new connection is established from there. The database will receive traffic as if it originated from the Client VPN endpoint, over protocol TCP and to port 5432. You can verify this with VPC Flow Logs.

Note: If you didn't use the initial template, you won't find ClientVPNEndpointSecurityGroup. In that case, create a new security group which accepts traffic from 0.0.0.0/0 over protocol UPD and port 443, and configure your database security group to accept traffic from this new security group over protocol TCP and port 5432.

Step 6: Retrieve the endpoint, username and password for your Postgres database

The initial template creates an RDS instance with a random password, stored in Secrets Manager. In this step you just retrieve these values. It's important to do it now though, because once you're connected to the VPN you won't be able to access the AWS console (since the VPN has no internet connection).

Note: You'll only need this step if you're using the initial template. If you already know your database connection info, skip to Step 7.

1. Open the Secrets Manager console

2. Click on the secret with a name ending in -RDSMasterSecret

3. Click Retrieve secret value

4. Copy the values for host, username and password

Step 7: Download and edit the Client VPN endpoint configuration file

In this step we're downloading the configuration file for the Client VPN endpoint, and modifying it to include the certificates that you created in Step 1. We're adding those certificates because the Mutual authentication configuration that we chose requires our VPN client to send the client certificate and key with its request.

1. Open the VPC console and click Client VPN Endpoints on the left

2. Select the Client VPN endpoint that you created in Step 2 and click Download client configuration

3. Find the client certificate and key you created in step 1. They're the files ~/simple_aws_certs/server.crt and ~/simple_aws_certs/server.key (replacing server with the name you used in Step 1)

4. Open the Client VPN endpoint configuration file using a text editor

5. Add at the end of it the contents of the client certificate and key, like this:
   Original contents of the Client VPN endpoint configuration file

<cert>
Contents of client certificate (.crt) file
</cert>
<key>
Contents of private key (.key) file
</key>

1. Find the line that specifies the Client VPN endpoint DNS name (it should be near the top), and prepend a random string to it. For example:

   Original DNS name: cvpn-endpoint-0102bc4c2eEXAMPLE.prod.clientvpn.us-west-2.amazonaws.com

   Modified DNS name: simpleaws.cvpn-endpoint-0102bc4c2eEXAMPLE.prod.clientvpn.us-west-2.amazonaws.com

2. Save and close the file. Share it with anyone who needs to connect to the VPN

In case you're wondering about that DNS change and why I just hardcoded simpleaws there (it's not related to using the initial template), here's the reason: In the config file there's this instruction called remote-random-hostname that's needed by the Client VPN endpoint, but which isn't parsed correctly by most VPN clients (I honestly have no idea why they can't just fix this). It tells the VPN client to add a random hostname to the DNS name, which most VPN clients don't end up doing. Since the VPN client doesn't do it, we do it manually. I used simpleaws because I'm a bit of a fan of the Simple AWS newsletter (go figure!), but any value is fine, including "asd".

Step 8: Connect to the Client VPN endpoint

Now everything is set up. All we have to do is download a VPN client, such as AWS Client VPN. I know, the name is confusing, but this is a desktop app, not the AWS service. I suggest that client because you just download it, add the file and click a button. I'm sure there are better clients that can handle more complex things, but we don't need anything more than this right now.

1. Download the AWS Client VPN application

2. In the AWS Client VPN app:

   1. Open it

   2. Click File (top left of the screen), Manage Profiles

   3. For Display Name, enter a name, such as Simple AWS

   4. For VPN Configuration File, browse and select the file you edited in Step 7

   5. Click Add Profile

   6. Select the profile you just created and click Connect

Here are more detailed guides for Windows, Linux and macOS.

Note: You will only be able to connect once the Client VPN Endpoint is in status Available. If it's still Associating, just wait a bit longer, it takes around 20 minutes.

Step 9: Connect to the Database over the VPN

Now that we're “in the VPN” (or more accurately, our traffic is being routed across the VPN), we can connect normally to the database, as if it was public.

We'll use the hostname, username and password you retrieved in Step 6. Your connection will go through the VPN, and then to the private IP address of the database.

1. Open your favorite database administration tool, such as DBeaver or PgAdmin

2. Enter the data you retrieved in Step 6

3. Test the connection

Advantages of AWS Client VPN compared to an EC2 jump host

Improved security

Client VPN is a managed service, so AWS takes care of all the underlying infrastructure and security patches. With a jump host, you need to keep it patched manually.

As a rule of thumb, we tend to prefer managed services for this exact reason. They're often more expensive (after all, we're paying AWS to do our jobs for us), but the increase in cost is usually significantly lower than the amount of money we save in engineer hours.

High Availability and Scalability

Client VPN endpoints are highly available by design. They work much better than an auto-scaling group of jump hosts, and are significantly easier to deploy and maintain.

Moreover, AWS Client VPN is built to scale and can handle a large number of connections without you needing to step in. With a jump host, network performance is determined by the instance type and size. Thinking of a way to implement a jump host that scales is a fun exercise, and should help you understand how much effort you're saving.

Easier management and better user experience

Managing SSH keys is a pain in the *. We could have used Session Manager instead, but we'd still be managing an IAM user per person (not a problem for employees, a bit more work per customer). And the user experience isn't ideal: Using an SSH tunnel with Session Manager is posible but not trivial, and while database admin tools usually support this, accessing other resources in the VPC is going to become really cumbersome. In contrast, with Client VPN you just click Connect and do everything normally.

Avoid granting SSH access

If someone can run an SSH tunnel through an instance, they can also SSH directly into that instance and run commands in it. Sure, you can limit this at the OS level (yay, even more work!), but you might always miss something. If it's just you and a couple more devs that you trust, you can surely live with that. But if you're granting access to customers and/or third parties, I'd rather not take that risk, even if I have to pay a bit more.

AWS Client VPN vs. EC2 Jump Host: Performance and Cost

Client VPN has a limit of 10 Mbps per user. That's not a lot! For reference, a t4g.nano instance has a baseline (not bursted) network bandwidth of 32 Mbps, and is priced at $0.0026/hour with Spot instances. So, in that sense EC2 Jump Hosts win both at maximum bandwidth and bandwidth/$.

Client VPN is priced at $0.10/hour per endpoint association + $0.05/hour per connection. In our case with 2 associations we have a base cost of $144/month plus $0.05 per hour per each person that is connected. For 10 people working 40 hours a week with the VPN, that's an extra $80/month, for a total of $224/month. That buys a lot of EC2 instances!

So, AWS Client VPN is significantly more expensive than a jump host. However, for a scenario of 10 people using it 40 hours a month it comes up to $22.40/month per person. I'd argue that's not an unreasonable amount. I bet you're already spending more than that on other tools and licenses. And you can bring it down to $11/month/person if you use only one association (you lose high availability though) and everyone uses it on average 4 hours a day.

Is that worth the reduced admin effort? Probably. Is that worth the peace of mind of not granting your customers or third parties SSH access to an EC2 instance inside your network? Definitely!

Best Practices for AWS Client VPN

Operational Excellence

• Monitor and Review Connection Logs: You can monitor the VPN connections and logs with CloudWatch. You need to enable this on the endpoint (it's off by default).

Security

• Fine-tune Security Groups: Only allow traffic to the database from the sources you want. In this case it should be the security group of the Client VPN endpoint, plus anything else your app needs to work properly (for example from the security group of your EC2 instances).

• Fine-tune Authorization Rules: In this case I allowed everyone access to everything in the VPC. You probably want to restrict this a bit. You can only use IP addresses, but you could restrict it to certain subnets by CIDR range. It sounds redundant if you already have security groups properly set up, but it's defense in depth.

Reliability

• Use More Than 1 Network Association: The Client VPN endpoint itself is highly available, but network associations are done against a specific subnet, which will reside in only one Availability Zone. Using more than one network association, each with a subnet in a different Availability Zone, eliminates this single point of failure and makes the connection highly available. Note that this only matters if the database itself is highly available.

• Consider Redundancy: Consider the scenario where the Client VPN service fails. If this service is critical to your operations, you want another way to connect to the database, even if it's less convenient. A jump host can be a good option for a backup connection. Consider your Recovery Time Objective (i.e. how long you can afford to stay down), and whether it's worth to preemptively set up the jump host and keep it stopped, write and test an IaC template to stand it up in a few minutes, or figure it out when the problem happens.

Performance Efficiency

• Plan CIDR Blocks: When setting up the Client VPN endpoint, choose a CIDR block that supports at least double the IP addresses that you plan to use, so network performance doesn't degrade. Each connection uses one IP address.

• Doing analytics? Consider a read replica: The most common reason I can think of to grant your customers direct access to your database is so they can perform analytics. If that's the case, and if you see regular database operations suffering because of these queries, you might want to create a read replica for them to use.

Cost Optimization

• Automate Non-Peak Hours Shutdown: If the VPN connection isn't needed 24/7 (for example during the weekends), consider automating shutdown during off hours to save costs. You can use AWS Lambda and EventBridge for this. It's not start and stop like with an EC2 instance, but it's still possible to automate.