Detecting and Fixing Vulnerabilities with Amazon Inspector

You launched an EC2 instance and connected through SSH to install some stuff (ideally this is not your production environment). Then your lunch arrived, and you forgot to remove the security group inbound rule that grants access to port 22 from IPs 0.0.0.0/0. That port has been open for a year now.

We're going to use the following AWS services:

• Amazon Inspector: Automated security assessment service that automatically analyzes apps and infrastructure and shows security findings.

• Systems Manager Session Manager: Systems Manager feature that lets you SSH into an EC2 instance in one click, from the browser, without opening port 22 to the world.

Use Amazon Inspector to find the vulnerability

The initial setup is an EC2 instance in a public subnet, with a public IP address, with a security group inbound rule for port 22 from IPs 0.0.0.0/0. Here's a template for that.

Step 1: Add your instances to Systems Manager

Systems Manager keeps an inventory of your EC2 instances, and lets you do several useful things like installing patches and running commands. We're setting it up because it's necessary for Amazon Inspector to analyze the instances, and later for Session Manager (which is not a separate service, but a capability of Systems Manager) to connect to the instances directly.

1. Open the Systems Manager console

2. On the menu on the left click Quick setup

3. Click the Get started orange button

4. Under Host Management, click Create

5. Under Targets, pick the instances where you want the Systems Manager agent installed, or leave it as All instances

6. Click Create

7. On the menu on the left click Fleet manager

8. If you've never used it before, click Get started

9. Click the Account management dropdown and click Configure Default Host Management Configuration

10. Turn on Enable Default Host Management Configuration

11. Click the IAM Role dropdown and select Create AWSSystemsManagerDefaultEC2InstanceManagementRole (recommended)

12. Click Configure

Step 2: Enable Amazon Inspector and review findings

Amazon Inspector continually analyzes EC2 instances, Lambda functions and other resources, looking for vulnerabilities. It has a default list of vulnerabilities to look for, and you can add your own.

1. Open the Amazon Inspector console

2. If you've never used Amazon Inspector:

   1. Click Get Started

   2. Click Activate Amazon Inspector

3. Click Findings on the left menu

4. Click on the finding titled Port 22 is reachable from an Internet Gateway - TCP to view the details

There's a free trial period of 15 days, so this test is going to be free. But if you leave it running, it costs $1.25/month per instance. You're the only one who can decide whether reducing the chance of getting hacked is worth that. The key is to make it an informed and intentional decision.

Step 3: Fix the findings

I set us up for an easy to fix finding: An open port that we forgot to close. Head over to Security Groups and close it! Alternatively, you can have a separate security group for SSH access, that you add to and remove from instances as needed (I do this). In that case, remove the offending security group.

1. While on the Findings page in the Amazon Inspector console, open once again the details of the finding

2. Next to Resource ID, click the instance ID, which looks something like i-06443ef7c99052746. It will open the EC2 console in a new tab

3. Click the Security tab

4. Under Inbound rules, look for the rule that has Port range = 22, Protocol = TCP and Source = 0.0.0.0/0. For that rule, click the value for the column Security groups (right next to "0.0.0.0/0"). It will open another browser tab with the security group

5. Click the Inbound rules tab and click Edit inbound rules

6. Find the rule with Port range = 22 and on the right of that rule click the Delete button

7. Click Save rules

Step 4: Deactivate Amazon Inspector

Like I said, $1.25/month per instance. Not a lot, but it can catch you off guard, especially if you're running this in a sandbox account that gets who knows what stuff deployed into it.

1. In the Amazon Inspector console, on the menu on the left, click on General settings

2. Click Deactivate Inspector

3. Type "deactivate" and click Deactivate Inspector

By the way, I didn't include "Wait until Inspector removes the finding" as a step in this guide because this is particular finding comes from a network reachability scan, and Inspector performs those every 24 hours. You can wait 24 hours if you want. If you're doing this for your own workloads instead of our single instance example, you definitely should wait 24 hours and check that findings were actually fixed.

Step 5: Connect to your instance using Session Manager

Session Manager is a way to securely SSH into EC2 instances without opening port 22 to the world. And it's way easier: Just click Connect twice (different buttons, same text) and you're in. I do miss MobaXTerm, but I definitely don't miss the security risks of having SSH keys, and the annoying process of rotating them.

1. Open the EC2 console

2. Go to Instances

3. Select your Instance

4. Click the Connect button

5. Click the Session Manager tab

6. Click Connect

Understanding Amazon Inspector

You might think "Why do I need to pay $1.25/month per instance, if I can just review my security groups once in a while?". I'll answer it with another question: When was the last time you reviewed your security groups?

In this case we viewed and fixed the finding manually, but you can create automated response to Amazon Inspector findings with EventBridge, such as notifying you through SNS or executing a Lambda function to fix it.

Keep in mind Amazon Inspector isn't the only security product out there! We've explored a bit of GuardDuty in the 7 Must-Do Security Best Practices for your AWS Account article, which will detect odd stuff in your CloudTrail logs (logs for all actions taken in the AWS account).

Best Practices for Amazon Inspector

Operational Excellence

• Automatically remediate common vulnerabilities: I picked this scenario because it's happened to me several times, and I'm sure it has happened to you as well. It's not that hard to write a Lambda function or a Systems Manager Automation that auto-remediates this, and other common vulnerabilities or things that we're prone to accidentally leaving misconfigured.

Security

• Least Privilege Access: We shouldn't have set up 0.0.0.0/0 as the port range in the first place, because our requirement is not to allow everyone to SSH into the instance, but to allow just ourselves to access it. We should have used our own IP address instead. Still, Sessions Manager is even better, and easier.

• Regular Patching: Amazon Inspector checks for unpatched software, and you can use Systems Manager Patch Manager to automate the patching process.

• Continuous Monitoring with CloudTrail and GuardDuty: CloudTrail logs all actions against the AWS API, from anywhere (CLI, web console, SDKs, etc). GuardDuty scans those logs for anomalies. This is important because not every vulnerability found by Inspector is going to be a mistake you made, some may be an actual security incident. Go back to the 7 Must-Do Security Best Practices for your AWS Account article for how to configure this.

Reliability

• Automated Backups: Security breaches may involve denying you access to valuable data, e.g. encrypting it or copying it and deleting it. Backups won't prevent the breach, but will minimize the impact. But only if they're properly secured! You don't want to let an attacker encrypt both the data and the backup.

Performance Efficiency

I've got nothing here in this article.

Cost Optimization

• Analyze the costs and the risks and the costs: Understand the costs associated with security measures, the risks of not applying security measures, and the costs associated with those risks. Acknowledge that no system is ever 100% secure, and that extreme, military-grade security is really expensive. Also acknowledge that basic security is really cheap and easy nowadays (it didn't use to be the case). Figure out what's the tipping point for you. And consider asking a consultant.

Recommended Tools and Resources

That means it does something similar to Inspector, but from the CLI and for multiple clouds. And it's open source!

I literally asked the creator how to use it to secure my stuff, and he told me to:

• Run it, view the findings and start fixing them. Not everything is critical. Frameworks are just for reference, to understand who says something is a vulnerability.

• If you're interested in a security audit for a certification, it's a good starting point but you should consult with a security pro