Detecting Failed Sign In Attempts to AWS and Alerting

Imagine this scenario: You're careful with security, and you set up Multi-Factor Authentication for your AWS IAM or IAM Identity Center user. At one point, a malicious agent of evilness figures out your password, either via phishing, keylogging, or any other technique. The only thing standing between them and $50k in bitcoin mined on your AWS account (and the $500k AWS bill paid with your credit card) is your MFA device. They don't have access it (yet!), so you're safe (for now!).

The correct response is obvious: Change your password, so they'll be back to square one. The problem with this is that most of the times you don't know that a password has become known until someone uses it. This is the reason why we should rotate passwords regularly!

Let me propose an extra layer of security: Get notified every time a login attempt fails because of a failed MFA check.

Understanding CloudTrail event logs

AWS CloudTrail is a service that logs all requests performed against AWS in your account. This includes all actions and requests (even unauthorized ones) done through the Console, CLI, AWS SDKs, and APIs.

By default (already enabled when you create your account) and for free, CloudTrail offers you a viewable, searchable, downloadable, and immutable record of all events that happened in the past 90 days, called Event History.

You can also create Trails, which let you export to S3 or CloudWatch Logs all or a subset of CloudTrail events. This way, you can use other AWS services like Athena and OpenSearch, or external tools like ElasticSearch, to analyze CloudTrail events. Event history is per region, while Trails can be created for all regions in an account, and even for all accounts in an Organization.

Events are JSON objects, which contain all the information of what was attempted, when, by whom, some other details on the request, what was the result, and sometimes some other details on the response, such as the reason.

For example, this is what a ConsoleLogin event that fails looks like (I hid a few details replacing them with HIDDEN):

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "HIDDEN",
        "accountId": "HIDDEN",
        "accessKeyId": "HIDDEN",
        "userName": "HIDDEN"
    },
    "eventTime": "2023-08-24T21:07:08Z",
    "eventSource": "signin.amazonaws.com",
    "eventName": "ConsoleLogin",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "HIDDEN",
    "userAgent": "HIDDEN",
    "errorMessage": "Failed authentication",
    "requestParameters": null,
    "responseElements": {
        "ConsoleLogin": "Failure"
    },
    "additionalEventData": {
        "LoginTo": "https://console.aws.amazon.com/console/home?HIDDEN",
        "MobileVersion": "No",
        "MFAUsed": "Yes"
    },
    "eventID": "HIDDEN",
    "readOnly": false,
    "eventType": "AwsConsoleSignIn",
    "managementEvent": true,
    "recipientAccountId": "HIDDEN",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "signin.aws.amazon.com"
    }
}

How are IAM and IAM Identity Center events logged in CloudTrail?

Naturally, CloudTrail logs events when an IAM or IAM IC user logs in to AWS. However, it's not just one event. Let me show you what happens behind the scenes with IAM IC and IAM.

CloudTrail Events for IAM Identity Center users logging in

These are the events that CloudTrail logs when an IAM or IAM IC user logs in:

• CredentialChallenge: AWS requested some form of credential, such as password or MFA device. Each of these is followed by UserAuthentication and one CredentialVerification event, and this sequence of three is repeated until either all necessary credentials are provided, or CredentialVerification fails.

• UserAuthentication: AWS receives the requested credentials.

• CredentialVerification: AWS checks whether the received credentials are valid. If they are, this event contains: "serviceEventDetails":{ "CredentialChallenge":"Success" }, and the process continues either by requesting the next credential or by authenticating. If the credentials received are invalid, this event contains "serviceEventDetails":{ "CredentialChallenge":"Failure" } and the process stops. The type of credential requested can be found in "additionalEventData":{ "CredentialType":"PASSWORD" }, the value of which can be PASSWORD for the regular password, TOTP for MFA devices that produce temporary codes, WEBAUTHN for web apps using WebAuthn, EXTERNAL_IDP for external identity providers, or RESYNC_TOTP to re-synchronize TOTP devices.

• Authenticate: Once CredentialVerification succeeds and no more credentials are required, this event is logged. This means the user successfully authenticated to IAM Identity Center. If you're authenticating with an external Identity Provider such as Google Workspaces, Microsoft AD or Okta, this is the only event you'll see.

• Federate: This event means the IAM IC user assumed an IAM role in an AWS account.

• ConsoleLogin: This event means the IAM IC user logged in to the AWS Console using the assumed IAM role.

The first four events will be logged on the AWS account and region where IAM IC is configured. The last two are logged in the AWS account where the user signs in, in the default region for that user (i.e. the one that's selected when the user signs in).

CloudTrail Events for IAM users logging in

IAM is much simpler. The only event is ConsoleLogin, and this is how you can identify what happened:

• If the user signed in successfully, the event contains "responseElements": { "ConsoleLogin": "Success" }

• If sign in failed, the event contains "responseElements": { "ConsoleLogin": "Failure" } and can contain "errorMessage": "Failed authentication"

• If the user used MFA (regardless of success or failure), the event contains "additionalEventData": { "MFAUsed": "Yes" }

• If the user is the root user, the event contains "userIdentity": { "type": "Root" }

All of these events are always logged in the us-east-1 region.

Identifying when signing in fails due to MFA

The last section was a bit detailed, but it contains all the information we need to identify when a login attempt fails the MFA check!

For IAM IC, we need to find CredentialVerification events which contain either "additionalEventData":{ "CredentialType":"TOTP" } or "additionalEventData":{ "CredentialType":"WEBAUTHN" }, and contain "serviceEventDetails":{ "CredentialChallenge":"Failure" }.

For IAM, we need to find ConsoleLogin events which contain "responseElements": { "ConsoleLogin": "Failure" } and "additionalEventData": { "MFAUsed": "Yes" }.

You can do this manually in the CloudTrail event history, but if we want to automate it and get notifications, we need to send it to CloudWatch Logs.

Sending CloudTrail events to CloudWatch Logs

1. Sign in to the AWS Console. For IAM IC, sign in to the root account of the Organization

2. Go to CloudTrail

3. Click Create trail

4. For Trail name, enter management-events

5. If you're configuring this for an organization, check Enable for all accounts in my organization

6. Select Create new S3 bucket and under Trail log bucket and folder enter a name, or leave it as default

7. Under AWS KMS alias, enter a name for a KMS key to encrypt the logs

8. Under CloudWatch Logs, check Enabled

9. Under Log group name, enter a name for the CloudWatch Logs log group

10. Under Role name, enter a name for the IAM Role that'll let CloudTrail put logs to CloudWatch Logs

11. Click Next

12. Leave these options as default and click Next again

13. Click Create trail

14. Wait until the Status column changes to Logging (green)

Configuring Alerts and Notifications for CloudWatch Logs

First, let's create a filter to view the failed logins in CloudWatch Logs. Follow the steps, and on Step 6 choose the right filter depending on whether you're using IAM or IAM IC.

Filtering failed login attempts due to MFA

1. Open the CloudWatch console

2. In the panel on the left, under Logs, click Log groups.

3. Click on the name of the log group that you created for the trail

4. Click the Metrics filters tab, and click the button Create metric filter

5. Under Filter pattern, enter the pattern that corresponds to IAM or IAM IC, depending on which you're using:

   1. For IAM: { ($.eventName = ConsoleLogin) && ($.additionalEventData.MFAUsed = "Yes") && ($.responseElements.ConsoleLogin = "Failure") }

   2. For IAM IC: { ($.eventName = CredentialVerification) && (($.additionalEventData.CredentialType = "TOTP") || ($.additionalEventData.CredentialType = "WEBAUTHN")) && ($.serviceEventDetails.CredentialChallenge = "Failure")}

6. Click Next

7. For Filter name, enter SignInFailedMFA

8. Under Metric namespace, enter CloudTrailMetrics

9. For Metric name, enter SigninMFAFailureCount

10. For Metric value, enter 1.

11. Click Next

12. Click Create metric filter

Configuring alerts for failed login attempts due to MFA

1. On the Metric filters tab, find the metric filter you just created, select it and click Create alarm

2. Under Whenever SigninMFAFailureCount is..., select Greater/Equal

3. Under than…, enter 1

4. Click Next

5. Under Send a notification to the following SNS topic, select Create new topic

6. Under Create a new topic…, enter SignInFailedMFAAlarm

7. Under Email endpoints that will receive the notification…, enter your email address

8. Click the Create topic button that's right below where you just entered your email address

9. Click Next

10. Under Alarm name, enter SignInFailedMFAAlarm

11. Click Next

12. Click Create alarm

13. Open your email inbox, open the email titled AWS Notification - Subscription Confirmation, and click Confirm subscription

Testing Alerts When Signing in to AWS Fails Due to Incorrect MFA Code

To test it, you'll need to attempt to sign in, and enter the incorrect MFA code. There's going to be a delay of a few minutes between when the event actually occurs and when you receive the notification. CloudTrail usually takes an average of 5 minutes to export the event logs to CloudWatch Logs, and the alarm could take a couple more minutes to fire.

You can also monitor the alarm in CloudWatch Alarms:

Recommended Tools and Resources for CloudTrail

If you want to get more practical experience with this, I recommend the AWS Management and Governance Tools Workshop.

CloudTrail has a feature called CloudTrail Insights, which reports on unusual events in your account. It's worth checking out, and probably enabling.