Observability with AWS X-Ray

Observability is the art and science of understanding what the heck is going on in your system. For simple systems it's rather easy: just look at the logs and timestamps. But it gets much more complex when a request can span several components, such as a Lambda function triggering another function, writing to the database, and that write operation triggering yet another Lambda function.

Here's where AWS X-Ray comes in. It's a service that collects data about requests that your application serves, and provides tools that you can use to view, filter, and gain insights into that data. For any traced request to your application, you can see detailed information not only about the request and response, but also about calls that your application makes to downstream AWS resources, services and APIs.

You're probably already storing your application logs in AWS CloudWatch. For a simple app, that's more than enough. However, here's what troubleshooting looks like in a distributed application:

Troubleshooting Without X-Ray

• You add logs to your Lambda functions. Then you add more logs just in case.

• When you have an issue, you open 10 tabs of CloudWatch Logs and try to figure out which log entries are related, based on timestamps and intuition.

• You finally figure it out (days later), and get a detective badge for the investigative work.

Troubleshooting With X-Ray

• You add AWS X-Ray to all your Lambda functions, DynamoDB tables, SNS topics, etc.

• When you have an issue, the X-Ray console gives you an overview of the whole system, and you can zoom in on a specific event and trace it all the way through your app.

• You don't get the detective badge, because figuring this out was actually pretty easy.

Understanding How AWS X-Ray Works

Segments and Subsegments

AWS X-Ray separates the data in segments and subsegments, to let you better understand how your requests are flowing through your application.

A default segment is created automatically when you set up AWS X-Ray. In addition to that, you can create custom segments and subsegments. This way you can separate tracing based on operations or functionalities.

Annotations and Metadata

Annotations are key-value pairs associated with traces that help in indexing and searching through the trace data efficiently. Metadata is data you can add to traces, usually for additional diagnostic information. The key difference is that annotations are indexed can can be used for searching through trace data, while metadata is not indexed and is used to get additional context or diagnostic data once you've already found the traces you need.

Filter Expressions

You can use filter expressions to filter your tracing data based on specific criteria, helping you isolate and focus on specific traces. Filter expressions can use logical operators such as AND, OR, and NOT, helping you search more efficiently.

Service Map

Service maps provide a visual storyboard of your application’s architecture, showing the relationships and data flow between the services involved. They let you have a comprehensive view of your system's operation, visualize how different components work together, and identify areas that can be optimized for better performance.

How to Set Up AWS X-Ray for Lambda

Let's see how to set up AWS X-Ray for a Lambda function that's using Node.js. It's very similar with other languages.

Step 0: Make sure your Lambda function has permissions for X-Ray

Make sure the IAM Role of your Lambda function has the permission AWSXRayDaemonWriteAccess:*, which is necessary for it to publish tracing information.

Step 1: Enable X-Ray in your Lambda function

1. Go to the AWS Lambda console and click on your function

2. Go to the Configuration tab

3. Scroll down to Monitoring tools and click on it

4. Enable the Active tracing option

Step 2: Install the AWS X-Ray SDK for Node.js

Run the following command at the root directory of your application:

npm install aws-xray-sdk

Step 3: Require the AWS X-Ray SDK

Add this at the top of your Lambda function code:

const AWSXRay = require('aws-xray-sdk');

Step 4: Configure X-Ray Segments

You need to call your code as part of an X-Ray Segment, so X-Ray knows that it's part of something that you want to trace.

const segment = new AWSXRay.Segment('name-of-your-function');

AWSXRay.captureAsyncFunc('name-of-your-function', function(callback) {
  // Your code
  callback();
});

If you're using the AWS SDK (for example to write to a DynamoDB table), you need to wrap the require statement for the SDK with the AWSXRay.captureAWS function:

const AWS = AWSXRay.captureAWS(require('aws-sdk'));

If you are using HTTP requests, you can wrap the require statement for the https package with the AWSXRay.captureHTTPsGlobal function:

AWSXRay.captureHTTPsGlobal(require('https'));

Complete example of a Lambda function without and with AWS X-Ray

Let's take this example Lambda function:

import { PutCommand, DynamoDBClient } from "@aws-sdk/client-dynamodb";

const dynamoDBClient = new DynamoDBClient({ region: "us-east-1" });

exports.handler = async (event) => {
  const params = {
    TableName: "BestNewsletters",
    Item: {
      id: { S: "123" },
      name: { S: "SimpleAWS" }
    },
  };

  try {
    const data = await dynamoDBClient.send(new PutCommand(params));
    console.log("Item inserted successfully:", JSON.stringify(data, null, 2));
    return { message: "Item inserted successfully" };
  } catch (error) {
    console.error("Unable to insert item. Error JSON:", JSON.stringify(error, null, 2));
    throw new Error("Unable to insert item");
  }
};

Here's how the same function looks with AWS X-Ray:

import { PutCommand, DynamoDBClient } from "@aws-sdk/client-dynamodb";
import * as AWSXRay from 'aws-xray-sdk-core';

const dynamoDBClient = new DynamoDBClient({ region: "us-east-1" });
AWSXRay.captureAWSv3Client(dynamoDBClient);

exports.handler = async (event) => {
  const segment = new AWSXRay.Segment('BestNewslettersHandler');
  AWSXRay.setSegment(segment);

  const params = {
    TableName: "BestNewsletters",
    Item: {
      id: { S: "123" },
      name: { S: "SimpleAWS" },
    },
  };

  try {
    const data = await dynamoDBClient.send(new PutCommand(params));
    console.log("Item inserted successfully:", JSON.stringify(data, null, 2));
    return { message: "Item inserted successfully" };
  } catch (error) {
    console.error("Unable to insert item. Error JSON:", JSON.stringify(error, null, 2));
    throw new Error("Unable to insert item");
  } finally {
    segment.close();
  }
};

How to set up AWS X-Ray for DynamoDB

To set up X-Ray for a DynamoDB table, follow these steps:

1. In the list of tables, click on the name of the table.

2. On the Table Details page, click the "Actions" dropdown and select "Modify".

3. In the "Advanced settings" section, click the "Edit" button that's next to the "AWS X-Ray tracing" setting.

4. In the "AWS X-Ray Tracing" dialog, select "Yes".

5. Click "Save".

Viewing Traces in AWS X-Ray

You can view tracing data in the X-Ray console in the AWS Management Console. The console provides several visualization tools, such as the service map, trace list, and trace details, that help you identify and diagnose performance issues, bottlenecks, and errors in your app.

Best Practices for AWS X-Ray

To get the most out of AWS X-Ray, follow these tips and best practices:

• Enable X-Ray for all components: It's important to enable it for all relevant components of your system. This includes Lambda functions, DynamoDB tables, and SNS topics.

• Visualize tracing data: Use the console to view a visual representation of your trace data. Use the API to programmatically access and manipulate trace data, and automate tasks such as performance analysis or error reporting.

• Use segments and subsegments to add context: Use segments to represent the overall flow of a request through the system. Use subsegments to represent specific actions within that flow. By adding context to your trace data, you can more easily understand and troubleshoot issues that may be occurring.

• Enable request sampling: Tracing can add overhead to your system, particularly if you are instrumenting a large number of functions or if your functions are being invoked frequently. To reduce this overhead, you can enable sampling in X-Ray. This will cause X-Ray to only trace a subset of requests, which can significantly reduce the overhead of tracing, as well as the costs.

• Use custom attributes: Custom attributes are key-value pairs that you can add to your trace data to provide additional context. Use custom attributes to add metadata about your requests, such as user IDs or request IDs.

• Use the X-Ray SDK to add error handling: The X-Ray SDK provides a captureFunc method that you can use to capture errors and add them to trace data.

Recommended Tool related to X-Ray

• The One Observability Workshop has a section about AWS X-Ray that you'll want to check out.

• The EKS Workshop has a section on using AWS X-Ray for monitoring.

• And if you need to do this outside AWS, Jaeger is an open-source observability tool similar to X-Ray.