Monitoring and Securing Serverless Endpoints with API Gateway

Amazon API Gateway is a fully managed service that lets you publish APIs. It supports REST, HTTP, WebSocket, and gRPC protocols.

Picture this scenario: You have serverless API endpoints, probably exposed through Lambda URLs or a basic API Gateway configuration. Things are going well, and you want to monitor and secure those endpoints in a scalable, low maintenance and cost-effective way.

How to set up Amazon API Gateway

1. Create an API Gateway API: Go to the API Gateway console and click on "Create API". Choose a name and a protocol (e.g. HTTPS), and select "Lambda Function" as the integration type.

2. Create a new resource: Resources are the path segments in the URL for your API. For example, if you want to create an API for managing users, you might create a resource called "/users". By the way, you should properly plan your API endpoints!

3. Create a new method for your resource: A method is an HTTP verb (e.g. GET, POST, DELETE) that specifies the type of action you want to perform on the resource. For example, if you want to retrieve a list of users, you would create a GET method.

4. Configure the integration for your method: This is where you specify the backend service that will process the request. In this case, you need to choose "Lambda Function" as the integration type and choose the appropriate Lambda function from the list.

5. Deploy the API to dev: Choose a stage for your API (e.g. "dev") and click deploy.

6. Test your API: Send a request (e.g. with Postman) to the API endpoint and verify that it returns the expected response. Try sending different types of requests (e.g. GET, POST, DELETE) and see if they are processed correctly by your Lambda function. You can find the API endpoint in the API Gateway console.

7. Set up IAM permissions: You need to set up your Lambda function to only be invoked from API Gateway. Go to the IAM console and create a new IAM policy with the permissions lambda:InvokeFunction and apigateway:*. Then, attach this policy to the IAM role that your Lambda function is using. Keep in mind, the Function URL that's already set up (if any) will stop working.

8. Deploy the API to prod: Once again, choose a stage for your API ("prod" in this case) and click deploy.

9. Test it again: Make sure it's working!

How to set up Amazon CloudWatch for API Gateway

1. Metrics are automatically set up, you can view them by following this guide.

2. To set up logs, go to your API's Settings and scroll down to the "CloudWatch Settings" section.

3. Select "Enable CloudWatch Logs".

4. In the "Log Level" dropdown menu, pick the log level that you want to use (e.g. "ERROR", "INFO").

5. In the "Log Group" field, specify the name of the CloudWatch Logs log group that you want to use. If the log group doesn't exist, it will be created automatically.

6. Click "Save Changes".

How to set up AWS Web Application Firewall (WAF) for API Gateway

1. Create a WAF security group: You can do this using the WAF console, the AWS CLI, or a CloudFormation template. Choose a name for your security group and specify the IP address ranges that you want to allow or block (if any).

2. Add a WAF policy to your API Gateway API: Choose a name for your policy, and specify the WAF security group that you created in step 1.

3. Test your WAF policy: You can do this using any tool that can send HTTP requests to your API Gateway API, such as Postman or cURL. Try sending different types of requests (e.g. GET, POST, DELETE) and see if they are allowed or blocked by your policy.

4. Configure your WAF rules: Choose a name and a type for your rule (e.g. SQL injection), and specify the conditions that you want to match (e.g. specific patterns in the request headers or body). You can also specify the actions that you want WAF to take when a request matches the rule (e.g. allow, block, count).

5. Update your WAF policy: Add the WAF rules that you created in step 4 to your policy.

6. Test your WAF rules: You can do this using the same tool that you used in step 4. Try sending requests that should match your rules (e.g. requests with malicious payloads), and see if they are allowed or blocked by your policy.

7. Monitor and troubleshoot your WAF policy: You can do this using Amazon CloudWatch. CloudWatch logs all requests that are allowed or blocked by your policy, and you can use this information to detect and fix issues. You can also set up CloudWatch alarms to be notified when there are unusual patterns of requests or when there are errors in your backend service.

Optimizing API Gateway in AWS

Note: Some of these are features, which you may not need for your particular use case (e.g. if your API is public you don't need authentication). Just read through the list, and pick the ones you need.

1. Design your APIs: A service is used through APIs. API design is akin to UX design for the service. If you don't design them well, you'll have a lot of problems after that 1 service becomes 5 or 10 (and I'm not even talking about microservices). Even if it's just 1, do your frontend self a favor and design your API.s For RESTful APIs, check the Richardson Maturity Model and Microsoft's guide for API design. If you're using gRPC, go to the source (not the source code!). Same for GraphQL.

2. Choose HTTP, REST or WebSocket: WebSocket is easy to pick or discard: If you need websockets or are using GraphQL, create a WebSocket API. If you're building a RESTful API, you have to pick between HTTP API and REST API. The difference? REST APIs are the real deal, with a lot of features that HTTP APIs don't have, such as API Keys, canary deployments, rate limiting, WAF integration, private endpoints. HTTP APIs are 70% cheaper. Best practice: If you're starting out, you'll be lucky to pay $10/month for API Gateway, so you're fine with REST APIs (price is $3.50 per million requests). Once you have a dozen services and you've figured out what features you actually need (and which you don't), picking the right type will save you a good chunk of money. Pick as early as you can, but it won't break your wallet (or system) if you pick wrong.

3. Set up your own domain: First, you need to own a domain, which you can purchase from Route 53 or (usually cheaper) from other registrars like CloudFlare or GoDaddy. If you purchase it from somewhere other than Route 53, you can still use Route 53 as your DNS server (optional). Next, create a TLS certificate in Certificate Manager. Then, set up your domain on API Gateway. Finally, set up your DNS to route traffic to your API.

4. Use CloudWatch to monitor and troubleshoot APIs: You can view API metrics, check the logs generated by APIs, and set up alarms to be notified when something goes wrong. Here's how to monitor REST, HTTP and WebSocket APIs.

5. Enable authentication: You can use IAM for other AWS services, Cognito for external users, or Lambda authorizers with a third-party service (e.g. Auth0, Okta) to protect APIs from unauthorized access.

6. Enable request validation: Request Validation allows API Gateway to check the request payload and reject requests that don't match your schema. This way, your backend doesn't even get called (you save $$) if the request is malformed.

7. Enable caching: If your Lambda function is idempotent (meaning it always returns the same output for the same input), you can enable caching. This reduces the number of calls to the backend service (saves $$) and decreases response time. You can also encrypt the cache.

8. Enable WAF: I already mentioned this above, but it's worth expanding a bit. First, set up WAF with the baseline rule group and whatever use case specific rule groups apply to your API. After that, check out the other managed rule groups, consider whether you need more rules, check the marketplace for other relevant rule groups, and consider writing your own. Check out the price, and consider whether it's worth it for you.

9. Set up CORS: If your API is going to be accessed from a different domain, you'll need to set up CORS.

10. Set up API keys: You can require that requests to your API include an API key. This is different than authentication because API keys don't expire (unless you set them to) and aren't linked to a user. Basically API keys are designed so that other developers use your API. You can also set usage limits per key.

11. Use API Gateway for private APIs: You can put API Gateway in front of your private APIs as well. What for? Well, things like CORS are useless for private APIs, but you can certainly make use of monitoring, logs, request validation or caching. And there's even a use case for authentication, if you're implementing a zero trust security model.

12. Use API Gateway in a VPC: If you're talking to other AWS resources from inside a VPC, you don't need an internet connection. You can set up a VPC Endpoint using PrivateLink. And you can do that for API Gateway APIs as well. You can even set endpoint policies.

13. Trace requests: Remember AWS X-Ray? You can also set it up for API Gateway.

14. Mock responses: If you just need a mock response, you can have API Gateway generate it directly. You can change that to a proper response later.

15. Transform request and response data: If you need to interconnect two services that send/receive data in different formats, you can set up API Gateway to transform the data in the request and/or the response.

16. You can also use OpenAPI: Here's how to import an OpenAPI spec into API Gateway.

17. Compress payloads: You can use deflate, gzip or identity content encodings to compress payloads, and API Gateway decompresses them.

18. Sell your APIs on AWS Marketplace: AWS Marketplace is a place where you can buy solutions and deploy them directly to your AWS accounts. You can also sell your solutions, including selling access to your APIs.

Recommended Tools and Resources

Check out this list of API Gateway tutorials.

If you want to see API Gateway in action, check out the serverless web app workshop.

If you want to secure your APIs, you're looking for the serverless security workshop.

I said that you should design your APIs. Swagger lets you easily document those designs.

If getting AWS Certified is among your new year's resolutions, let me recommend Adrian Cantrill's courses. With their mix of theory and practice, they're the best I've seen. I've literally bought them all (haven't watched them all yet).