AWS Organizations Deep Dive part 2

When you start with AWS you usually create a single account, get familiar with some services, and deploy a few workloads. Then your usage grows and everything becomes a lot more difficult. AWS Organizations makes those problems a bit easier to deal with.

In this series we're going to talk about AWS Organizations, with a focus on how to implement a secure and sane multi-account strategy (sane as in it doesn't drive you mad). This article is part 2 of the series, where cover best practices and suggestions. Part 1 can be read here, which covers the key features of Organizations.

These best practices are based on the Organizing Your AWS Environment Using Multiple Accounts whitepaper by AWS, and on my own experience as a cloud architect, consultant, and developer of a landing zone product (not my own, I was an employee).

Multi-Account Structure Best Practices

Designing your overall AWS multi-account structure involves balancing isolation and collaboration, and deciding how to manage provisioning and operations at scale. What's the right balance between central control and team autonomy? How do you structure your accounts to minimize overhead while maximizing security?

First, use AWS accounts as your primary isolation boundaries. Accounts provide hard separation not just for access control, but also for fault isolation and service quotas. It's generally recommended to separate different environments (prod vs. dev) and different projects or teams into different accounts. This way, if one account has issues or hits limits, it doesn't affect others.

When deciding between centralization and decentralization, consider what aspects benefit from economies of scale versus what needs team autonomy. And especially consider the reasons things have for changing and who is responsible for making those changes. Typically, it makes sense to centralize foundational components like security, identity management, and networking. These benefit from consistent implementation across workloads, performed by expert specialists. Meanwhile, application workloads can be decentralized to the teams responsible for them, giving those teams greater agility.

This approach creates some trade-offs you'll need to navigate. Centralization provides consistency and easier compliance but can create bottlenecks if teams must wait for central groups to make every change. Decentralization improves agility but may lead to inconsistent implementations or duplicate efforts. This is the fundamental dichotomy between governance and agility, and finding the right balance between freedom and restrictions is very tricky. The ideal solution is called Platform Engineering, where internal teams work not on serving application teams directly but on developing self-service tools that application teams can leverage. This ensures that guardrails and best practices are always in place, without requiring manual work from internal teams and without creating delays for application teams. Of course, reaching this point requires significant organizational maturity and a considerable investment, but the resulting developer experience really pays off. It's a very long topic though, so let's get back to Organizations and how you'd structure accounts.

For example, you might have a central Identity account that manages authentication for all accounts, a Network account that manages VPCs and transit gateways, and a Security account that aggregates findings from all accounts. But individual application teams would manage their own workload accounts within guardrails established by these central services.

Creating New AWS Accounts

As your account numbers grow, you'll need efficient ways to provision and configure new accounts. You have two main options:

1. Use AWS Organizations APIs like CreateAccount to programmatically create accounts from the management account. Then use AWS CloudFormation StackSets to deploy baseline infrastructure to new accounts automatically. Many organizations build an internal "account vending machine" that handles both the account creation and post-creation configuration.

2. Implement event-driven automation where new account creation triggers Lambda functions to apply your organization's baseline configurations automatically.

For networking across accounts, consider whether you want isolated networks per account or a more connected approach. Many organizations use a hub-and-spoke model with a transit gateway in a central Network account that connects to VPCs in various workload accounts. You can use AWS Resource Access Manager to share network resources like subnets or transit gateway attachments between accounts.

As your account numbers grow, central visibility becomes another important aspect. Implement cross-account monitoring using CloudWatch to aggregate logs and metrics, and set up a centralized logging account where all CloudTrail logs, Config logs, and other operational logs are collected. Use AWS Organizations features like CloudTrail Organization Trail to automatically log all API calls across accounts to a central S3 bucket, so you can access all events from a single place (and so they can't be deleted to hide an attacker's steps!).

Also, please maintain documentation about your account structure, including an internal registry of accounts with metadata like owner, purpose, and OU membership. Establish clear processes for requesting new accounts and reviewing the account inventory regularly to identify unused or misconfigured accounts. This is very tedious, but it helps a lot when dealing with audits or making org-wide changes.

Landing Zones and Control Tower

Setting up a well-designed multi-account environment from scratch involves many complex tasks: creating accounts, establishing organizational units, configuring logging, implementing security controls, and more. AWS provides a few solutions to help you establish this foundation quickly and according to best practices defined by them and by yourself.

AWS Control Tower is a managed service that automates the setup of a secure and opinionated multi-account AWS environment. It's essentially AWS's opinion on how to set up a multi-account landing zone with the right guardrails and structure in place. When you implement Control Tower, it automatically creates a baseline environment with:

1. An organization in AWS Organizations with well-structured OUs

2. A Log Archive account for centralized logging

3. An Audit account for security tooling

4. Pre-configured guardrails (which are combinations of SCPs and AWS Config rules)

5. A dashboard to manage and monitor your environment

Under the hood, Control Tower is orchestrating many AWS services on your behalf. It's creating and configuring the organization structure in AWS Organizations, setting up roles for cross-account access, enabling CloudTrail across accounts, establishing S3 buckets for log storage, and implementing AWS Config rules to validate compliance.

Control Tower provides an Account Factory that lets you provision new accounts with a few clicks or via API. These accounts come pre-configured with standard baselines: CloudTrail is enabled, AWS Config is running, the required IAM roles are created, and security guardrails are applied. This standardization ensures all accounts meet your security and operational requirements from day one.

The guardrails in Control Tower come in two types:

1. Preventative guardrails that use SCPs to stop actions before they occur, like preventing the creation of S3 buckets with public access or disabling regions you don't want to use.

2. Detective guardrails that use AWS Config rules to check for compliance and flag violations, like detecting unencrypted EBS volumes or resources missing required tags.

Control Tower isn't just for new AWS environments. It can be enabled in existing organizations, though this requires careful planning to avoid disrupting current operations. The service maintains a concept of "drift": when resources under Control Tower management are changed outside of Control Tower's processes, it can detect and help you remediate this drift.

If you already have an Organization set up without Control Tower, it's typically worth the effort of enabling Control Tower if you're expanding and creating new accounts, or suddenly face some regulation or compliance requirements. You should create an OU with accounts that are not managed by Control Tower yet, move all your accounts there, apply Control Tower to your Organization except for that OU, and then slowly move the accounts back to their original OUs (now managed by Control Tower). By doing it one account at a time you can do all the necessary prep work for each account and resolve any problems causing minimum disruption to your activities. Examples of disruption can be an AWS Config rule killing your EC2 instances because they lack a certain Tag, which is easy to fix by setting up the tag in your Auto Scaling Group configuration, but is not something you want to happen in production or to all your accounts at the same time.

If Control Tower is too prescriptive for your needs, you might consider a custom landing zone. A landing zone is essentially the foundational structure for your AWS environment, including the account structure, network design, security controls, and access management. You can build this yourself using AWS services like Organizations, CloudFormation StackSets, and custom automation.

The AWS Landing Zone Accelerator is an open-source solution built on CDK that helps you deploy a customizable landing zone aligned with AWS best practices. It has more flexibility than Control Tower but requires more effort to implement and maintain.

When deciding between Control Tower and a custom landing zone, consider factors like speed of deployment, customization needs, operational complexity, and your existing environment. Control Tower offers faster setup but less flexibility, while custom approaches give you complete control but require more engineering effort.

For most organizations just starting their multi-account journey, Control Tower provides the simplest path to a well-architected foundation. You can always extend it with customizations as your needs evolve, and the Control Tower team is always working on making it more flexible.

Account Segmentation: Purpose-Driven Account Types

AWS's multi-account best practices include creating several purpose-driven accounts to support governance and infrastructure needs in addition to your application workload accounts. But why is this segmentation so important, and how does it enhance your security posture?

The most fundamental account is the Management Account (also called the payer account). This is the first account that hosts your AWS Organization and serves as the billing center for all other accounts. Because of its privileged position, you should strictly limit what runs in this account – ideally, it should be used only for managing AWS Organizations, SCPs, and consolidated billing. Avoid deploying actual workloads here to reduce security exposure (remember that SCPs don't affect this account).

A Log Archive Account provides centralized, immutable storage for logs and audit data from all other accounts. CloudTrail logs, Config snapshots, and other operational logs should be collected here. Under the hood, this account typically contains specially configured S3 buckets with strict access controls and retention policies. Apply strong safeguards like SCPs that deny deletion of any S3 objects here, to enforce that logs are append-only.

A Security Audit Account (sometimes called Security Tooling account) is used by your security team to deploy and run security tools that need organization-wide visibility. This account typically becomes the delegated administrator for services like Security Hub, GuardDuty, and IAM Access Analyzer. Behind the scenes, it's using cross-account roles to gather security information from other accounts and maintaining a centralized database of findings.

For organizations with complex networking needs, a Shared Network Account centralizes network infrastructure management. This account might host a transit gateway, VPN connections, Direct Connect gateways, and shared VPCs that other accounts can use. By centralizing network management, you ensure consistent configuration of routing, security groups, and network ACLs across your environment. A simple trick to save money if you use a lot of NAT Gateways (e.g. for dev environments) is to place them here and access them through transit gateway or shared subnets.

An Identity Account can be used to manage identity providers and directory services, especially if you're integrating with on-premises identity systems. This account will host AWS IAM Identity Center, plus possibly AWS Managed Microsoft AD or AD Connector to provide federated access across your organization.

Some organizations also create specialized accounts like:

• Infrastructure Tooling Account for operational tools like CI/CD systems

• Shared Services Account for common services that multiple teams use

• Backup Account for centralized management of AWS Backup across accounts

These foundation accounts typically fall into the Security OU (Log Archive, Security Tooling) and Infrastructure OU (Network, Identity, Shared Services) as discussed earlier. Not every organization needs all of these accounts initially, you can start with just the Management, Log Archive, and Security Tooling accounts and add others as you need them.

For application accounts, AWS recommends segmenting by purpose: separate accounts for each production workload (to contain blast radius), separate accounts for development/testing of those workloads, and possibly accounts for sandbox/experimentation. These accounts would typically live in the Workloads OU, often further split between Production and Non-Production sub-OUs.

This account segmentation provides clear separation of duties and stronger security. Your security logs are in an account that application administrators can't modify, ensuring their integrity. Compliance requirements like PCI DSS or HIPAA can be addressed by isolating sensitive workloads in dedicated accounts with stricter controls. And if a particular service requires specialized oversight, it can get its own dedicated account.

When implementing account segmentation, use a clear naming convention to make the purpose of each account immediately evident. For example, prefix account names with their purpose: "Security-Audit", "Security-LogArchive", "Infra-Network", "Workload-AppX-Prod", "Workload-AppX-Dev", etc.

Note that migrating a workload to another account is pretty complex, to the point that I'm referring to it as migrating and not just moving. It's worth to spend a few minutes or even hours considering the right account structure before deploying stuff, it will save you weeks down the line.

Monitoring and Observability in Multi-Account Environments

When you have dozens or hundreds of AWS accounts, maintaining visibility across all of them becomes a significant challenge. How do you ensure that you can detect issues, monitor performance, and maintain security across your entire AWS footprint? It's easy, you just manually log into each AWS account and… nah, I'm just kidding.

The key is implementing a comprehensive monitoring and observability strategy that aggregates data from all accounts while maintaining appropriate access controls. This strategy should cover operational metrics, application performance, security events, and compliance status.

AWS offers several services designed for cross-account monitoring. CloudWatch cross-account observability allows you to view and analyze metrics, logs, and traces from multiple accounts in a single account. This feature uses a source-destination model: source accounts share their monitoring data with a monitoring account, which can then create dashboards and alarms across the entire organization.

Behind the scenes, this sharing is implemented through resource-based policies that permit the monitoring account to access CloudWatch data from source accounts. You can set up this sharing manually, but AWS Organizations makes it easier by allowing you to designate a member account as a monitoring account and automatically enabling sharing for all accounts in specified OUs.

For security monitoring, AWS Security Hub provides a comprehensive view of your security posture across accounts. It collects security findings from services like GuardDuty, Inspector, Macie, and IAM Access Analyzer and aggregates them in a central dashboard. When configured as the delegated administrator, your Security account can view and manage Security Hub for all accounts in your organization.

AWS Config Aggregator performs a similar function for configuration compliance. It allows you to view the configuration of resources across all accounts in your organization, making it easier to identify misconfigurations or policy violations. You can deploy organization-wide AWS Config rules that check for compliance with your security and operational standards.

For log aggregation, a central logging strategy is essential. This typically involves:

1. Setting up an organization-wide CloudTrail that records API activity from all accounts to a central S3 bucket in your Log Archive account

2. Configuring VPC Flow Logs from all accounts to be sent to a central location

3. Aggregating application and system logs via CloudWatch Logs with cross-account subscriptions or third-party logging solutions

When implementing this centralized logging, consider the volume of data. Large organizations may generate terabytes of logs daily. Set up appropriate retention policies and consider using tools like Amazon Athena, OpenSearch, or third-party SIEM solutions to make this data searchable and actionable.

One challenge in a multi-account environment is determining who should receive alerts. In a single account this is straightforward, but with many accounts, you need to route alerts to the right teams. This is where proper account metadata becomes important. Maintain a registry of account owners and responsibilities, and use this information to configure alert routing. Keep in mind that alerting everyone of everything quickly leads to alert fatigue.

CloudWatch Metrics Insights and CloudWatch Logs Insights allow you to query across multiple accounts, making it easier to identify patterns or issues that span your organization. These tools can help you answer questions like "Which applications are experiencing the highest error rates?" or "Where is most of my network traffic coming from?" Moreover, nowadays they're powered by AI, so you can almost literally ask those questions.

Security and Compliance in Multi-Account Environments

While multi-account setups enhance security isolation by default, they also introduce complexity in ensuring consistent security across your organization. I mean, sure, I've been babbling about a Security account and share security stuff, but you have to admit that would be easier to implement for a single account than for a dozen accounts with different security needs. Still, it can be done, and the results are often much better (the key being the different security needs).

Start with centralized logging and monitoring. Ensure every account sends logs to a central, secure location – typically the Log Archive account. This includes CloudTrail logs for API calls, AWS Config snapshots, VPC Flow Logs, and CloudWatch Logs for critical systems. Enable a CloudTrail organization trail that automatically logs events from all accounts in your organization, so you're not relying on each account to set up CloudTrail individually.

Behind the scenes, this organization trail creates a resource-based policy on the destination S3 bucket that allows each member account's CloudTrail service to write to it. This ensures that even if a particular account is compromised, an attacker can't easily cover their tracks by disabling logging or deleting logs.

Implement detective controls that work across your organization. Use AWS Security Hub and Amazon GuardDuty in all accounts, designating your Security account as the aggregator for findings. This gives your security team centralized visibility into potential issues across your entire cloud footprint. Similarly, deploy AWS Config rules or conformance packs to check each account against security best practices, like ensuring encryption is enabled or detecting public S3 buckets.

For identity and access management, use AWS IAM Identity Center for centralized human access as discussed earlier. This gives you a single source of truth for users and makes it easy to revoke access when someone leaves or changes roles. For the root user in each account, implement the safeguards mentioned earlier – ideally removing root credentials from member accounts entirely and implementing an SCP that denies all actions performed by the root user.

Network security requires special attention in a multi-account environment. If accounts are connected via VPC peering or a transit gateway, ensure you maintain appropriate segmentation with security groups, NACLs, and possibly AWS Network Firewall at central chokepoints. One account shouldn't be able to freely access another's internal resources unless explicitly intended. Use AWS Firewall Manager to deploy consistent security group rules or WAF configurations across accounts.

Apply the principle of least privilege not just within accounts but across accounts. The team managing security tooling should have only audit access to application accounts, not full administrative privileges. Application developers shouldn't have access to modify central logging or security accounts. By using accounts as separation boundaries, you physically enforce this segregation – developers simply have no credentials in the Log Archive account.

For incident response, plan for cross-account scenarios. If one account is compromised, how will you investigate and contain the issue? Ensure you have cross-account CloudTrail enabled so you have logs even if an attacker tries to clear local evidence. Prepare an SCP "kill switch" that can quickly quarantine a compromised account by denying all or most API actions. You can even create a Quarantine OU for this purpose.

Encryption and data security should be consistent across accounts. Use AWS Config to verify that all accounts meet your encryption standards. For example, checking that S3 buckets have default encryption enabled and EBS volumes are encrypted. For KMS keys that need to be used across accounts, consider centralizing key management in a security account or implementing consistent key policies across accounts.

For continuous compliance, treat your multi-account environment as a constantly audited domain. Use AWS Audit Manager or third-party compliance tools to automatically assess all accounts against your required controls. Use AWS Organizations Tag Policies to ensure resources are consistently tagged across accounts, which helps with both cost allocation and security attribution.

Finally, establish clear governance processes around account management. Require approval for creating new accounts and ensure they fit in your OU structure with all baseline controls applied. Periodically review your account inventory to identify accounts that are no longer needed or are misplaced in the wrong OU.

Conclusion and Next Steps

So, lots of best practices. The question now is, should you do all of them? And the answer isn't “it depends”. It's “Yes… eventually”. Here's some general guidance on when you should do each of these things:

• Creating an Organization: Right away.

• Creating all the Accounts I mentioned: Why not? They're free

• IAM Identity Center, basic SCPs, CloudTrail Org Trail: Yes, all of those

• Shared resources like CI/CD: When you need them, build them as shared

• Networking: This is somewhat more complex and you don't need it right away

• All that other security stuff: Not before you have at least one part-time security person

• Monitoring and observability: Do logs right away, build the rest progressively as you need it

The best moment to do all of this is right before you need it, but that's often hard to get exactly right. So, do the basics (they're pretty easy to implement) and grow the rest progressively. Just don't forget about it. The absolute last thing you ever want to do is migrate a workload to another AWS account, so plan accordingly and implement everything properly before you reach that point.