Automating Deployment of Configuration Updates

Suppose you have an app deployed in some EC2 instances, and there's a configuration file that you need to update simultaneously across all instances. First you did this manually in each instance, but it was prone to errors. So you moved the file to an S3 bucket, and you're connecting via SSH into every instance and pulling the new file from S3. You'd like this to happen simultaneously and automatically, with no downtime.

This article is a guide on how to create a configuration profile in AppConfig and set up CodePipeline to update all instances when the file in S3 is updated.

Step 0: Deploy the initial setup

This guide is sort of a workshop, and to follow it properly you'll need to deploy a few initial resources. Here's the initial setup, and you can deploy it here:

Once that is deployed, follow these steps to test the application:

• Open the CloudFormation console

• Select the initial state stack

• Click the Outputs tab

• Copy the value for EC2InstancePublicIp

• Paste it in the browser, append :3000 and hit Enter/Return

• Verify that the message says "to do: write thank you message"

Step 1: Create the AWS AppConfig application

An application in AppConfig is a logical unit that handles a set of capabilities and features for a specific application. This is where you're going to group everything else.

• Open the AWS Systems Manager AppConfig console

• Click Get Started. If you don't see a Get Started button, go to the Applications tab and click Create application

• Enter a name and description for the application, such as SimpleAWS38

• Click Create application

Step 2: Create an environment in AppConfig

An environment is a logical deployment group of AppConfig targets, such as applications in a beta or production environment. Having different environments for a single app lets you keep the actual values separate, while still keeping all the configuration resources grouped together.

• Go back to the AppConfig console

• On the Applications tab, click on the application you just created

• Go to the Environments tab and click Create environment

• Enter a name and description for the environment, such as "Dev"

• Click Create environment

Here we could also configure CloudWatch Alarms, so that deployments made to this environment are automatically rolled back if a CloudWatch Alarm is triggered. This is especially useful to roll back failing configurations.

Step 3: Create an AppConfig freeform configuration profile

A configuration profile is a collection of settings stored in AppConfig, which can be accessed by applications. In this case we're creating a Freeform Configuration, but another option is feature flags, which we'll talk about in a future issue.

• Go back to the AppConfig console

• On the Applications tab click on your application

• On the Configuration profiles and feature flags tab, click Create

• Select Freeform Configuration and click Select

• Enter a name and description

• Under Configuration source, select AWS CodePipeline

• Click Next

• Under Add validators, click the Remove button for Validator 1

• Click Create configuration profile

In these instructions I told you to remove the validator, because our config file doesn't have a specific format. If you're dealing with properly formatted configuration files, such as a JSON, then a validator will prevent a deployment from even starting, in the case where the config file doesn't match the schema defined for the validator.

Step 4: Create a CodePipeline pipeline

CodePipeline is AWS's service for managing CI/CD pipelines. In this case, however, we're not dealing with a regular CI/CD pipeline where the source is a git repository and the goal is to deploy that code. Our source is an S3 bucket, and CodePipeline is effectively serving us to execute an action when a file is uploaded to that bucket. There's no Build step, and our Deploy step is sending that configuration file to AppConfig. We're not just moving the file around though, we're effectively Deploying the file, in a new Deployment in AppConfig, for that specific Application, for that specific Environment.

• Go to the CodePipeline console

• Click Create pipeline

• Enter a name for the pipeline

• Click Next

• For Source provider, select Amazon S3

• In Bucket, select the name of your S3 configs bucket (which you set when launching the initial template)

• In S3 object key, enter "review.txt.zip"

• Click Next

• Click Skip build stage and accept the warning by clicking Skip again

• Click Next

• For Deploy provider, select AWS AppConfig

• For Application, select the application you created in Step 1

• For Environment, select the environment you created in Step 2

• For Configuration profile, select the configuration profile you created in Step 3

• For Deployment strategy, select AppConfig.AllAtOnce

• In Input artifact configuration path, enter "review.txt"

• Click Next

• Review the information and click Create pipeline

In this case we're choosing the deployment strategy AllAtOnce, but we could opt for a gradual rollout. This plays well with the automated rollbacks when a CloudWatch Alarm is triggered.

Step 5: Test the application again

Just check that everything is still working.

• Go back to the browser tab where you pasted the public IP address of the instance and refresh the page

• Verify that the message still says "to do: write thank you message"

Step 6: Make a change to the configuration file and upload it to S3

This step is to simulate a configuration change. We're still not ready to use the configuration from AppConfig, but since the pipeline can take a few minutes, it's better to do this now.

• Go to the S3 console

• Click on the bucket you created with the initial setup

• Click on the file called "review.txt.zip"

• Click Download

• Unzip the file

• Open the file with a text editor

• Write a short review (one or two sentences) of the Simple AWS newsletter that I can publish on the website

• Save the review.txt file with the changes

• Zip the file into a file called "review.txt.zip"

• Go back to the S3 console

• Click Upload

• Click Add files

• Find and select the file review.txt

• Click Upload

Step 7: Update environment variables

The configuration isn't actually pushed to the instance (notice that so far we haven't ever referenced the instance), but instead to a configuration repository inside AppConfig. It's the application's responsibility to fetch it from that repository. Here's what the code to do that looks like:

const application = process.env.APPCONFIG_APPLICATION;
const environment = process.env.APPCONFIG_ENVIRONMENT;
const configuration = process.env.APPCONFIG_CONFIGURATION;
const clientId = process.env.APPCONFIG_CLIENTID;

const fileNameFallback = process.env.FILE_NAME;
const localFilePath = './' + fileNameFallback;

async function fetchAppConfig() {
  const params = {
    Application: application,
    Environment: environment,
    Configuration: configuration,
    ClientId: clientId,
  };

  try {
    const data = await appConfigClient.send(new GetConfigurationCommand(params));
    const charCodes = new Uint8Array(data.Content);
    const message = String.fromCharCode(...charCodes);
    return message;
  } catch (error) {
    console.error(`Failed to fetch AppConfig: ${error}`);
    return null;
  }
}

app.get('/', async (req, res) => {
  let message = await fetchAppConfig();
  if (!message) {
    try {
      message = fs.readFileSync(localFilePath, 'utf-8');
    } catch (e) {
      console.log('Error: ', e);
      res.status(500).json({ error: 'An error occurred while reading the file' });
      return;
    }
  }
  res.set('Cache-Control', 'no-store, no-cache, must-revalidate, private');
  res.json({ message: message });
});

We need to set those environment variables with the ID of the Application, Environment and Configuration Profile that we just created, so the code knows where to go fetch that config.

• Go back to the AppConfig console

• Click on your application

• Copy the Application ID

• Go to the Environments tab and click on your environment

• Copy the Environment ID

• Go back to your application, go to the Configuration Profiles and Feature Flags tab and click on your configuration

• Copy the Configuration Profile ID

• Go to the EC2 console

• Select the instance that was created with the initial template

• Click Connect

• Go to the Session Manager tab and click Connect

• Type the following commands, replacing REPLACE_APPCONFIG_APPLICATION_ID for the ID of the AppConfig application, REPLACE_APPCONFIG_ENVIRONMENT_ID for the ID of the environment, and REPLACE_APPCONFIG_CONFIGURATION_ID for the ID of the configuration.
  Note: You'll need to copy and paste them one by one

sudo su

echo "APPCONFIG_APPLICATION=REPLACE_APPCONFIG_APPLICATION_ID" >> /home/ec2-user/app/.env

echo "APPCONFIG_ENVIRONMENT=REPLACE_APPCONFIG_ENVIRONMENT_ID" >> /home/ec2-user/app/.env

echo "APPCONFIG_CONFIGURATION=REPLACE_APPCONFIG_CONFIGURATION_ID" >> /home/ec2-user/app/.env

echo "APPCONFIG_CLIENTID=simpleaws" >> /home/ec2-user/app/.env

source /home/ec2-user/app/.env

kill -9 $(ps aux | grep '[n]ode src/index.js' | awk '{print $2}') && node src/index.js > /home/ec2-user/app/app.log 2>&1 &

Step 8: Test the app again

Now we should see the message!

• Wait for the pipeline to finish if it hasn't. You can view the progress in the CodePipeline console

• Go back to the browser tab where you pasted the public IP address of the instance and refresh the page

• Verify that the message has changed and now says what you wrote in Step 6

Step 9: Send me that review you just wrote (optional)

You just wrote some nice words about Simple AWS! If you want, send them to me on LinkedIn and I'll publish them on the Simple AWS website. Please specify the name you want me to publish it under, if not I'll just use Anonymous to preserve your personal information.

If you don't want me to publish it, that's totally cool.

Benefits of Using AppConfig and CodePipeline

I believe the first thing to discuss here is, if AppConfig isn't pushing the config file into the instance, what's the difference between storing the configuration file in S3 and storing the configuration "file" in AppConfig?

Here's the thing: Using AppConfig isn't as easy as I made it look like (as if that was easy!!). What you need to do for a proper AppConfig configuration is start a configuration session and then fetch the latest configuration. This way, when a new configuration deployment happens, the configuration is pushed to all open sessions. I started doing it that way, but the step by step would have taken like 15 steps.

One of the benefits is the deployment strategy, which we can't appreciate with just 1 instance. When we're using AllAtOnce, all sessions get the new config at the same time. However, we can also do a rolling update, where sessions are gradually updated, without doing a rolling update at the Auto Scaling Group level (i.e. re-creating the EC2 instances). And the third option is a canary deployment: First a small percentage of sessions get the new config, then the rest.

Another benefit is the automated rollbacks. We can configure CloudWatch Alarms that are associated with an Environment, and when a new configuration deployment happens, those alarms are monitored and, if they trigger, the deployment is automatically rolled back. We could do that manually with S3, rolling back to a previous version of the object on a bucket with versioning enabled. But why do it manually?

Finally, we have feature flags. It's way too long to really discuss here, but the short of the long is this:

• Feature flags are boolean variables (hence the name flags) that enable or disable features or parts of the code. The code is wrapped in an if statement, and if the flag is true, the code runs.

• The idea is that feature flags are a configuration, so we can deploy the code that supports the new feature, with that configuration turned off, and the app acts like before the deployment. Then we can enable the feature with just a configuration change.

• This is why feature flags are a part of AppConfig.

• Feature flags are often used in Trunk-Based Development (TBD), where everyone commits their code directly to main (previously called master, previously called trunk). Since this often leads to incomplete code being committed, that code is hidden behind feature flags, so it doesn't impact the app.

• Feature flags is an easy problem to solve at the technical level, using AppConfig or some other apps.

• It's actually hard to solve at a process level, since you need to determine how to create new feature flags, when to delete feature flags, how all of that impacts other developers in the same team, how to mix development feature flags (for still in development features) with production feature flags (features that are deemed ready, and are released through feature flags), and how to avoid feature flag hell: there's a million feature flags and nobody knows what 60% of them do.

Something worth pointing out is that, after these changes, the instances no longer need access to S3. They'll be getting their configuration from Systems Manager AppConfig now. So, if you had already set up a VPC Endpoint for S3 like we discussed in a previous article about VPC Endpoints, you should delete it and instead create a VPC Endpoint for Systems Manager.

Best Practices for AWS AppConfig

Operational Excellence

• Use CI/CD for Configuration Changes: This is why we're using CodePipeline.

• Use CloudWatch Alarms: As I mentioned, deployments can be automatically rolled back if a CloudWatch Metric triggers an Alarm. The obvious example metric would be percentage of responses that are 5XX.

Security

• Use Secrets Manager: Secrets Manager can't handle configs as well as AppManager, but it's the most secure way to manage secrets like database passwords.

Reliability

• Automated Rollbacks in CodePipeline: Set up automated rollbacks in your CodePipeline to automatically revert changes if a pipeline execution fails. This is separate from reverting the deployment from AppConfig, this is about some error in CodePipeline.

• AppConfig Deployment Strategy: Use canary or linear deployments to gradually deploy configuration changes. This will minimize the blast radius if there's an issue during deployment, such as a bad configuration.

• Use AppConfig Validators: Use Validators to ensure the config file conforms to a predetermined schema, thus completely avoiding deploying a badly formatted file.

Performance Efficiency

• Optimize Config File Size: Minimize the size of configuration files for faster deployments.

Cost Optimization

• Don't push micro updates unless you have to: AppConfig is priced at $0.0000002 per configuration request and $0.0008 per new configuration received. You can't avoid configuration requests, but you can minimize new configurations received if you only group config changes and push them together. For reference, for 2000 servers (that's a lot!) you're paying $8.64/month for configuration requests and, with 3 configuration pushes a day, $144/month for new configurations received, for a grand total of $152.64 (that's a lot! but if you have 2000 servers I bet that's less than 1% of your AWS bill)