Suppose you ask an agent to move an appointment to Tuesday afternoon. It replies with a friendly confirmation, the right time, and the right address. But the scheduling system still has your original appointment. Or perhaps the agent moves the appointment correctly, but forgets to tell you the location.

Both interactions need improvement, but the work required is different for each failure mode. One involves the agent's actions, the other involves how it explains their result.

We'll use this hypothetical scheduling agent to work through agent optimization on AWS. Our goal is to get the agent to complete more requests correctly, and to do it with as little work as possible on our side. Amazon Bedrock Advanced Prompt Optimization and Amazon Bedrock AgentCore Optimization are going to help us automate different parts of the process.

This article is sponsored by Inngest

Agents, like all of us, often need to recover from failure. But they shouldn't start from scratch.

Inngest's durable execution checkpoints your agents every step of the way.

So your agents can pick up where they left off, and avoid doubling up on token & compute spend.

Inngest offers a generous free tier and a sleek local dev server for testing.

You always need to start with a failure mode you can describe precisely. “The agent isn't reliable” doesn't tell you what to change. Instead, “The agent creates a second appointment when asked to reschedule the existing one” gives you something to investigate, and a clear goal to optimize towards.

From there you need to pick examples where that happened, and read the session trace: the record of the model's outputs, tool calls, and returned results. Use that to investigate precisely what went wrong. Did the agent choose the wrong operation? Did it send the wrong appointment ID? Did the correct operation return an error? Each of these causes will require a different fix. AWS's agent debugging guide dives a bit deeper into the topic.

For our scheduling example, the first experiment we need to run depends on what the trace shows:

This is part of what makes agent optimization difficult. Changing one instruction can affect several decisions, and the same agent can behave differently across repeated attempts. You need enough of the execution history to understand where the failure began, so you can know what to do next and how to measure if it's improving. Anthropic's agent evaluation guide makes the useful distinction between the conversation transcript and the final state of the environment. Use both.

Once you know precisely what you want to optimize, you'll know which of these AWS services can help you.

Amazon Bedrock Advanced Prompt Optimization, or AdvPO, takes a prompt template, example inputs, and an evaluation method. It runs the examples through your selected models, scores the responses, and uses that feedback to rewrite the template. Then it repeats the process, and keeps improving.

That removes the manual cycle of editing a prompt, running the same examples, comparing responses, and deciding what to try next. You still supply the task and the definition of a good result, it does its best to change the task so its results approach what you define as “good”.

For our agent's incomplete confirmations, AdvPO is a sensible starting point. We can isolate the model call that turns a scheduling result into a message, and focus on that. The input already contains the new appointment details, and the output should communicate them accurately.

The input file is JSONL, with one record per prompt template. Each record contains its examples and evaluation configuration. Here's an example record, it's expanded for readability but in the actual file it would be a single line. Note that this is just one example, not a complete evaluation dataset, which should contain multiple lines.

The {{request}} and {{result}} placeholders receive their values from each sample. Each variable occupies its own object in inputVariables. Reference responses are optional, though highly recommended, since they give the evaluator a concrete example of the desired answer and thus a way to understand what a good answer looks like. You can find the complete schema at the input documentation.

It's important to include examples of failures. If every sample contains a successful reschedule, you give the optimizer little reason to distinguish between confirming an action and explaining that it couldn't happen. Add representative variations, including cases the existing prompt already handles well, and keep some examples aside for checking the result afterward (this is called a held-out set in Machine Learning jargon).

AdvPO provides three evaluation approaches: natural-language steering criteria (which is what the example above uses), a custom LLM judge with your own rubric, or a Lambda function that computes a score. A judge is useful for semantic requirements such as an accurate, complete explanation. Lambda is useful when code can check exact fields, values, or structured output. I wrote about an experiment we ran at work using a Lambda evaluator.

Choose based on what factors make an answer wrong. A beautifully written confirmation containing the wrong appointment time must score poorly, so anything that measures how well-written it is won't work for this case. Remember that the optimizer follows the feedback you give it.

If part of your template already works well, selective optimization lets you mark editable sections with <advpo:optimize>. Content outside those blocks is preserved, and the returned prompt has the tags removed.

Before submitting, make sure your identity has the right permissions to manage optimization jobs, invoke the selected models, and read and write the S3 locations. The S3 bucket where you put the JSONL file must be in the job's AWS Region. Custom Lambda evaluation and KMS encryption add their respective permissions, of course. Here's the list of prerequisites, which also describe cross-Region inference used during optimization.

In the Bedrock console:

The API equivalent is CreateAdvancedPromptOptimizationJob. If you're trying to improve the prompt, start with your current model. You can also do a model comparison, in that case include the current model and candidate replacements. The service supports up to five models, ten templates per job, and 100 examples per template. Current limits and supported Regions are important to check before preparing a larger input.

The results contain optimized templates for the target models, sample scores, cost estimates, and time to first token. TTFT tells you when a response begins, so it doesn't measure how long the whole agent task takes, but it's almost always the best metric to measure user-perceived latency. Read the changed responses alongside the scores, then compare the original and selected template on the examples you held aside (your holdout set, in ML jargon).

There are two different costs to consider for AdvPO. The optimization job consumes inference for generating responses, judging them, and rewriting prompts, plus Lambda usage if you choose it. The resulting prompt then has its own recurring inference cost when you use it, which might be higher than the cost of the original prompt if the change that improved the response made your prompt longer. AdvPO has no separate service charge beyond its underlying usage. Check Bedrock pricing for how much models cost.

AdvPO evaluates the supplied examples independently. It can see conversation history or tool results you provide, but it doesn't execute those earlier actions. Our confirmation examples let us test how the model describes a supplied scheduling result. The agent still needs to produce that result correctly, throughout its full execution. That's where AgentCore Optimization comes into play.

AgentCore Optimization starts with the agent's current configuration and session traces. Its recommendations propose changes to system prompts or tool descriptions, with explanations of the changes.

Let's go back to the case of our agent selecting the create_appointment tool when the user asked to reschedule. If the descriptions make both tools sound like reasonable choices for the action you're asking the agent to take, it's going to get confused about which to pick. A description should make the distinction clear: one creates a new appointment, the other changes an existing appointment identified by its ID. We need to improve that.

Tool-description recommendations analyze selection patterns directly. You provide names, descriptions, and traces, and receive proposed descriptions with explanations. In the AgentCore CLI, use --type tool-description and provide each description with --tools "name:description". This mode omits the evaluator parameter.

System-prompt recommendations work differently. You supply one numerical evaluator to define the direction of improvement, and that's what AgentCore uses to determine whether it's making progress on improving the response. Builtin.GoalSuccessRate is a good fit for task completion, and Builtin.Helpfulness fits more open-ended interaction quality. You can also use a custom evaluator, which can express a requirement specific to your agent. The result is revised prompt text and an explanation. AWS's system-prompt guide documents these choices in more depth.

You can use CloudWatch to get all the necessary data from your agent's past sessions. Configure AgentCore Observability and Transaction Search, invoke your agent, and let the telemetry arrive. You can also supply captured OpenTelemetry-compatible spans from tests, so you don't need a production deployment before you can generate recommendations. See the setup requirements and supported trace sources for more info on how to set this up.

To improve on failures in the agent's overall instructions, the following example illustrates a system-prompt recommendation using seven days of traces. It assumes an existing AgentCore CLI project. Replace MyAgent and the file path with your project's runtime and current prompt.

--wait waits for the asynchronous recommendation job. Replace RECOMMENDATION_ID in the following command with the job ID returned above to retrieve the proposed prompt and its explanation:

Per the service quotas, you can send up to 20 sampled sessions per recommendation and a 20,000-character prompt limit. This means that providing a long lookback doesn't mean every session was analyzed. You need to choose evidence that represents the behavior you want to improve, not send every session and hope for the best.

Read the proposed change before applying it (especially in prod). If a recommendation resolves ambiguity by changing what the agent is allowed to do, it hasn't necessarily preserved your intended behavior, even if technically it did improve on the specific issue you're trying to fix. The candidate prompt should still express the task you actually want performed.

Apply the candidate to a separate agent version, so you can A/B test and roll back easily if needed. You can deploy another endpoint, or use a configuration bundle: an immutable version of settings such as the prompt, model ID, and tool descriptions. Bundles are optional, but I recommend them. Remember that your agent must read and apply the configuration, just creating one won't replace a prompt hard-coded in your application.

Run the original and candidate agents on the same scenarios, including straightforward rescheduling, ambiguous appointment requests, and unavailable slots. Start each trial with fresh conversation state and independent or reset appointment data and slot availability. Otherwise, the first run can change the task faced by the second.

Check the resulting appointment state and confirmation. Include cases that previously worked so you can make sure that the change that helped one behavior didn't break something else. If the difference isn't clear, repeat the relevant trials and compare outcomes across attempts.

AgentCore batch evaluation scores recorded sessions and returns aggregate results. However, it doesn't take a revised prompt and rerun your agent. Generate fresh sessions with the candidate first, since rescoring the original sessions would only tell you about the original agent. The SDK's dataset runners can handle invocation, telemetry collection, and evaluation together, though that interface is in public preview as of writing this.

Then you should confirm a change under live usage, and for that AgentCore A/B testing splits Gateway traffic between the original version and your new, hopefully improved one. Traffic is sticky by session, keeping every message in a conversation on the same version, so users won't perceive changes in quality within the same session. You can run this for a while and then compare the sessions from both versions.

The experiment results that you can see include per-evaluator changes and statistical uncertainty. Based on this you should check whether the improvement is large enough to matter and whether another relevant measure regressed. With this you'll have enough information to know whether you should deploy the new version or not.

You can also do this for agents hosted outside AgentCore Runtime, but they'll require instrumentation, reachable Gateway targets, and Gateway tracing to associate scored sessions with variants. That's useful when you want managed experiments around an existing agent, but it is setup work to account for.

Recommendations, batch evaluation, and A/B testing are generally available. Recommendation generation has no separate charge, you pay for new evaluations and the underlying resources. If you use batch evaluation, you get a 25% discount from standard evaluation rates. Live experiments consume Gateway, evaluation, and execution resources at their normal rate, with telemetry billed separately. The AgentCore pricing page has all the details, I'm afraid it would be hard to p.

The choice follows the work you need to do:

For example, you might use AdvPO to improve the appointment confirmation in our example agent, and to compare its current model with a less expensive candidate. After selecting a promising template/model combination, you integrate it into the agent. Then you run complete scheduling scenarios with both versions, and use AgentCore's live experiment if you need evidence from real user sessions.

You don't need an AgentCore recommendation job just to validate an AdvPO candidate. Nor do you need AdvPO after AgentCore has already produced a useful recommendation. If your existing test setup can establish the relevant outcomes, AdvPO can be enough for the optimization work. If traces point to unclear tool descriptions, AgentCore can take you from that evidence through a proposed change and its validation.

For a model task you can express through examples and a scoring method, start with Bedrock Advanced Prompt Optimization. For changes informed by agent sessions, use AgentCore recommendations and its evaluation tools. Combine them when you need both prompt/model exploration and managed experiments around the complete agent.

Measure the outputs, and make sure you're not regressing behavior in other scenarios. Also, measure the cost, since it may change significantly. Use A/B testing to test the new version with live data, then promote it if it's better.