AI Operations – Why Your Telemetry Now Has a Job Title

The hype train of AI continues to steam along at such a pace that even the most diligent people are struggling to keep up. Its hard most days not to hear or read someone celebrating the rise of the Agents (if you don’t picture Neo fighting all the Agent Smith’s in The Matrix Revolutions film maybe you are a little too young).

Agents are incredibly useful and powerful tools however I can’t help but feel whenever a conversation is in progress and something needs to be done someone will always pop up with the comment “We could build an agent for that” well yes, you could but we often forget the question does it actually need and Agent?

Recently I have been on the rounds with AWS attending a few events and while Agents still feature heavily in the programming, I have noticed a shift towards the view of deterministic vs non-deterministic outcomes.

This got me thinking about the operations space and how this may start to impact how operations is done, and what is the impact on how we setup monitoring, alerting and logging and the view that this is going to become more important for AI systems than our Support engineers.

For those of you who have been around operations long enough you will be well versed in my next sentence.  Every operations team carries a private list of lies their monitoring tells them.  That alarm that always flaps on a Monday morning, “It’s ok Jim its 10:16 on Monday morning that alarm always flaps at this time”, or the Service that logs an Error when everything is perfectly fine “I wish someone would look into that bug”. This info is not written down, it just lives rent free in the engineer’s minds, and this quietly turns bad telemetry into good decisions.

I wrote earlier this year about the AWS DevOps Agent and the 3am moment it is built for: the phone buzzes, the alert makes no sense, and you start stitching context together from fifteen tabs. The agent now does that stitching for you. What I did not say loudly enough is what that changes underneath. The human at 3am was never just a reader of alerts, they were the error correction layer, translating imperfect signals using knowledge accumulated over years of action.

Now we are handing that translation job to agents that have never met your Monday mornings. They read exactly what you give them, and they act on it.

This post asks two questions. How accurate does logging, monitoring and alerting now need to be? And once an agent is looking at that data, which decisions should stay deterministic, and which can safely be left to a non-deterministic approach?

When I tested the AWS FinOps Agent, I asked it to look back over twelve months of costs and show me where spend had grown fastest. It came back with a headline: KMS costs up 400% in a single month. That is the kind of number that gets a meeting booked.

So, I had to check this out, there had been no surge in KMS key creation in that month. The agent had misaligned its dates, and the real jump in keys sat in the month before, most likely tied to the billing cycle when those keys were generated. A few more prompts and it reached the right answer.

Nothing in that example was broken, the data was real and the agent was reasoning in good faith. The failure was context: a time boundary a human would instinctively sanity check and the model did not.

That is the pattern to watch for, Agents are very good at consuming signals and but are much less inclined to doubt them. Both AWS agents I have tested lean on context you supply. The FinOps Agent reads your account-to-owner mappings, team definitions and tagging conventions. The DevOps Agent builds a topology of your estate and uses Skills to encode your runbooks. Where that context is missing or wrong, the agent does not hesitate. It reasons confidently from a bad position.

The old failure mode of poor telemetry was a confused engineer, who slowed down and asked a colleague. The new one is a fast, fluent across all your data, with a potential wrong answer that may trigger an action.

Accuracy is not a feeling. It breaks down into four properties you can configure, test and audit.

Alarms are loud but there is information in the silence

CloudWatch alarms handle gaps in data through a setting called treat missing data. There are four options: missing, notBreaching, breaching and ignore. The default is missing, which moves the alarm to `INSUFFICIENT_DATA` when every datapoint in the evaluation range is absent.

That is a state, not a page callout. Unless you attach actions to `INSUFFICIENT_DATA`, a service that has stopped reporting entirely can sit there quietly, and an agent triggered by alarm actions never wakes up. The opposite choice, notBreaching, turns the same gap into OK, so a dead service looks healthy.

For a metric that should always report, treating missing data as breaching is the setting that makes silence loud. For a metric that only emits when something goes wrong, such as the DynamoDB throttling example in the AWS documentation, notBreaching is correct. Neither is right everywhere, and that is the point, it is a per-alarm decision someone must make deliberately, and an agent downstream will trust whichever one you made.

Your flapping alarms are feeding Agent’s noise

Evaluation Periods and Datapoints to Alarm turn a single breach into a 3 out of 5 judgement. This means that you need to find a sweet spot with them, set them too tightly and the alarm that flaps every Monday morning sends an agent on the same fruitless investigation every Monday morning. If you set them too loose and detection lags behind the damage. The agent cannot tell which mistake you made, so it inherits it.

One Trail for Hansel and Gretel

When an agent answers, “who changed what, and when”, that answer is only as good as your CloudTrail coverage. An organisation trail, created from the management account or a delegated administrator, logs events for the management account and every member account. Any account outside it is a blind spot, and an agent has no way to know the evidence is missing. This is where compliance and controls across all accounts become critical and ensuring your deployment processes ensure all accounts are deployed and enrolled into the same way.

Structure and Context

Free-text logs force an agent to guess at meaning. Structured JSON logs with consistent field names let Logs Insights and metric filters work on fields instead of regular expressions and leaves less room for misunderstanding.

Context is the part my FinOps testing exposed. Account-to-owner mappings, tagging conventions and Skills are the written-down version of what your best engineer carries in their head. If it is not written down, the agent does not have it.

A simple test for all four, if you onboarded a new engineer tomorrow could they act correctly using only the alert and its linked runbook? If the answer to this is “no” an agent cannot either.

Use non-deterministic AI to think, and deterministic mechanisms to act. That one rule settles most of the arguments I hear about agents in operations.

Where a non-deterministic approach earns its place

– Correlating signals across tools that were never designed to talk to each other

– Forming root cause hypotheses

– Summarising an incident for a human in plain English

– Enriching a ticket with context before anyone opens it

– Handling failures nobody has seen before

– In each case the cost of being wrong is a bad hypothesis, which a human can discard.

Where a deterministic approach wins

– Known failure modes with a known fix

– High blast radius changes

– Compliance-bound actions

– Anything that must behave identically on a Tuesday afternoon and on the night of an audit

EventBridge rules, SSM Automation, Config remediation, Auto Scaling and SCPs are not exciting, and that is exactly why you can trust them. In each case the cost of being wrong is a bad action, which may not be reversible.

Two questions place any given decision: have we seen this situation before, and can the action be undone?

Autonomy is a dial, and it only turns up as an agent earns trust. Neither of the AWS agents I have reviewed can change your infrastructure, which tells you where AWS thinks the dial should start.

Take one alarm on one service and give it two lanes. A known failure goes down the deterministic lane and acts. Anything else goes down the agent lane and gets investigated. The EventBridge rule is the director, and the routing rule is the design decision that matters.

The deterministic lane

In this example an Eventbridge rule matches the alarms you already have a runbook for. The rules target is an SSM Automation runbook, where the fix is not easily undone, the runbook contains an approval step, this ensures that an engineer has the final say on the implementation action.

The Agent Lane

A second rule uses anything-but matching on the same list of alarm names, so every alarm without a runbook goes to the DevOps Agent, through its built-in integration or a webhook. Its only write actions are tickets and support cases, so the worst case is a wrong hypothesis in a channel, this is the cheap kind of wrong, going back to the speed at which agents can operate with implementation rights, its fixed one thing, broke another and thought it would be handy if it re-wrote your code all before the kettle boiled for the coffee you are now going to need to unpick what your friendly little helper has done for you.

The promotion for predictability

Each time the agent finds a root cause that keeps coming back, it is time to promote this to the deterministic lane by encoding the fix as a runbook and move that alarm to the deterministic lane. The agent becomes a discovery engine for deterministic automation, and over time the agent lane shrinks to the genuinely novel. Having predictability means a better mean time to resolution.

There are still failure modes in using the agent and there are still things we need to consider and control, every failure mode below has a control, and most of those controls are boring and deterministic. That is rather the point of this post.

What you will notice above is what the controls all have in common.  none of them asks the model to be better, they ask you to make the environment around the model to be more predictable.

If agents take over the first hour of an investigation, the human role does not disappear. It moves upstream, and four things change.

1. From triaging to reviewing – The engineer’s job shifts from working out what is broken to judging whether the agent’s analysis and proposed fix are sound. That is a different skill, and it needs a baseline of how often the agent is right.

2. From knowing the runbook to writing it down – Tribal knowledge becomes Skills, automations and context files. The engineer who used to be the person everyone pinged at 3am becomes the person who encoded what they knew so nobody else needed a crystal ball

3. From closing tickets to owning signal quality – The DevOps Agent surfaces recommendations in an Observability category, flagging gaps in monitoring, alerting or logging that slowed detection. Someone must own acting on those, or the backlog becomes another graveyard of ignored action items.

4. Owning the improvement loop – As the agents produce better results to their investigations, the job of creating runbooks and promoting resolutions to the deterministic lane becomes a constant loop of improvement.

For a managed service provider this changes what a customer is really buying. Less of it is hours of ticket handling. More of it is the quality of the signal, the design of the guardrails, and the evidence that the automation is behaving and this drives towards outcomes not tickets closed.

It also suggests what to measure. Time to detect, how often the agent’s root cause matched what a human concluded, and the keep, discard and implemented rate on its recommendations tell you more about operational maturity than ticket volume does.

Anyone pushing you towards a massive programme of work to take on the world with Agents should be given a wide berth…. I am a big advocate of the acronym KiSS (Keep it Simple Stupid) and its approach (just for clarification I am referring to myself as the stupid part), it is easy to come up with all these wild ideas of what you could do but genuinely you need to start small and pick one alarm.  Once you have got a few under your belt the process will accelerate so don’t think slow progress is not acceptable.

Pick one alarm and action path that has genuinely caused you pain, and work through it in four steps.

1. Audit the signals four properties – Check the missing data setting, the evaluation periods, whether the logs are structured, and whether the owner and runbook are written down anywhere an agent could read.

2. Make the known fix deterministic – If you already know what the fix is, route the alarm through EventBridge to an SSM Automation runbook, with an approval step where the action is not safely reversible.

3. Let an agent investigate in parallel, read only – Point it at the same alarm and let it produce a root cause hypothesis without any ability to act on it.

4. Compare for a month – Put the agent’s analysis next to what your engineers concluded, and note where it was right, where it was wrong, and why.

The wrong answers will be the most valuable output. Each one points at a gap in your telemetry or context that an engineer was quietly covering for. These can then be Fixed, and you have improved your estate for humans and agents alike.

The real shift in this era is not that machines can now act on our monitoring, it is that our monitoring now must be good enough to be acted on. The engineers who used to translate bad signals in their heads are still needed, they just do their best work earlier, in the configuration, long before 3am.