Site icon Blog.Chiffers.com

Build an Agent That Earns Its Autonomy: A Sampled Audit in Azure Logic Apps

Craig Chiffers

TL;DR: In my last post I argued that autonomy should be earned, not granted. This is the hands-on sequel. One Azure Logic App that calls an actor agent, checks it with an independent auditor, sends anything risky to a human, and then adjusts its own audit sample rate based on measured results. The template, scripts and a simulation are in the companion repo. Build the thing that earns the trust before you build the thing that spends it.

In my last post I made the case for a ladder instead of a leap. Start with a Logic App that owns the control flow, then hand the agent more authority one rung at a time, and only when the data says it has earned it. The obvious next question is: lovely, but what does that actually look like in a workflow?

A quick recap for anyone arriving here first. The Tuesday queue was a logistics workflow where a Logic App has an agent read carrier invoices and then waits for a person to approve every result. It started well, drifted into a rubber stamp, and one day the same bill got paid twice. Awkward. That is rung one of the ladder: the agent is a step, and a human gate sits on every run. The way out was not to remove the gate but to demote it, using what I called the autonomy ratchet. Think of it as a dial that controls how much of the agent’s work gets checked. It turns up slowly, one small step at a time, each step earned with evidence, and it snaps straight back the moment the evidence turns bad. The name comes from the ratchet in a socket wrench, which clicks forward and cannot slip back until you pull the release.

This post is the build. If you only read one half, read this one, because it is the half you can run.

Fair. A ladder diagram does not deploy itself. So here is a working example, built on the stack a lot of you already run: Azure Logic Apps (Consumption), Azure OpenAI, Table Storage and managed identities. No custom connectors, no containers, and nothing you cannot read in one sitting.

The goal is narrow on purpose. This is not a framework. It is one pattern, the sampled audit with an autonomy ratchet, which is the mechanism that lets you demote a human gate from “blocks every run” to “checks a slice of runs” without simply crossing your fingers. It implements the audit, ratchet and kill switch from steps four and five of the migration path in the last post, and the demo workflow is called invoice-triage, which is the Tuesday queue from that story.

The Pattern in One Minute

Every run goes through the same ten steps, and only two of them involve a model. The rest is plumbing, which is exactly where you want your plumbing.

  1. Trigger. An HTTP request arrives with a task and an action class: reversible or irreversible.
  2. Read state. Pull the autonomy mode and sample rate for this workflow from Table Storage.
  3. Kill switch. If the switch is on, return a 503 and terminate the run.
  4. Actor agent. An Azure OpenAI deployment proposes a decision, with a confidence score and a rationale.
  5. Audit needed? In gated mode, always. In sampled mode, a random slice. Irreversible actions and low confidence are always audited, whatever the mode.
  6. Independent auditor. A different model, called with a different identity, passes or fails the actor’s decision.
  7. Policy gate. Irreversible, low confidence or a failed audit goes to a human. Everything else executes.
  8. Human or execute. A Teams escalation, or your real downstream action.
  9. Autonomy ratchet. Update the failure rate and decide whether to promote, hold or demote.
  10. Persist. Write the new state and one audit row, then respond.

The interesting design choice is where the authority sits. The agent proposes. Deterministic rules decide whether that proposal is trusted. And a small, boring piece of arithmetic decides how much scrutiny the next run gets. None of it relies on the model grading itself kindly.

State Lives in One Row

The whole autonomy history of a workflow fits in a single Table Storage row, and that is a feature. It keeps the ratchet explainable on a slide, which matters when the person asking the questions is a CFO.

The row holds the current mode (gated or sampled), the sample rate, the running audit and disagreement counts for the current evidence window, the kill switch, and the last ratchet decision. Seeding it looks like this:

{
  "PartitionKey": "invoice-triage",
  "RowKey": "state",
  "mode": "gated",
  "sampleRate": 100,
  "audits": 0,
  "disagreements": 0,
  "killSwitch": false,
  "lastDecision": "none"
}

I talk to Table Storage with a plain HTTP action and the Logic App’s managed identity, rather than a connector. That means no stored keys and no connection object to rot. The Table REST API wants an x-ms-version header and, to be safe, an x-ms-date header, and an Accept of application/json;odata=nometadata keeps the payloads tidy. A PUT to the entity URL is an insert or replace, which is all the ratchet needs.

One gotcha worth knowing up front: the numbers and booleans must go in as real JSON numbers and booleans, because the workflow parses them with a typed schema. If you seed the row from the CLI as strings, the typed schema will reject the row and your run will fail at the parse step. The repo includes a seed.sh that sends proper JSON and creates both tables for you.

The audit trail goes in a second table, one row per run: the task, the actor’s decision, confidence and rationale, whether it was audited, the verdict and reason, the outcome, the sample rate at the time and what the ratchet decided. That table is your self reporting. You do not bolt a report on later, you query the log.

Two Models, Two Identities

An auditor only counts if it is genuinely independent of the actor, and in a Logic App that is a deployment decision, not a prompt. If the auditor is the same model, called with the same identity, reading the same context, you have built an agent that marks its own homework. It will mostly agree with itself.

So the demo uses two separate Azure OpenAI deployments, and two separate identities. The actor is called with the Logic App’s system assigned managed identity. The auditor is called with a user assigned identity that only has permission to call the model. The HTTP action makes that explicit:

"authentication": {
  "type": "ManagedServiceIdentity",
  "identity": "@parameters('auditorIdentity')",
  "audience": "https://cognitiveservices.azure.com"
}

Both identities need the Cognitive Services OpenAI User role on the Azure OpenAI resource, and the Logic App’s system identity also needs Storage Table Data Contributor on the storage account. Give role assignments a few minutes to propagate before you decide something is broken.

Both calls use JSON mode and a temperature of zero, so the output is a parseable object rather than a paragraph. The actor returns a decision, a confidence between 0 and 1 and a rationale. The auditor returns pass or fail and a reason, and is told in the system prompt that it did not make the decision and should fail anything it is unsure about.

There is a real trade-off in what the auditor sees. In the demo it gets the task, the decision and the actor’s rationale. That is realistic, but a persuasive rationale can talk an auditor round. If you want a harder audit, withhold the rationale and make the auditor reach its own view from the task alone. Try both and compare your disagreement rates. That experiment is more useful than any opinion I could give you.

The Policy Gate Is Deliberately Dull

The gate that decides whether a human gets involved is three conditions in a single If action, and I would not make it any cleverer. Escalate when the action is irreversible, when the actor’s confidence is below 0.7, or when the audit verdict is fail. Otherwise, execute.

The first condition is the important one. Autonomy is granted per class of action, never globally, so an irreversible action goes to a human whatever the ratchet says. In the demo that means a payment run is always audited and always escalated, even when the same workflow has earned a 5% sample rate on everything reversible. Drafting a reply and moving money are not the same kind of verb, and your gate should know the difference.

A tip on the second condition. A language model’s self reported confidence is a hint, not a calibrated probability. I treat it as a tripwire that sends borderline cases to a human, never as a measurement. The ratchet’s numbers come from the auditor, not from the actor’s opinion of itself.

One thing I kept simple on purpose: the escalation in the demo is a notification. It posts a message to a Teams webhook and records the outcome as escalated_to_human. For a real workflow you want the run to wait for an approval and carry on from the answer, which is a natural next step with the Approvals connector or an adaptive card. I kept the demo small so the pattern stays visible.

The execute branch is a placeholder Compose action. Replace it with your real call to the ERP, the claims system or the ticketing tool.

The Ratchet: Slow to Trust, Quick to Distrust

The ratchet is five tiny Compose actions and no code, and it is where the whole idea either works or does not. I split it into separate actions on purpose. When something looks odd, you open the run history and read each number in order instead of decoding one monster expression.

The rules are simple. Every audited run updates two counters: audits and disagreements, where a disagreement is a failed audit. Then:

The decision itself looks like this. It reads left to right, which is the point:

@if(less(outputs('Calc_Audits'), parameters('minAudits')), 'hold',
  if(greater(outputs('Calc_Rate'), parameters('demoteAbove')), 'demote',
    if(and(greaterOrEquals(outputs('Calc_Audits'), parameters('windowAudits')),
           lessOrEquals(outputs('Calc_Rate'), parameters('promoteBelow'))),
       'promote', 'hold')))

All four thresholds are Logic App parameters, so you can tune them per workflow without touching the logic.

Why the asymmetry matters

My first version used the same evidence for both directions, and I only found the problem because I simulated it before trusting it. I wrote a small script that runs the exact same rules against thousands of synthetic runs with a burst of model drift in the middle, and it turned up two flaws.

The first was re-promotion in the middle of an incident. With symmetric thresholds, one lucky window of 20 clean audits promoted the agent while it was still misbehaving. The fix is that promotion needs a bigger window than demotion. Trust is slow to earn and quick to lose.

The second was drift hidden by old data. When the counters only reset after a decision, a long clean history diluted a new failure rate until it was never noticed. A window that resets every time it fills fixes that.

The simulation also taught me something worth knowing. Across 200 random seeds with an error rate jumping from 1% to 12%, the first demotion came a median of 284 runs after the drift began, and 580 runs at the 90th percentile. In the example I plotted, about 60 bad outputs ran unaudited before the ratchet noticed. Your floor sample rate is your blast radius. That is exactly why irreversible actions are audited every time regardless of mode, and why the floor is a number you should choose deliberately rather than inherit.

Two caveats before anyone quotes those figures. It is a simulation, and it assumes the auditor catches every error it samples. Real auditors are not perfect, so measure yours against human review too. And the thresholds in this demo are deliberately quick so you can watch them move. As the last post explains, 20 clean audits prove far less than they feel like they do, so size your real windows from the cost of a wrong outcome.

Running It Live Without Praying

A demo that depends on a language model misbehaving on cue is not a demo, it is a gamble. So the template has a demoMode parameter that lets the trigger carry a forceVerdict of pass or fail. The auditor still runs, but its verdict is overridden, which makes the ratchet fully repeatable. For the demo, deploy with demoMode on, minAudits at 5 and windowAudits at 10 so the thresholds are reached in a few calls instead of dozens.

Then walk through four phases. Print the state row between them with show_state.sh and let the audience watch the numbers move.

  1. Ten clean runs. Routine invoices go through, every run is audited and passes, so on the tenth run the ratchet promotes the sample rate from 100 to 50.
  2. The auditor starts failing outputs. Imagine a carrier changes its invoice format and the agent starts misreading it. At a 50% sample rate it takes roughly ten to twenty runs to collect enough audits, and then the ratchet demotes straight back to 100. Failed audits also escalate to Teams, so this phase shows the human gate working as well.
  3. An irreversible action. In the Tuesday queue this is the payment run. It is audited and escalated every time, whatever the sample rate says.
  4. The kill switch. Set killSwitch to true on the state row, call the workflow again and watch it return a 503.

One thing to know: a forced verdict only counts when the run was actually audited. On an unsampled run the verdict stays skipped, which is correct, because nothing was checked.

And treat demoMode like a party trick: brilliant on stage, retired before real data arrives. It lets a caller override the auditor, which is the exact opposite of what you want in production, so switch it off first.

Where This Goes Next

This is a pattern demo rather than a production platform, which is exactly why it fits on a few screens. The good news is that every gap has a clear upgrade, and each one is a nice weekend of work. Here is the path from demo to something you would point at real work.

Treat the simulation numbers as a map of the trade-off, not a forecast. Your error rates, your floor and your tolerance for slow detection will all be different, and you should measure them on real cases from your own workflows.

Where to Start

Pick one workflow where mistakes are cheap and reversible, and where you already have history to build an evaluation set from. Run it gated for a while. Look at the failure rate before you promote anything. Choose the floor sample rate on purpose, knowing it is also the size of the hole a drifting model can walk through. Then let the ratchet earn each step.

The point is not to automate the human out. It is to spend human attention where the data says it is needed, and to be able to prove it. That is the version of autonomy a CFO, a regulator and an on-call engineer can all live with.

Back to the Tuesday queue one last time. Suppose the carrier changes its invoice format on a Friday. In the old design, a tired reviewer could approve the misreads for weeks. In this design the auditor’s disagreements pile up, the ratchet snaps back to checking everything, payments stay with a person throughout, and the trace shows exactly which invoices were affected. Same agent, same mistakes, a very different Tuesday. That is the whole argument of this short series: you do not need to choose between a gate that gives false comfort and an agent nobody can check.

The two natural extensions are a proper approval step in place of the Teams notification, and moving the audit off the critical path. If you build either before I do, I would love to hear how it goes.

Earn the autonomy first. Then go build something.

If you try this pattern, or you have a better way to handle the audit and escalation steps, I would love to hear about it. Drop a comment below.

Further Reading

0 0 votes
Article Rating
Exit mobile version