The observability stack
If you cannot see it,
you cannot trust it.
Agents make many small decisions out of sight. Observability makes those decisions visible to engineers, operators and auditors, each at the level of detail they need.
Each row is one step of an agent run, nested under the step that called it. The marked step ran long and is flagged.
01 · Capture
Record each step of every run with versions, sources, tool calls and outcomes, redacting at the source.
02 · Trace
Connect model calls, tools, sub-agents and side effects under one trace.
03 · Audit
Write accountable events to an append-only, linked trail separate from debug logs.
04 · Watch
Track quality, safety and cost signals and alert on changes in behaviour.
05 · Explain
Replay runs, investigate incidents and produce evidence and reports for people.
The goal is to understand and correct agent behaviour. Capture what you need to explain decisions, protect the people in the data, and keep nothing you cannot justify.
01 / What to capture
Record the whole run, not just the reply.
An agent run is a chain of decisions: what it was asked, what it read, what it planned, which tools it called and what came back. If you only log the final answer, you cannot explain how it got there.
The minimum record
Request, user and tenant, agent and prompt version, model, retrieved sources, each tool call with arguments and result, approvals, final output, cost and latency.
Decide what not to keep
Classify fields by sensitivity. Hash, redact or drop secrets and personal data at capture time rather than trying to clean logs later.
A refund run records 14 events: request, two document reads, a policy lookup, a proposed refund, the approval, the execution and the reply sent.
Pick any completed run and reconstruct every step from records alone, without asking an engineer.
02 / Tracing
Connect every step with one trace.
Agents call models, tools, sub-agents and other services. A trace ties all of those steps together under one identifier, so a single request can be followed end to end.
Spans for each step
Model calls, retrieval, tool calls, sub-agent runs and approvals each become a span with timing, inputs, outputs and status, nested under their parent.
Open standards
Use OpenTelemetry and its generative AI conventions where possible, so traces flow into the tools your team already uses rather than a closed silo.
A slow customer reply traces to one sub-agent retrying a CRM lookup five times. The waterfall shows it in seconds.
Every external side effect carries the trace ID, and traces cross service and sub-agent boundaries without breaking.
03 / Audit trails
An audit trail is evidence, not a log file.
Debug logs help engineers. Audit trails answer accountable questions: who did what, on whose authority, using which information, and what changed as a result.
New records lock onto the end of the chain, each linked to the one before. Nothing in the chain can be edited.
Append-only records
Write audit events to storage that cannot be edited or deleted by the agent or ordinary operators. Link each record to the previous one so gaps and tampering are detectable.
Actor, authority, effect
Each event names the human user, the agent identity, the approval it relied on, the policy version applied and the before-and-after state of what changed.
A disputed supplier payment traces to the agent proposal, the approver, the approval digest, the policy version and the bank confirmation.
An auditor with read-only access can answer who, what, when and why for any action, and an integrity check reports no gaps.
04 / Privacy in logs
Logs are a data store too.
Traces collect prompts, documents and tool results, which often include personal or confidential data. Treat observability data with the same care as the systems it describes.
Redact at the source
Mask identifiers, card numbers, health details and secrets before they leave the service. Keep a reference to the source record instead of a copy.
Access and retention
Separate who can read traces from who can read audit records. Set retention per data class and delete on schedule, with deletion itself logged.
Support traces store a customer ID and ticket link, not the full conversation. Audit records are kept for seven years; debug traces for thirty days.
A sample of traces is scanned regularly for leaked secrets or personal data, and access reviews cover observability tools.
05 / Metrics
Watch the signals that matter.
Dashboards should answer whether the agent is doing its job well, safely and affordably right now, not just whether the servers are up.
Quality and safety
Task completion, escalation rate, human override rate, policy denials, guardrail hits and user corrections, per workflow and per version.
Cost and performance
Latency per step, tokens and cost per task, tool error rates and retries. Track trends against the release baseline.
Override rate on one workflow climbs from 3% to 9% the day after a prompt change. The dashboard ties it to the new version.
Every metric has an owner, an expected range and a link from the chart to example traces.
06 / Alerting
Alert on behaviour, not just errors.
Agents rarely crash when they go wrong. They keep running and do the wrong thing confidently. Alerts must catch changes in behaviour as well as failures.
Three signals stream in. When one leaves its normal range, an alert is raised.
Behavioural alerts
Sudden changes in tool usage, unusual destinations, spikes in denials or overrides, loops, runaway cost and actions outside normal hours.
Actionable routing
Each alert goes to an owner with the affected traces, a runbook and a one-click pause for the workflow.
An agent starts calling the export tool forty times an hour instead of twice. The alert pauses the workflow and pages security.
Alerts are tested with injected incidents, and time from alert to pause is measured.
07 / Replay & investigation
Be able to rerun what happened.
When something goes wrong, the fastest path to a fix is replaying the exact run with the same inputs, versions and tool results, then changing one thing at a time.
The recorded run sits above; the replay runs below. Where they part, the difference is marked.
Capture for replay
Store versions of prompt, model, tools and policy with each run, plus the tool results the agent saw. Replays use recorded results, never live side effects.
From incident to test
Every investigated failure becomes a regression case in the evaluation set, so the same mistake is caught before the next release.
A wrong refund is replayed against the previous prompt version, which handles it correctly. The diff identifies the instruction that changed.
A replay of a recent run reproduces the original output, and replays cannot trigger real actions.
08 / Reporting
Turn records into answers for people.
Leaders, auditors and regulators do not read traces. They need periodic, plain-language evidence that agents are operating inside agreed limits.
Regular reports
Volumes, autonomy levels used, approvals and overrides, incidents and their fixes, policy changes and evaluation results, per workflow.
Evidence on request
Generate the full record for one customer, one decision or one time window quickly, with integrity checks attached.
A monthly report shows 12,400 agent actions, 380 approvals, 14 overrides and one incident with its root cause and fix.
A request for all agent actions affecting one customer is answered completely within an agreed time.
Worked example / investigations
Answer the hard question.
Three illustrative questions show how traces and audit records answer them. Reference examples, not records from a live system.
- Question
- A customer asks why they received 40 USD instead of 60 USD.
- Records used
- Trace for the run, retrieved policy version, agent proposal, approval record.
- Finding
- The agent applied the previous returns policy, cached before an update.
- Action
- Clear the cache on policy change, add a regression case, and send the customer the difference with an explanation.
# Illustrative audit event, not a production schema
event_id: evt_01J9...
prev_hash: sha256:4be1... # links to previous event
trace_id: tr_7c2...
time: 2026-10-09T08:14:22Z
actor:
user: finance.approver@company
agent: ap_agent@v12
action: release_payment
target: invoice/INV-2210
authority:
approval: apr_88f... (exact_action_digest, single_use)
policy: payments_policy@v7
before: { status: approved_for_payment }
after: { status: paid, amount: 4200.00 USD }
data_refs: [po/PO-8812, invoice/INV-2210] # references, not copies
hash: sha256:91ac...If the audit trail cannot be written, sensitive actions should not run. An action that leaves no evidence is an action nobody can stand behind.
Before production
Ten questions before you rely on it.
Use these when reviewing an agent before production. Attach a sample trace and audit record to each answer.
0 / 10 reviewed in this session
Further reading
Go deeper.
This guide draws on open observability standards and AI governance guidance. Record-keeping obligations vary by sector and jurisdiction; check what applies to you.
Technical guidance, not a certification or a claim that these practices are already in place at your organisation.
See everything
your agents do.
Bring an agent or a plan for one. We can help design its tracing, audit trail, dashboards and alerts.