Key findings

• Correlate every signal around one end-to-end workflow run.

• Record model, prompt, retrieval, tool, policy, and deployment versions.

• Separate metadata observability from sensitive content capture.

• Alert on actionable symptoms and burn rate, not every individual model error.

• Join technical telemetry to sampled quality evaluations and business outcomes.

Understand the four observability layers

Logs are timestamped records of events such as a policy denial or failed tool call. Traces connect spans across the full execution path and show where time and failure accumulated. Metrics aggregate behavior over time, such as request rate, latency, error rate, token use, and review percentage. Evaluations assess whether the output or trajectory was useful, grounded, safe, and compliant.

No single layer is sufficient. A 200 response can contain a wrong answer; a correct answer can arrive after the customer’s service-level target; a cheap model call can trigger an expensive loop; and a high aggregate quality score can hide a serious failure in one case type. Observability should answer what happened, why, how often, to whom, at what cost, and whether the business outcome was acceptable.

Define a trace for the complete AI workflow

Start a trace at the business trigger, not only at the model API. Create spans for input validation, retrieval, model calls, policy checks, tool selection, authorization, tool execution, human approval, downstream writes, retries, and fallback. Propagate trace context across queues, webhooks, services, and integrations. W3C Trace Context standardizes headers for carrying request identity between distributed components.

Separate diagnostic context from metric dimensions. Use a bounded, reviewed set of workflow, environment, model, outcome, and error-category values for metric labels. A tenant pseudonym is not automatically low-cardinality or anonymous; do not attach tenant, request, conversation, or trace identifiers to every metric series. Keep only necessary, access-controlled correlation context in traces or logs. Record durations, token counts, and retries as measurements, not identifier-like labels.

Use structured, correlated logs

Emit machine-readable events with timestamp, severity, service, trace ID, span ID, workflow run ID, actor or workload identity, event type, result, and safe context. Define a small event taxonomy for request accepted, model completed, tool proposed, authorization allowed or denied, approval granted or rejected, write committed, fallback used, and incident flag. Consistent fields make investigation and alerting reliable.

OpenTelemetry supports correlation by carrying trace and span identifiers in log records. Keep audit events distinct from debugging logs and protect them against unauthorized alteration. Establish retention by purpose. Security and decision records may require longer retention than detailed debugging content, while raw content may need no storage at all. Test that investigators can move from an alert to the relevant trace and audit event.

Protect prompt, response, and tool content

Full content is valuable for debugging and quality review but can contain personal data, credentials, confidential documents, or malicious payloads. Capture metadata by default and make content capture an explicit, risk-assessed feature. Redact or tokenize identifiers, exclude secrets, restrict access, encrypt storage, apply short retention, and audit viewing. Sampling should follow a documented purpose and lawful basis where personal data is involved.

OpenTelemetry’s generative-AI conventions can represent model operations, token use, finish reasons, messages, and tool calls, but the project notes that content recording is opt-in. Treat telemetry as another production data system subject to minimization and deletion. Never assume observability vendors are outside the data flow; include them in architecture, subprocessor, location, and access reviews.

Review the export, not just the dashboard

Before enabling an instrumentation package, define a field allowlist for each exported signal. For every field, record its purpose, value constraints, destination, access owner, and retention rule. Review a synthetic success, error, retry, and rejected tool call against that contract. Check the emitted telemetry after processors have run; a tidy dashboard can hide sensitive fields that still reached the backend.

For a hypothetical support workflow, a metric may use an enumerated outcome such as completed or needs-review. The free-text ticket subject, email address, full URL, model answer, and tool arguments do not belong in that metric. A trace identifier can help investigate one execution, but using it as a metric label creates a separate attribute combination for each trace. Use the backend’s supported trace-correlation mechanism, such as exemplars, instead of copying identifiers into all series.

Pin the instrumentation and convention versions used by the deployment. OpenTelemetry’s GenAI conventions are under development and now live in a dedicated repository; review changes before upgrading. Keep instructions, inputs, and outputs off by default, and assess any content-recording or external-storage option separately. Hashing an identifier alone does not prove anonymity. A package upgrade should trigger a new synthetic export check, not an assumption that the previous field review still applies.

Measure reliability, cost, and quality

Build a dashboard with request rate, end-to-end latency percentiles, success and fallback rate, tool errors, provider errors, retries, queue age, approval time, token use, cost per completed case, budget consumption, and duplicate or partial writes. Segment by workflow version, model, tool, and important case type. Use service-level indicators that reflect user outcomes, not only infrastructure uptime.

Add quality through sampled human review, delayed ground truth, customer correction, or automated graders that have been validated for the task. Track serious errors separately. Monitor changes against a baseline and release threshold. Quality labels often arrive later than technical telemetry, so design a join key from production run to eventual outcome without exposing unnecessary identity.

Create alerts that lead to action

Alert on sustained user-facing error or latency, exhausted budget, repeated tool denial, unusual data access, permission change, high retry or loop depth, queue growth, sudden token inflation, missing traces, failed audit logging, quality regression, and serious incident events. Route each alert to an owner with severity, context, runbook, and expected response time.

Use rates and windows to control noise. A single routine model refusal may be normal; a rising refusal rate after a prompt release is actionable. Security-sensitive events such as credential exposure or unauthorized writes may need immediate paging. Define warning, pause, and automatic containment thresholds. Test alerts with synthetic failures and verify that notification, trace links, kill switch, and fallback all work.

Operate incidents and continuous improvement

An incident view should reconstruct the trigger, versions, inputs in controlled form, retrieved sources, model and tool sequence, authorization, approval, downstream effects, and related runs. Preserve evidence according to policy, contain access, stop or roll back the workflow, notify required parties, and identify both technical and process causes. Avoid spreading sensitive content through tickets and chat during investigation.

After resolution, add the failure to the evaluation set, improve the relevant control, and verify the fix in staging and production. Track recurring error categories and control effectiveness. Google Cloud’s current agent-observability guidance similarly emphasizes prompts and responses, tool usage, latency, token use, security and safety, and quality evaluation. The tooling can vary; the operating questions should remain portable.

Start with a minimum viable stack

An SMB can begin with OpenTelemetry instrumentation, one collector, an existing log and trace backend, a metrics dashboard, and the team’s current incident channel. Instrument one critical workflow end to end before collecting everything. Standard names and portable export reduce dependence on one backend and make vendor changes less disruptive.

The minimum production gate is simple: every run is identifiable; every external action is attributable; versions are known; errors, latency, token use, and cost are measurable; serious policy events alert an owner; sensitive content is controlled; and the team can disable and investigate the workflow. Expand sampling and quality evaluation as the use case grows.

Sources and methodology

This article synthesizes the primary sources below as of the publication date. Forecasts and recommendations are directional scenarios, not guarantees; they should be tested against your workflow, data, risk tolerance, and current vendor documentation.

OpenTelemetry: OpenTelemetry Documentation (accessed 2026-07-30)

OpenTelemetry: OpenTelemetry Logging Specification (accessed 2026-07-30)

OpenTelemetry: Generative AI observability with OpenTelemetry (accessed 2026-07-30)

World Wide Web Consortium: W3C Trace Context (accessed 2026-07-30)

Google Cloud: Agent observability (accessed 2026-07-30)

OWASP Foundation: OWASP AI Agent Security Cheat Sheet (accessed 2026-07-30)

OpenTelemetry: handling sensitive data

OpenTelemetry: metric cardinality and trace correlation

OpenTelemetry GenAI span conventions

Next: use the AI Automation Incident Response Plan to turn alerts and traces into a controlled first-hour response.