A monitor that records only tool status and success can show that a call completed; it cannot show whether the arguments were correct or the result supported the answer. Other monitors and evaluators can inspect content. A request-linked trace adds context about which step ran, what category of input it used, what result came back, and how the workflow proceeded.
That is the practical difference between AI monitoring and observability. Monitoring watches selected signals and alerts on conditions the team has defined. Observability lets an engineer follow related events across one execution and investigate behavior that aggregate metrics cannot explain. Neither replaces the other: monitoring finds a signal across traffic; a trace supports investigation of a particular execution.
Monitoring detects configured symptoms
Monitoring usually covers values that can be counted or timed: request volume, latency, provider errors, token use, queue depth, and tool-call failures. Teams can set alerts for known conditions such as timeouts or a sudden rise in failed requests.
Those signals are useful, but they answer a limited question: did something cross a condition we chose to measure? A successful request can still produce an incorrect answer. A tool can return a technically successful response that is irrelevant to the task. If the system reports success and latency stays within bounds, infrastructure metrics may remain normal.
Observability connects the steps
A trace relates spans and events to an execution. In OpenTelemetry, a span represents an operation; spans can be nested and carry attributes and timestamped events. Applied to an AI workflow, those links can show where a request was routed, which model or tool operation ran, whether state changed, and where the final response was assembled. OpenTelemetry's tracing API describes this span and event model.
Consider this illustrative scenario: a support assistant retrieves no useful records, but its final answer still looks plausible. A request counter and tool success rate may show no error. A trace that records the retrieval operation, a redacted or categorized query representation, the result count, and the workflow's next step can narrow the cause. It does not prove why the answer was wrong; the engineer may still need to inspect the retrieval configuration, evaluation results, or application code.
For a production agent, useful trace context often includes:
- workflow and operation names, model/provider identifiers, duration, and outcome;
- routing choice and state transitions, with stable references to relevant configuration;
- tool name, a safe representation of its arguments, response status, and result reference;
- retrieval source identifiers or counts where they explain context assembly;
- evaluation or human-review signals tied to the same execution when available.
The goal is to make an execution investigable, not to record every internal model computation. A trace can expose a sequence of operations and the evidence available to the application without collecting private chain-of-thought.
Treat trace content as sensitive data
Prompts, retrieved documents, tool arguments, and outputs can contain customer data, credentials, or other sensitive material. OpenTelemetry's GenAI semantic conventions flag several content-bearing attributes as potentially sensitive. A standard field name does not make its value safe to export.
Before recording content, decide what question the trace must answer. Prefer operation names, status, counts, identifiers, and references over raw payloads. If text is necessary, filter or redact it before export and keep the smallest useful excerpt. Apply tenant-aware access controls, retention limits, and deletion behavior to traces as deliberately as to application data. Check that query tools, dashboards, exports, and incident attachments preserve the same tenant boundary.
A trace should also state its limits. Sampling, dropped spans, redaction, or retention expiry can leave gaps. A trace that omits payloads may narrow the investigation without establishing the cause; a trace that includes them can increase privacy and access risk. Choose instrumentation for the diagnostic question and the data boundary together.
Test whether traces answer the incident question
Start from a failure mode the system can safely reproduce in a test environment. For example, configure a retrieval test where a known query yields no relevant records. Verify that the trace links the request to the retrieval operation, records the result signal needed to distinguish empty from successful retrieval, and shows what the workflow did next. Then verify that a reader with the intended role can access the trace, while another tenant cannot, and that retention follows the configured limit.
This test checks whether the evidence supports an explanation of the workflow. It does not establish that every future failure will be diagnosable or that the system is reliable. Keep output-quality evaluation and operational alerts alongside traces; they answer related but different questions.
For a practical tracing example, see LLM observability with LangSmith, Prometheus, and Grafana. For the decision to escalate a trace finding into a wider review, see What Agent Observability Should Trigger a Production Audit. Those examples complement the distinction here: monitoring identifies signals, while linked execution evidence gives an investigation a path to follow.