Operations and service assurance
An AI assistant in production is a business service built from a model, a retrieval pipeline, tool servers and the operator’s own systems. Most of it fails the way any distributed service fails, and conventional monitoring catches that. One part fails silently: the answers get worse while every health check stays green. Assurance has to cover both.
- User and channelhealth check green
Care agent or customer asks a question
Infrastructure and application- Latency: time to first token, end to end
AI tracing and evaluation- Sampled production answers reviewed
- Assistant / agenthealth check green
Plans the run, calls tools, writes the answer
Infrastructure and application- Agent runs: completed, failed, abandoned
- Steps per run
AI tracing and evaluation- Trace of each step
- Loops that never finish
- Model gateway and modelhealth check green
Routes calls to a pinned model version
Infrastructure and application- Token consumption and cost
- Rate limits, quotas, gateway throughput
AI tracing and evaluation- Model version changes
- Regression set scored after every change
- Retrieval and vector storehealth check green
Finds the passages the answer should rest on
Infrastructure and application- Ingestion job failures
- Vector store load
AI tracing and evaluation- Empty results, low relevance
- Citations to superseded documents
- Documents indexed but garbled
- MCP / tool servershealth check green
The agent’s hands
Infrastructure and application- Up / down
- Synthetic call through the tool
AI tracing and evaluation- Tool-call log: unusual tool calls, injection attempts
- Operator systemshealth check green
APIs behind the tools, e.g. TMF621, TMF622
Infrastructure and application- Downstream API errors
AI tracing and evaluation- Bulk reads, data leaving approved destinations
The answers get worse while every health check stays green. Only the evaluation lane sees it.
- AvailabilityOK
- ErrorsOK
- LatencyOK
- CapacityOK
Illustrative shape, not data. A stale document, a provider-side model update or a garbled index each look like this.
What to monitor
| Signal | Why it matters | Conventional monitoring enough? |
|---|---|---|
| Agent runs | Completed, failed, abandoned; steps per run; loops that never finish | Partly. Counts yes; understanding why a run went wrong needs a trace of each step |
| MCP server availability | Tool servers are the agent’s hands; when one is down the agent cannot act | Yes for up/down. A synthetic call through the tool is needed to prove it actually works |
| Downstream API failures | Errors from the operator APIs behind the tools (for example TMF621 trouble tickets, TMF622 product ordering) | Yes. These are ordinary API metrics |
| Retrieval quality | Empty results, low relevance, citations to superseded documents | No. Needs AI-specific tracing of what was retrieved for each question |
| Ingestion failures | Documents that failed to parse or index, permission sync lag | Yes for job failures; no for documents that indexed but came out garbled |
| Answer quality and evaluation drift | A regression set scored after every change, and sampled production answers reviewed | No. Needs evaluation tooling and people who know the domain |
| Model version changes | A provider-side update can change behaviour with no change on your side | No. Pin versions and re-run evaluation when one changes |
| Latency | Time to first token and end-to-end; agents with many tool calls add up | Yes, with per-step breakdown from traces |
| Token consumption and cost | Cost scales with usage, context size and agent step count | Partly. Metering is ordinary; attributing cost to a journey needs traces |
| Capacity | Provider rate limits and quotas, gateway throughput, vector store load | Yes |
| Security incidents | Injection attempts, unusual tool calls, bulk reads, data leaving approved destinations | Partly. Security monitoring needs the tool-call log to see agent behaviour |
In practice this means two layers. Infrastructure and application monitoring (tools such as Grafana, Datadog, Dynatrace or Splunk are examples of the category) covers availability, errors, latency and capacity. AI tracing and evaluation tools, open-source or commercial, record each run step by step (prompt, retrieval, tool calls, output) and score answers against test sets. The IT service management platform (ServiceNow is one example) is where alerts become incidents. The layers must share a run identifier, or nobody can follow a complaint from the ticket to the trace.