LLM Observability for Internal AI Assistants: Langfuse, LangSmith, or Built-In?
The week after an internal AI assistant launches, engineering gets a new pager duty: “what did it just tell someone, and why?”
Answering that is LLM observability. Here is the engineering view of the three ways to get it — and which one fits which team.
What You Actually Need to See
- Traces — for each answer: the query, the retrieved chunks and their scores, the prompt assembled, the model response, the citations attached
- Quality signals — confidence distribution, unanswered-question clusters, thumbs-down rate
- Safety signals — retrieval-time ACL decisions, so you can prove who could surface what
- Freshness — per-source index age; stale indexes produce confidently outdated answers
Option 1: Dedicated LLM Observability (Langfuse, LangSmith, Phoenix)
If you built a custom stack — your own RAG orchestration, your own pipelines — these are the right tools: trace-level visibility, eval harnesses, prompt versioning. The costs are real, though: instrumentation is your code to write, dashboards are your dashboards to watch, and none of these tools knows your ACL model unless you teach it.
Option 2: Your Existing APM
Datadog and friends see latency, errors, and token spend — necessary operational hygiene, blind to answer quality. An assistant confidently citing a stale policy is a healthy service in APM terms. Keep it, but do not call it observability for this system.
Option 3: A Platform That Ships It Built-In
If the assistant runs on a platform that controls ingestion, retrieval, and delivery, observability can be structural instead of bolted on: every answer citation-traced, confidence-scored, permission-checked by design, with the query analytics and per-source health in the product. That is the approach we took with IntelloWork — its content/retrieval/surface planes each report health independently. The buyer’s checklist for this is monitoring internal AI assistants.
The Decision Rule
Custom stack → dedicated LLM observability, budgeted as a first-class workstream, not an afterthought. Platform stack → demand built-in traces, confidence metrics, and ACL audit in the demo, and keep APM for the infrastructure underneath. Either way, decide before launch — retrofitting observability onto a system leadership already distrusts is the expensive order of operations.
Related engineering guides: secure RAG for private data · AI-driven relevance tuning · build vs buy: enterprise AI chatbot.
The four traces that make a bad answer explainable
Observability for an internal assistant is not general application monitoring with a language model bolted on. When a colleague forwards a screenshot of a wrong answer, you need to reconstruct four things, and if any of them was not recorded at the time you cannot reconstruct it later.
- The resolved query. What the retrieval layer actually searched for after rewriting, expansion or conversation-history stuffing. This is where a surprising share of bad answers originate, and it is almost never what the user typed.
- The candidate set with scores. Which chunks came back, from which documents, with keyword and vector scores and post-rerank position. Without this you cannot tell a retrieval failure from a generation failure, and those have completely different fixes.
- The assembled context. Exactly what text reached the model, including truncation. Context that was retrieved but silently dropped at the token limit is a classic and invisible cause of confidently incomplete answers.
- The entitlement decision. Which permission filter was applied and on whose behalf. This is the trace your security team will ask for, and it is the one most teams forget to log.
Metrics worth a dashboard
Token counts and latency percentiles are table stakes. The metrics that actually drive decisions are narrower.
- Groundedness rate. The share of answers whose claims are supported by retrieved passages, sampled and scored continuously rather than measured once at launch.
- Retrieval miss rate. Queries where no chunk cleared the relevance floor. A rising miss rate is usually a content gap, not a model problem, and it tells your documentation team what to write next.
- Refusal rate, split by correctness. Refusing when the corpus has no answer is healthy. Refusing when it does is a retrieval regression wearing a polite mask.
- Cost per resolved question. Not cost per thousand tokens. Cost per question that a user did not have to escalate.
- Time to first token. Perceived responsiveness is dominated by this, not by total generation time. Streaming turns a four-second answer into an acceptable one.
Sampling, retention and the privacy problem
Full-fidelity tracing of every request is expensive and, for an internal assistant, a compliance liability. Traces contain user questions and retrieved document text, which means they contain personal data and confidential content. Three practices keep this sane: sample aggressively for routine traffic and capture at full fidelity only for errors, low-confidence answers and thumbs-down feedback; redact or hash user identifiers in the trace store while keeping a reversible mapping under separate access control; and set a retention period on traces that is shorter than your general log retention, because nobody debugs a four-month-old answer.
All three interact with the obligations set out in our guide to enterprise AI chatbot security and compliance, in particular the requirement that erasure requests reach every store holding personal data. A trace database is such a store, and it is routinely forgotten.
Wiring observability into the release cycle
Observability that is only consulted after complaints is a post-mortem tool. What makes it pay for itself is running your evaluation set through the same instrumented path on every change — a prompt edit, a chunking change, a model upgrade, a reranker swap — and diffing the results. Groundedness and correct-refusal rate become release gates rather than opinions. That habit is what turns “the new model feels better” into a number, and it is the difference between an assistant that improves and one that drifts.
The same discipline underpins the vendor tests in our enterprise chatbot evaluation checklist, and it is one of the axes we use when comparing enterprise AI chatbot platforms.
Frequently asked questions
What is LLM observability?
LLM observability is the practice of recording enough about each request to a language model application — the resolved query, the retrieved candidates and their scores, the assembled context, the permission decision, the generated output and user feedback — that any individual answer can be explained and any change in quality can be attributed to a cause.
Is a dedicated LLM observability tool necessary, or is APM enough?
An existing APM handles latency, errors and cost well and will not handle retrieval candidates, groundedness scoring or evaluation runs without significant custom work. If your assistant is a small internal tool, instrumenting your APM is defensible. If retrieval quality is something you actively tune, a purpose-built tool pays for itself in the first regression you catch before users do.
How much traffic should be traced at full fidelity?
A small percentage of routine traffic plus one hundred per cent of errors, low-confidence responses and negative feedback. That combination keeps cost and privacy exposure bounded while ensuring the requests you will actually be asked about are always captured.
Do trace stores create a compliance obligation?
Yes. Traces typically contain user questions and document text, which means personal data and confidential content. They need a defined retention period, access control, and inclusion in your erasure workflow. Treat the trace store as a first-class data store, not as debug output.
Written by Tarun Gupta, founder and AI architect at Exuverse and the architect behind Intellowork. Last reviewed 31 August 2026.