Evaluating Enterprise Chatbot Platforms: An Engineer’s Due-Diligence Checklist
Procurement evaluates chatbot platforms on demos and price. Engineering inherits whatever gets signed.
This is the due-diligence checklist we use before a platform touches production — the questions that predict whether it survives contact with real content, real permissions, and real users.
Retrieval: The Part That Decides Everything
- Hybrid retrieval (vector + lexical) with reranking, or vector-only? Exact terms — SKUs, error codes, names — break pure-vector systems; see hybrid RAG
- Can it read your existing indexes (Solr, Elasticsearch) in place, or does it demand re-ingestion? Why that matters
- What is the chunking strategy, and can you inspect retrieved chunks per answer?
- Is there a confidence threshold with honest “I don’t know” behaviour — tunable, per pipeline? Relevance tuning is where quality lives
Security: Where Vendors Fail the Audit
- Are ACLs enforced inside retrieval — as index-level filters on every query — or checked at the application layer where a reranker can leak past them? The failure modes: secure RAG for private data
- SSO with group-to-role mapping enforced at login, so access follows your directory
- Where do prompts, retrieved content, and conversations go — and are they used for vendor model training?
Operations: The Six-Month Questions
- Re-indexing on content change — automatic, and how fast?
- Observability: per-answer traces, confidence distributions, unanswered-question clusters — built-in or your integration project? Our comparison: Langfuse, LangSmith, or built-in
- One pipeline serving every channel (web, WhatsApp, Slack, Teams), or per-channel configuration drift?
- Model portability — can the underlying LLM be swapped as the market moves?
Running the Evaluation
Insist on a pilot with your real content and your hundred most-asked questions — never the vendor’s demo corpus. Score answer rate, citation accuracy, and permission correctness, in that order. The buyer-side category map lives on IntelloWork’s blog: the best enterprise AI chatbot platforms, categories first — and yes, IntelloWork is the platform we built to pass this exact checklist, which is why the checklist exists.
If you are weighing building instead: build vs buy, honestly and what a build actually costs.
Scoring vendors without a spreadsheet fight
Most evaluation spreadsheets fail the same way: two hundred weighted rows, every vendor scores between seventy and eighty per cent, and the decision gets made in the corridor anyway. The fix is to stop scoring features and start scoring outcomes on a fixed test set. Four numbers are enough to separate real platforms from convincing demos.
- Answer accuracy. Percentage of your hundred test questions answered correctly, judged by the person who owns the content, not by the vendor.
- Groundedness. Percentage of answers where every factual claim is supported by a cited passage you can open. This is the number that predicts whether users keep trusting the assistant in month three.
- Correct refusal rate. Of the questions your corpus genuinely cannot answer, how many produced an honest “I don’t know” rather than a plausible invention. Vendors rarely volunteer this metric, which is exactly why it discriminates.
- Permission leakage. Binary. Any answer that surfaces content the test user was not entitled to see is a disqualification, not a deduction.
Run the same hundred questions against every shortlisted vendor. It takes a day per vendor and replaces weeks of argument. The full method, including how to build the question set, is in our comparison of enterprise AI chatbot platforms.
Ten questions that separate a platform from a wrapper
Ask these in the technical call. The quality of the hesitation tells you as much as the answer.
- Is retrieval hybrid, and can I see the keyword and vector scores for a given query?
- Are permissions evaluated inside the retrieval query, or applied to results afterwards?
- How long after a document’s permissions change does the assistant stop using it?
- Can I export the evaluation set and the results, or do they live only in your dashboard?
- What happens when retrieval returns nothing relevant? Show me.
- Which region processes inference, and is that in the contract?
- Can I bring my own model keys or point at a self-hosted endpoint?
- Who is on the sub-processor list, and what notice do I get when it changes?
- Can your staff read my tenant’s content, under what circumstances, and is it logged?
- If I leave, what do I take with me — connectors, chunking configuration, evaluation set, nothing?
Red flags in a vendor demo
Demos are optimised environments. A few patterns reliably indicate that the optimisation is doing heavy lifting.
- The corpus is theirs, not yours. Insist on a demo against a sample of your own documents, including the badly formatted ones. Politely decline a demo on a curated sample.
- Citations link to a document rather than a passage. A link to a ninety-page PDF is a gesture at provenance, not provenance. See what counts as a real citation.
- The assistant never says it does not know. Ask it something the corpus cannot answer, in front of them. Watch what happens.
- Permissions are described as “role-based” without further detail. Ask which system is the source of truth for entitlements and how often it is consulted. Our note on role-based access in AI chatbots covers the distinctions that matter.
- No answer on inference region. If the account executive cannot say where inference runs, the security review will discover it later and less pleasantly.
What the reference call should actually cover
Vendors supply references who are happy. That is fine; ask them operational questions rather than satisfaction questions. How long between signature and first useful answer? What broke in month three? How much content clean-up did you do, and did you expect it? What does your monthly bill look like against the original estimate? Who maintains the evaluation set, and does anyone still look at it? How did you handle the first bad answer that reached a customer?
The last question is the most informative. Every deployment has one. What you are listening for is whether the organisation had a process, or whether it was a scramble.
Frequently asked questions
How long should an enterprise chatbot evaluation take?
Four weeks is enough and more than four is usually procrastination. Week one builds the corpus and the hundred-question test set, week two connects the shortlisted platforms, week three scores answers, week four runs the permission tests and reference calls.
How many questions should the evaluation set contain?
One hundred is the practical minimum for a signal you can trust, drawn from real tickets or search logs rather than invented. Include at least ten that the corpus cannot answer, and at least ten that require combining two documents. Keep the set after selection — it becomes your regression suite.
Should we run a paid pilot or a free trial?
A short paid pilot usually produces better engineering support and a more honest picture than an unsupported free trial, provided the contract lets you walk away. What matters more than the commercial framing is that the pilot runs against your real content and your real permissions.
What is the most common evaluation mistake?
Testing the chat interface instead of the retrieval layer. Interfaces are easy to build and easy to like. Retrieval quality, permission fidelity and refusal behaviour are what determine whether the system is still trusted six months later, and none of them are visible in a demo unless you deliberately test for them. The security dimension is covered in depth in our guide to enterprise AI chatbot security.
Written by Tarun Gupta, founder and AI architect at Exuverse and the architect behind Intellowork. Last reviewed 31 August 2026.