Evaluating Enterprise Chatbot Platforms: An Engineer’s Due-Diligence Checklist
Procurement evaluates chatbot platforms on demos and price. Engineering inherits whatever gets signed.
This is the due-diligence checklist we use before a platform touches production — the questions that predict whether it survives contact with real content, real permissions, and real users.
Retrieval: The Part That Decides Everything
- Hybrid retrieval (vector + lexical) with reranking, or vector-only? Exact terms — SKUs, error codes, names — break pure-vector systems; see hybrid RAG
- Can it read your existing indexes (Solr, Elasticsearch) in place, or does it demand re-ingestion? Why that matters
- What is the chunking strategy, and can you inspect retrieved chunks per answer?
- Is there a confidence threshold with honest “I don’t know” behaviour — tunable, per pipeline? Relevance tuning is where quality lives
Security: Where Vendors Fail the Audit
- Are ACLs enforced inside retrieval — as index-level filters on every query — or checked at the application layer where a reranker can leak past them? The failure modes: secure RAG for private data
- SSO with group-to-role mapping enforced at login, so access follows your directory
- Where do prompts, retrieved content, and conversations go — and are they used for vendor model training?
Operations: The Six-Month Questions
- Re-indexing on content change — automatic, and how fast?
- Observability: per-answer traces, confidence distributions, unanswered-question clusters — built-in or your integration project? Our comparison: Langfuse, LangSmith, or built-in
- One pipeline serving every channel (web, WhatsApp, Slack, Teams), or per-channel configuration drift?
- Model portability — can the underlying LLM be swapped as the market moves?
Running the Evaluation
Insist on a pilot with your real content and your hundred most-asked questions — never the vendor’s demo corpus. Score answer rate, citation accuracy, and permission correctness, in that order. The buyer-side category map lives on IntelloWork’s blog: the best enterprise AI chatbot platforms, categories first — and yes, IntelloWork is the platform we built to pass this exact checklist, which is why the checklist exists.
If you are weighing building instead: build vs buy, honestly and what a build actually costs.