70% of ticket triage handled by agents
Retrieval over the firm's own documentation, with escalation rules the operations lead controls directly — and an evaluation harness proving accuracy before anything reached a client.
Where they were
A 400-person services firm was fielding 6,000 internal support requests a month, most of them answerable from documentation nobody could find.
An earlier off-the-shelf chatbot had been switched off after two weeks because it confidently invented policy.
The operations lead was, reasonably, skeptical that a second attempt would go better.
What we did
01Built the eval set before the agent
800 real historical questions with correct answers, graded by the firm's own subject-matter experts. Nothing shipped until it beat the bar on that set — which also gave the ops lead a number to hold us to.
02Grounded every answer with a citation
The agent quotes the source document and links to the paragraph. When it can't find grounding, it says so and escalates rather than improvising.
03Gave the ops lead the controls
Confidence thresholds, topic-level enable/disable, and escalation routing all live in an admin surface. No deploy needed to turn off a category that's behaving badly.
04Rolled out one department at a time
IT first, then HR, then finance. Each expansion required passing the eval bar on that department's own question set.
“The difference from our first attempt was that this one tells you when it doesn't know. That single behavior is why people trust it.”
What it was built with
Let’s talk about yours.
Mention this case study in your note and we’ll come to the call with the specifics of how it went — including what we’d do differently.