Almost any AI feature demos well. The interesting question is what happens in month three, when the novelty is gone and someone is accountable for a wrong answer.
In our experience the features that stay switched on share four properties, and none of them are about the model.
1. It can say it does not know
A system that always answers will eventually answer wrongly with full confidence. Retrieval confidence thresholds, explicit refusal paths and routing to a human are not fallbacks — they are the feature.
2. Every answer is traceable
Citations are not decoration. When a user can see which document an answer came from, verification takes two seconds instead of two minutes, and trust survives the first mistake.
3. There is an evaluation suite
Prompts are code. Changing one without a regression suite over real historical cases is deploying untested code to production, and it will silently degrade the cases you were not thinking about.
- Build the eval set from real historical inputs, not synthetic ones.
- Include the cases where refusal is the correct answer.
- Run it in CI. A prompt change that drops the score does not merge.
4. Cost and latency are monitored like any other resource
Token spend behaves like a database query nobody indexed: fine in testing, alarming at volume. Track cost per resolved task from day one, not per request.
And keep provider access behind an interface. Model pricing and capability move quickly enough that switching should be a configuration change.
We wrote this because we had to solve it. See what we built or tell us what you're working on.