We shipped a contract review system for a law firm in January. The model achieved 99.2% accuracy on clause extraction — a number we were genuinely proud of. In the first week of deployment, usage was lower than expected. We ran user interviews and discovered the problem immediately: the lawyers did not trust it.
Not because it was wrong. Because it did not tell them when it might be wrong.
Confidence Is Not the Same as Certainty
This is the foundational insight for AI interface design. AI systems have variable confidence across different inputs. A contract clause that closely matches training data will be classified with high confidence. An unusual clause — one the model has seen few examples of — will be classified with lower confidence. The accuracy number is an average across both.
Users know this intuitively, even if they cannot articulate it. When an AI gives them a confident answer on everything, they learn quickly that the confidence is not meaningful, and they start treating all outputs with blanket skepticism. This is rational behavior. It is also a UX failure.
The fix is to surface confidence explicitly. Not as a raw probability — "73.4% confident" is not actionable — but as a meaningful signal that guides user behavior. "High confidence," "Review recommended," "Uncertain — human review required" are interfaces. The underlying probability score is infrastructure.
Error Handling Is a Feature
Every AI system will be wrong sometimes. How it handles being wrong determines whether users continue to trust it after the first error.
The worst pattern: the system gives a confident, wrong answer and provides no mechanism for the user to flag the error, correct it, or understand why the mistake happened. The user loses trust, tells their colleagues, and the system gets a reputation it will never recover from.
The best pattern: the system clearly communicates when it is uncertain, provides a friction-free way for users to correct errors, and treats corrections as feedback that improves future performance. Users who can visibly see their corrections improving the system develop trust proportional to the improvement. They become advocates rather than skeptics.
Human Override Must Be Obvious
A cardinal rule: users must never feel trapped by an AI recommendation. If the path to overriding or ignoring the system's output is unclear or requires effort, users will feel coerced — and coercion is antithetical to trust.
In the legal system, we made override a first-class action. Every AI-generated clause classification had an "Override" button at the same visual weight as the "Accept" button. Not buried in a menu, not requiring a confirmation dialog, not labeled in a way that implied the user was doing something unusual. Accepting and overriding were equal options. Override rates were used as a signal for model improvement, not as a metric to minimize.
Explanation Should Match User Mental Models
Explainability is frequently implemented as showing users the retrieved context or highlighting tokens with high attribution scores. This is technically correct and often practically useless.
Lawyers do not think in terms of attention weights. They think in terms of precedent and risk. An explanation that says "this clause was classified as high-risk because it resembles these 3 clauses from prior contracts that were flagged by the firm" is useful. An explanation that highlights the words with high cosine similarity to the training centroid is not.
Match your explanations to how your users actually reason about the problem, not how the model internally represents it.
The Trust Flywheel
When these pieces are in place — calibrated confidence signals, clear error handling, visible override, user-relevant explanations — something interesting happens. Users trust the system enough to use it. Using it generates corrections. Corrections improve accuracy. Better accuracy generates more trust. The flywheel spins.
The opposite also happens. When users do not trust the system enough to use it seriously, you do not get the corrections that would improve it, accuracy stagnates, and the system never earns the trust it needed to get off the ground. Trust is not a post-launch problem — it is a product requirement.