We built a lead scoring system for a client whose sales team had already been given one and had quietly stopped looking at it.
The first one was not broken. It produced scores, the scores were not obviously wrong, and nobody used them. That is the normal outcome, and it is worth understanding why, because it is not a modelling problem.
The number arrives with no argument attached
A salesperson looking at Lead score: 82 has been given a conclusion and no reasoning. They cannot tell whether 82 means this company looks like the last ten deals, or that somebody opened three emails, or that the model likes companies in a particular postcode for reasons nobody has examined.
So they do what any competent person does with an unexplained instruction from a system they did not build: they check it against their own judgement. And once you are checking every score by hand, the score has cost you time rather than saved it.
Trust is the constraint, not accuracy. A model that is right 70% of the time with a stated reason gets used. A model that is right 85% of the time with no reason gets ignored, and 85% of nothing is nothing.
Two things fix it, and neither is a better algorithm
Train it on their own history, not a generic model. A scoring model that has never seen your closed-won deals is pattern-matching against somebody else’s business. It cannot tell you why a lead looks good, because “good” was defined elsewhere. Trained on your own win history, the model has something specific to point at — this account resembles the ones you actually closed, and it can name the resemblance.
It also means the model inherits your real constraints. If you have never successfully sold to companies under twenty people, that shows up on its own. You do not have to encode it as a rule and then maintain the rule.
Make every score carry its reason. Not a feature-importance chart. One line a salesperson can read before a call: scored high — same industry and headcount band as four closed deals this year, and they posted an operations role last month.
That sentence does two jobs. It gives the rep something to open with, and it lets them disagree. When a rep says “that reason is wrong, that company is a reseller”, you have found a gap in the data — and you only ever hear it because the reason was visible.
Let the team overrule it, and log it
The instinct is to hide the override so the system looks authoritative. Do the opposite.
Make overriding easy, record it, and read the log monthly. Overrides are the highest-quality feedback you will get about the model, and they arrive free from people who already know the answer. A system nobody can argue with is a system nobody corrects.
Where AI genuinely helps here
Not in producing the number — a regression on your own history goes a long way. The AI earns its place in the messy parts: reading job postings, news and company descriptions, and turning unstructured text into a signal the score can use. That is language work, and it is the kind of work a model is actually good at.
And put a confidence threshold on it. When the model is unsure, it should say so and route to a person rather than guessing with authority. A system that admits uncertainty on 15% of rows is trusted on the other 85%. One that never admits it is trusted on none.
The honest caveat
Scoring only works if you have enough closed-won history to learn from. If you have closed eleven deals, you do not have a scoring problem — you have a volume problem, and a model trained on eleven examples will produce confident nonsense.
We will tell you when that is the case, which is not the same as telling you what you want to hear.
We run this on our own product — NameToProfile does ICP scoring on live data, so this is not theory. If you want to know whether it is worth doing in your business before committing to building it, that is what the AI Opportunity Audit is for: two weeks, fixed fee, and an honest list of what not to automate.