The short version
AI doesn’t fix a broken process, it accelerates it, which means a GTM motion with inconsistent qualification, undefined ICP and messy CRM data gets those same problems faster and at higher volume once AI is layered on top.
Before introducing AI agents into research, scoring or outreach, the process underneath needs to be defined well enough that a new hire could follow it from a written playbook. If you can’t explain your qualification logic in five bullet points, an AI agent can’t apply it consistently either; it will just hide the inconsistency behind a confident-sounding output.
The readiness test before any model
Ask three people on your team to write down, separately, how they decide whether an account is worth pursuing. Compare the answers. If they differ materially, you have no process to automate. Reconcile into five bullets, agreed by the whole team, with examples of accounts that pass and fail. That document is your specification for any AI system.
Then test the data. Pick the five fields the AI will depend on, such as industry, headcount, region, owner and stage. Export them and count distinct values and blanks. If industry has 140 variants, or half of headcount is empty, fix that first. A day spent standardising picklists often beats a month of prompt tuning.
Shadow mode as a diagnostic
Run the AI alongside humans for two weeks without letting it act. For each account, record what the AI recommended and what the rep decided. Disagreements fall into three buckets: the AI is wrong, the rep is inconsistent, or the definition is unclear. Only the first is a model problem, and in my experience it is rarely the largest.
Review the disagreements in a weekly 30-minute session with the reps. Each resolution becomes a written rule or a data fix. This also builds trust: reps who helped define the rules defend the system, whereas reps handed a score from nowhere ignore it within weeks, regardless of its accuracy.
Scoping the first live segment
Go live on one narrow segment, for example mid-market software accounts in a single region, where data quality is best and the ICP is clearest. Set a stop condition in advance: if acceptance falls below a threshold or an error reaches a customer, switch back to manual for that segment. Make the rollback a button, not a project.
Expand one segment at a time and re-run the readiness checks on each. Every new segment brings new data quirks, such as different naming conventions or missing fields. Teams that expand everywhere at once lose the ability to tell which problem is causing which symptom, and end up blaming the model.
Worked example (illustrative)
A hypothetical team wants an AI account scorer. Before launch, it spends three weeks on definitions and data, then runs shadow mode on 400 accounts.
What changed: Agreement rose mostly because definitions and data were fixed, not because the model changed. The disagreement review made reps co-owners of the rules.
Do it: the steps
Tick them off as you go. Your progress stays on this device.
Traps to avoid
- ×
Buying an “AI SDR” to solve a problem that is really an undefined ICP.
- ×
Fixing disagreements by adding exceptions to the prompt. The prompt becomes a pile of patches that nobody can maintain, while the underlying definition stays unclear.
- ×
Skipping a named owner for the process. If nobody owns the playbook, it drifts, and the AI keeps applying last quarter’s rules.
What to track
- Shadow-mode agreement
- accounts where AI and human decisions match divided by total reviewed; aim for 85% or more before going live
- Field completeness for the five key fields
- non-blank, valid-value records divided by total; above 95% is a sound threshold
- Duplicate rate
- duplicate records divided by total accounts; under 5% before automating routing
Benchmarks are typical planning ranges, not guarantees. Calibrate to your own data.
Questions people ask
Can AI help clean up the mess itself?
Yes, for narrow jobs like normalising free-text industry values or spotting likely duplicates, with a human approving the mapping table. Use it as a cleaning assistant, not as a substitute for deciding what the standard values should be.
How do I convince leadership to delay the AI rollout?
Frame it as risk and speed. A two-to-three week readiness phase prevents a failed pilot that costs trust. Show a short plan: definitions, data fixes, shadow mode, then a single-segment launch, each with a measurable exit criterion and date.