How do you check AI call scores against your managers?

Why check AI scores at all?

Because you'll act on them. If a score decides who gets coached on what, you need to know it matches how your team judges a call. A vendor's accuracy number, measured on someone else's calls and scorecard, doesn't tell you that.

There's research that AI judges can agree with people about as often as people agree with each other. Zheng and colleagues found that strong AI judges reached "over 80% agreement, the same level of agreement between humans" (Zheng et al., arXiv, Dec 2023 (v4)). But they were judging chatbot answers, not sales calls, so it's a reason to test, not a reason to skip the test.

How do you run the test?

1. 1

   Pick calls already scored

   Use calls your managers or QA team scored in the normal way, on your current scorecard. Aim for a real mix: good calls, bad calls, every rep.

2. 2

   Hide the scores

   Send only the recordings and the scorecard. The AI must not see the human scores.

3. 3

   Score them blind

   The AI scores every item on every call, with a quote as evidence for each answer.

4. 4

   Compare item by item

   Not just the total. Two cards can both total 70 and disagree on half the items.

5. 5

   Read the disagreements

   Listen to the calls where they differ. Decide who was right, and why.

What should you measure?

Start with percent agreement: the share of item scores where the AI and the manager gave the same answer. It's easy to explain. Its weakness, as McHugh explains, is that it doesn't account for agreement that would happen by chance (McHugh, Biochemia Medica, 2012). If 95% of calls pass the greeting item, two graders who always say "yes" will agree 95% of the time without listening.

Cohen's kappa corrects for that. It runs from −1 to 1, where 0 is the agreement you'd expect by chance and 1 is perfect agreement. McHugh notes that many texts treat 80% agreement as the minimum acceptable level, and argues that a kappa below 0.60 means the raters don't agree well enough to rely on (McHugh, Biochemia Medica, 2012).

What should you compare the result to?

Your managers' agreement with each other. If two of your managers scoring the same calls only agree 75% of the time, an AI that agrees with them 75% of the time is doing as well as a third manager. Run the human-to-human comparison on a few of the same calls first; it's the fairest bar.

Greeting and purpose96%

Open questions88%

Need confirmed61%

Concern handled79%

Specific next step92%

What do you do with the disagreements?

Most disagreements come from the question, not the grader. "Did the rep handle the objection well?" invites two answers. "Did the rep acknowledge the concern, answer it, and check it was resolved?" invites one. When an item scores low on agreement, rewrite it and test again.

What you see	Likely cause	Fix
Managers disagree with each other too	The question is vague	Rewrite it as a yes/no about something you can hear
AI is stricter than managers on one item	Managers give credit for effort	Agree on what counts, write it into the question
AI misses something said on the call	The transcript got it wrong	Check the transcript at that time stamp
Disagreements cluster on one rep or one call type	Accent, noise or an unusual call	Review those calls by hand and report it to your vendor

Transcription is worth checking directly. Speech recognition is not equally accurate for everyone: a study of five commercial systems found an average word error rate of 0.35 for Black speakers compared with 0.19 for white speakers (Koenecke et al., PNAS, Mar 2020). If one group of reps seems to score lower, check the transcripts before you coach.

Is a one-time check enough?

No. Scorecards change, call types change, and people drift. SQM Group recommends calibrating human graders at least monthly (SQM Group, 26 Jan 2023). Do the same for AI scores: each month, have a manager score a few calls blind and compare.

How does Callyst run this test?

We call it the Blind Match. You send 100 calls your QA team has already graded, we score them blind on your scorecard, and we show you where we agree and where we don't, item by item. It costs $750, credited if you sign an annual plan. We show you the result on your own calls rather than quote a number. If you'd like to start smaller, the free 10-call check gives you a scored report on 10 of your calls within 24 hours.