How do you run a call calibration session?

What is a calibration session?

SQM Group describes call calibration as a meeting where QA evaluators, supervisors and agents "listen to the same recorded call and rate it using their quality assurance form", then discuss their ratings (SQM Group, 26 Jan 2023). The goal is that a rep gets the same score whoever reviews their call.

Why do graders drift apart?

Because questions leave room for judgement, and people fill it differently. Over time, each grader develops their own habits. Call Centre Helper warns of a sneakier version: some analysts score the call they know is being calibrated one way, and everyday calls another (Call Centre Helper, 7 Jun 2017). It suggests tracking each analyst's scores over time against the team's average to spot that.

Who should be in the room?

Everyone who scores calls, and the people who coach from the scores. An ICMI article recommends including supervisors and leadership, not just the QA team (ICMI, 22 Jun 2017). Call Centre Helper suggests one person leads and makes the final call when the group can't agree (Call Centre Helper, 7 Jun 2017).

How do you run it?

1. 1

   Pick 2–3 calls in advance

   Different reps, lengths and outcomes. ICMI suggests pre-selecting a mix and sending it out for scoring ahead of time.

2. 2

   Everyone scores alone first

   SQM: each participant evaluates the same calls on their own before the session. No discussion before scores are in.

3. 3

   Put the scores side by side

   Item by item, not just totals. Highlight every item where answers differ.

4. 4

   Discuss only the differences

   Play the moment in the call. Each person says what they heard and why they scored it that way.

5. 5

   Decide and write it down

   The leader settles it. Reword the question or add an example so the answer is obvious next time.

Call Centre Helper suggests keeping a written record of the agreed principles, which it calls a "Calibration Monitor" document (Call Centre Helper, 7 Jun 2017). New graders read it before they start.

How often should you calibrate?

SQM Group recommends at least monthly, preferably weekly, and whenever QA standards change. It sets the goal of keeping evaluators' scores within a 5% variance of each other (SQM Group, 26 Jan 2023).

How do you know it's working?

Track agreement from session to session. The simplest number is percent agreement: the share of items where graders gave the same answer. McHugh notes that many texts treat 80% as the minimum acceptable agreement, but warns that percent agreement doesn't account for agreement by chance; Cohen's kappa does (McHugh, Biochemia Medica, 2012).

Item	Session 1 agreement	Session 2 agreement	Change made
Need confirmed	55%	85%	Added: "the rep must repeat the need and get a yes"
Concern handled	70%	80%	Added an example of a partly handled concern
Specific next step	90%	95%	None

Should AI scores be calibrated too?

Yes. Treat the AI like one more grader: put its scores in the same side-by-side view, and when it disagrees, decide who was right the same way. See how to check AI call scores against your managers.