How to automatically score customer service calls

How does automatic call scoring work?

An AI reads a transcript of the call and answers each question on your scorecard: did the agent do this, and how well? You get a score for each item and for the call, instead of a score for the few calls a manager had time to hear.

Other tools describe it the same way. Zendesk says its AutoQA scores every conversation against categories you describe in plain language, updated 21 Sep 2026, and evaluagent lists automated scoring of every conversation and custom scorecards, as of 26 Oct 2026 (both read 26 October 2026). The scorecard is where the quality comes from, so it is the first step below.

Step 1: What should I put in the checklist?

List the things you can point to in a recording. For a service team that is usually the same few moments on every call: how the agent opened, whether they found out what the caller needed, whether they gave a clear answer, and how they ended.

Write each item as one thing you could point to in a transcript. "Confirmed the caller's name before discussing the account" is easy to judge. "Was professional" is harder, so split it into what professional looks like on your team.

Use a yes/no item for things that either happened or didn't. Use a score on one scale, for example 1 to 10, for things that come in degrees. If an item only applies to one direction of call, mark it inbound only or outbound only.

Here is a made-up scorecard for an inbound service line. It is an invented example, not a real team's checklist.

Item	Type	Applies to
Opened with the company name and their own name	Yes/no	All calls
Confirmed who they were speaking to before discussing the account	Yes/no	Inbound only
Asked what the caller needed and repeated it back	Scale, 1 to 10	All calls
Gave a clear answer or a next step	Scale, 1 to 10	All calls
Told the caller what would happen next and by when	Yes/no	All calls
Closed by asking if there was anything else	Yes/no	All calls

Flag	Severity
Promised a refund or credit the agent is not allowed to give	High
Required disclosure not given	Medium

Keep the list to what you will act on. A Callyst scorecard holds 1 to 20 items and up to 12 flags, so there is room, and a shorter list is quicker to read.

Step 2: How do I turn the checklist into a scorecard?

You don't need to write code. Paste your checklist, or upload it as a PDF, DOCX or TXT file up to 2 MB (up to 20,000 pasted characters), into the free scorecard builder. It needs no sign-up, and it drafts a scorecard for you to read. The checklist text goes to our AI model provider to write the draft, so leave real customer names and details out of it.

Read the draft before you use it. An AI wrote it from your checklist, so check that each item says what you meant. The draft is for reading, so if something is off, change the checklist and build again. You can build up to 3 drafts per network per day.

Checklist points or weights don't carry over. Callyst uses your items and your wording, and judges the overall score for the whole call. If one thing must happen on every call, make it a yes/no item or a flag.

Prefer to write the file yourself? You can send a scorecard in Callyst's JSON format (up to 200 KB). The scorecard guide has the format and an example.

Step 3: How do I get recordings to Callyst?

Callyst scores recordings, so export them from your phone system. It works with any phone system that can export recordings. You don't need to send transcripts, because Callyst transcribes the audio for you.

For the free check, upload up to 10 calls as .wav or .mp3 files, 50 MB each. Our guide to uploading call recordings covers exports.

The steps:

1. Open /free-check. Enter your name, work email and company. Gmail, Outlook and other free addresses can't be used, and each company gets one free check.
2. On the same form, add your scorecard (a JSON file, or a checklist we turn into a draft; the checklist text goes to our AI model provider to do this). Add it before you press Get my free check: the upload page that follows has no scorecard field. If you add nothing, Callyst uses its standard sales scorecard.
3. We email you a sign-in link. Click it to confirm your work email and open your private upload page. The link works once and expires after 15 minutes.
4. Upload your calls and tick We have permission to share these calls. Tick it only if you have the right to record these calls and share them with Callyst, including any consent that call-recording laws require. That part is your responsibility.
5. When every file is uploaded, press Done, build my report. Your scored report arrives within 24 hours.

Voicemails, unanswered calls, duplicates, calls over 90 minutes and, by default, calls under 60 seconds are set aside and not scored. Audio is deleted 30 days after scoring, and no recording is kept longer than 35 days.

Step 4: How should I review the AI scoring?

Read it the way you would read a colleague's QA notes. Look at four things.

• Item scores and quotes. Item scores come with quotes from the call. Each quote is checked word for word against the transcript. If no quote checks out, the score shows without one.
• The overall score. Each call gets an overall score on the same scale, with a short written reason. It is judged for the whole call, not added up from the items, so read it next to the items.
• N/A items. Callyst marks an item N/A when it doesn't apply to the call or there was nothing to judge. N/A items have no score and are left out of the averages. A call that isn't a real two-way conversation, such as one where only one side is heard, gets every item marked N/A and no overall score.
• Flags. Flags are a separate list for a person to review, and raising one doesn't change the call's score. Each flag the AI raises comes with a quote that matches the call word for word. Add your compliance checks to the scorecard and Callyst flags the calls that miss them, with the quote. A flag whose quote isn't in the transcript is dropped.

Then check it against your own ear. Pick a handful of calls, listen, and see whether you would have scored them the same way. If an item keeps landing somewhere you wouldn't, the fix is usually in how the item is written. Reword it before you score more calls.

What might review notes look like?

This is a made-up illustration of the kind of notes a reviewer might write while reading a scored call. It is not the product's screen layout, and it is not from a real call or a real customer.

Fictional exchange: Agent: "Before I go on, can I confirm I'm speaking with the account holder?" Caller: "Yes, this is Sam Rivera, the account holder." Agent: "Thank you, Sam. Let me pull up your account."

Item (fictional)	Result	Reason
Confirmed who they were speaking to before discussing the account	Yes	The agent asked, the caller confirmed, and only then did the agent open the account.
Told the caller what would happen next and by when	No	The call ended without a time for the next step.

A reviewer reads each reason and, where there is a quote, checks it against what was said. The sample report shows how a real report is laid out.

Step 5: How do I use the results?

Use them to decide who to coach and on what.

• Start with flags. Read them first, listen to the calls they point to, and decide what happens next.
• Look for patterns. An item that scores low across many calls is usually a training topic or a script problem, not one agent's problem.
• Coach on specifics. Pick one call, show the agent the item that scored low and the reason beside it, and agree one thing to try on the next call.
• Tighten the scorecard. After a round of review, reword the items that were hard to judge before you score more calls.

What does it cost to keep going?

The free check covers up to 10 calls. After that, Callyst is $200 a month for 1 to 3 reps, $280 for 4 to 7 reps, then $40 per rep from 8 reps. Each rep brings 1,000 calls a month into a pool shared by the team, and extra calls are 5 cents each. There is no setup fee, billing is monthly and there is no annual contract. The free check doesn't commit you to buy. See pricing.

Good to know

Callyst scores calls in English. Scores, flags and coaching notes are generated by AI and can be wrong. They are an aid, not a decision. Review them before acting, especially before any step that affects a person's job or pay.

Ready to try it?

Run a free check on up to 10 of your calls with your own scorecard, or see what you'll get in the sample report.