Inter-Rater Agreement: Read a Two-Reviewer Call Table

By Trellus Team

Published November 3, 2026

Two reviewers agree on 17 of 20 sales calls. That is 85% observed agreement. Before treating it as a reassuring result, look at which ratings matched, what happened to the calls they couldn't assess, and whether they were applying the same rule.

Inter-rater agreement describes how often different reviewers give the same item the same rating. This guide works through one binary criterion with two reviewers. It also calculates unweighted Cohen's kappa, which compares observed agreement with a baseline calculated from each reviewer's category proportions. Neither number tells you whether the reviewers were right about the calls.

Give both reviewers one question

Suppose a manager wants to check this criterion: “Did the buyer agree to a specific next action during the reviewed segment?” The team needs examples of what counts as agreement, what counts as a proposal without agreement, and when the available conversation is insufficient to judge.

Use permitted recordings and give both reviewers the same relevant segment and surrounding context. Each person records an initial rating before seeing the other's answer. Keep those initial ratings after the discussion. A jointly agreed answer is useful for the call record, but it can't replace the independent answers when you calculate agreement.

For this fictional exercise, there are two substantive categories: criterion met and criterion not met. “Unable to assess” is recorded separately. The latter could mean the recording ends before the buyer answers; it doesn't mean the buyer declined. If a criterion doesn't apply to a call, record that separately too and explain the inclusion rule.

The existing calibration-call guide has a worksheet for discussing the scoring rule. The calculation here answers a narrower question: how did two people's independent ratings line up on the same set of calls?

Keep the incomplete pairs visible

Imagine 22 calls were selected for review. Both reviewers can assign one of the two substantive categories on 20 calls. On call 21, reviewer A cannot assess the buyer's answer. On call 22, reviewer B cannot assess it. Those two calls stay in the review log with their reasons, outside the binary agreement calculation.

The paired rating coverage is therefore 20 / 22, or about 90.9%. Report that beside the agreement result. Removing a difficult recording from a calculation doesn't make the missing evidence disappear. This example describes the 20 assessable pairs; it doesn't establish how the reviewers would have rated the other two calls.

If you regularly encounter missing ratings, inspect that pattern before comparing review rounds. A round with mostly easy, complete recordings may produce a different result from one with interrupted calls or unclear audio. Call-outcome completeness covers the related job of keeping unknown outcomes visible.

Build the table from the initial ratings

Here are the fictional counts for the 20 complete pairs. Reviewer A's ratings are the rows; reviewer B's are the columns. Each call appears in exactly one of the four inner cells.

Download the 22 fictional initial ratings as a CSV to rebuild the table. The paired-binary column identifies the 20 calls used below; the two incomplete pairs keep their recorded ratings and reasons.

They agree on seven “met” ratings and ten “not met” ratings. The other three calls are disagreements: two where A said met and B said not met, and one in the opposite direction. All four cells sum to 20, not 22.

Observed agreement = (7 + 10) / 20 = 0.85, or 85%.

Start with this table even if a reporting tool offers a coefficient immediately. It shows the direction of disagreement and how often each person used each category. A single percentage hides both.

Calculate Cohen's kappa without losing the counts

For two reviewers who rate the same items in categorical form, unweighted Cohen's kappa uses this calculation:

Kappa = (observed agreement − expected agreement) / (1 − expected agreement).

The expected agreement term comes from the reviewers' marginal category proportions. In this example A used met on 9 / 20 calls and not met on 11 / 20. B used met on 8 / 20 and not met on 12 / 20.

The baseline is (9 / 20 × 8 / 20) + (11 / 20 × 12 / 20) = 0.18 + 0.33 = 0.51. Then kappa is (0.85 − 0.51) / (1 − 0.51) = 0.34 / 0.49, or approximately 0.694.

That baseline is a mathematical comparison using the two category distributions. It isn't an observation that the reviewers guessed on 51% of calls. Kappa of 0.694 is also not “69.4% accurate.” Accuracy would require an appropriate reference judgment and a different comparison.

The formula and two-reviewer design are described in the published methodological paper Computing Inter-Rater Reliability for Observational Data. This sales example uses its own fictional counts. For a flag evaluated against reference labels, use the separate precision and recall worked example.

Open the three disagreements

The calculation gives the manager three calls to discuss. It doesn't say which reviewer should change an answer.

On call 8, the rep suggested Tuesday and the buyer asked for an email with options. A marked the criterion met; B marked it not met. They should check whether the rule requires an accepted action or merely a proposal. The buyer may have agreed to receive an email without agreeing to a Tuesday meeting. The specific action matters.

On call 9, A treated “send it over” as agreement to a follow-up conversation. B heard agreement only to receive material. Revisit the surrounding words and record what the buyer accepted. A broad “next step” label can hide that distinction.

On call 10, B noticed an explicit agreement near the start of the supplied segment that A overlooked. They should check that moment together. The response might be a corrected rating, a clearer example in the rule, or a change to how reviewers find the relevant passage. Preserve the original ratings and record the discussion's result separately.

After changing a rule, try it on another suitable set of calls independently. Re-rating the same three discussed examples can show that people remember the discussion; it offers little evidence about how they will handle a new case.

Read the category mix before comparing scores

Consider another fictional set where both reviewers assign “not met” to every call. Observed agreement is 100%. Both marginal proportions are also entirely in that category, so expected agreement is 1. The kappa denominator is zero: the coefficient is undefined here, not automatically 1.

Even without that extreme case, category proportions affect kappa. Keep the four-cell table, criterion and sample selection beside any comparison. A different mix of calls can change the coefficient. Don't call a score increase a coaching improvement when the recordings, rule or reviewers also changed.

Agreeing with each other also leaves room for a shared mistake. Two reviewers might consistently treat a proposed meeting as booked because the scorecard tells them to. A high agreement result won't discover that problem by itself. Check whether the criterion describes the behavior the team actually needs to know.

Save a review note someone else can reproduce

A useful note for the worked example would read:

Two reviewers independently rated the same next-action criterion on 22 selected calls. Twenty had complete binary rating pairs; two had an unable-to-assess rating. On the 20 complete pairs, counts were 7 met/met, 2 met/not met, 1 not met/met and 10 not met/not met. Observed agreement was 85%; unweighted Cohen's kappa was approximately 0.694. The three disagreements need discussion. The calls were selected for this exercise, so these figures aren't an estimate of agreement across the team's entire call library.

For your own review, save the call-selection rule and period, criterion version, reviewer names, initial ratings, missing-rating reasons, four counts and calculation. Keep call references in the team's permitted review location rather than copying prospect information into a widely shared worksheet.

This calculation covers one categorical criterion and exactly two reviewers. A total score, ordered rating scale or larger reviewer group needs a method suited to that design. Choose that method before reducing a scorecard to one convenient number.