Inter-Coder Reliability for AI-Generated Interview Transcripts

By the YumiLM Team

Inter-coder reliability (also called inter-rater reliability, or IRR) is the check that tells you whether two or more coders are applying the same codebook consistently, rather than each reading their own meaning into it. It's usually reported as a number — percent agreement, Cohen's Kappa, or a related statistic — calculated on a subset of the corpus that multiple coders code independently. This guide covers how to run that check, what Cohen's Kappa actually corrects for that percent agreement doesn't, the "unitization problem" that has to be solved before any agreement statistic means anything, and — specific to research teams working from AI-generated transcripts — how transcript quality (verbatim vs. cleaned text, timestamps, speaker attribution) shapes whether a low reliability score reflects genuine interpretive disagreement or just inconsistent source material.

For the codebook-building and general coding workflow that inter-coder reliability sits inside, see How to Analyze Interview Transcripts — this guide goes deeper on the reliability check specifically and assumes a codebook already exists.

Key Takeaways

Factor	Why it matters for inter-coder reliability
Percent agreement	Simple and intuitive, but doesn't correct for chance — can look high even with a weak codebook
Cohen's Kappa	Corrects for chance agreement; values above roughly 0.60 are generally treated as acceptable, above 0.80 as strong
Unitization	Coders must agree on where one codable unit ends and the next begins before code-level agreement is even measurable
Verbatim vs. cleaned transcript	Coders working from differently-cleaned versions of the same recording are effectively coding different text
Timestamps	Let a disagreement be traced back to the exact recorded moment during adjudication, not just re-read from the transcript
Speaker attribution	If your unit is "one speaker turn," inconsistent or missing turn boundaries create disagreement before content is even coded

What inter-coder reliability is, and why it's reported
Percent agreement vs. Cohen's Kappa
The unitization problem
How transcript quality affects inter-coder reliability
Running an inter-coder reliability check, step by step
Where AI transcription helps, and where it doesn't
Reporting inter-coder reliability in a paper or dissertation
Common pitfalls
Privacy and confidentiality considerations
Checklist
Sources

What inter-coder reliability is, and why it's reported

Inter-coder reliability measures how consistently two or more independent coders apply the same codebook to the same data. In practice, it's calculated on a subset of the full corpus — commonly somewhere in the 10–20% range, though the right proportion depends on corpus size, code count, and how high-stakes the findings are — where multiple coders code the same material independently, without discussing their decisions first, and their codes are then compared statistically.

The point isn't just producing a number for a methods section. A codebook that looks precise on paper can still be applied inconsistently once real coders start using it on real, messy interview data — definitions that seemed clear can turn out to have edge cases nobody anticipated. Running an inter-coder reliability check surfaces those problems while there's still time to fix the codebook, rather than after the full corpus has already been coded around a definition two coders were quietly interpreting differently.

Percent agreement vs. Cohen's Kappa

Percent agreement is the simplest reliability statistic: the number of coding decisions both coders agreed on, divided by the total number of decisions, multiplied by 100. It's easy to compute and easy to explain, but it has a well-known weakness — it doesn't correct for agreement that would happen by chance alone. If a code is rare (say, applied to 5% of units) or a codebook has very few codes to choose from, two coders can rack up a high percent-agreement score largely by both defaulting to the majority code, without their judgment actually converging on anything.

Cohen's Kappa corrects for that: κ = (P₀ − Pₑ) / (1 − Pₑ), where P₀ is the observed proportion of agreement and Pₑ is the proportion of agreement expected by chance given how often each code was used. A widely cited interpretive scale, reproduced in McHugh's (2012) review of the statistic, treats values below 0 as no agreement, 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement — in practice, most qualitative methods sections treat Kappa above roughly 0.60 as acceptable and above 0.80 as strong.

One limitation worth knowing before you rely on Kappa alone: it can look surprisingly low even when raw agreement is high, if a code is either very rare or very common across the dataset — a pattern methodologists call the kappa paradox. O'Connor and Joffe's (2020) review of intercoder reliability debates in qualitative research covers this directly, and is worth reading before you report a single Kappa value as if it settles the question on its own — for a code with a skewed distribution, report percent agreement alongside Kappa and interpret both together, rather than either number in isolation.

The unitization problem

Before any agreement statistic means anything, coders have to be comparing decisions about the same stretch of text. That sounds obvious, but it's the step qualitative reliability guidance most often skips over — and it's specific to text-based coding in a way it isn't for, say, coding a fixed set of survey responses.

Campbell, Quincy, Osserman, and Pedersen's (2013) work on coding in-depth semistructured interviews names this directly as the "unitization problem": two coders reading the same transcript can genuinely agree on what a passage means and still disagree on where it starts and ends — one might code a whole multi-sentence answer as a single unit, another might split it into two or three. Standard agreement statistics assume the unit boundaries are already fixed and shared between coders; if they aren't, you're not really measuring code agreement, you're measuring a mix of code agreement and boundary disagreement, and the resulting Kappa understates how much the coders actually agree about meaning.

The practical fix Campbell et al. propose is to unitize the transcript before independent coding starts — agree on a consistent segmentation rule (by speaker turn, by sentence, by a fixed-length window, or by some other explicit convention) and apply it to the transcript first, so both coders are working from an identical set of pre-defined segments and the only thing left to disagree about is which code applies to each one. This is where transcript formatting stops being a cosmetic detail and starts directly affecting whether your reliability numbers mean what you think they mean.

How transcript quality affects inter-coder reliability

Verbatim vs. cleaned transcripts

If one coder is working from a verbatim transcript — fillers, false starts, and self-corrections preserved — and another is working from a version that's been cleaned up, they aren't looking at identical source text, even if it's nominally "the same interview." A self-correction like "I mean, actually — no, that's not right" reads very differently coded verbatim than it does once cleanup has smoothed it into a single confident sentence. Decide your transcription convention once, document it, and apply it consistently across every transcript both coders will see — see Verbatim Transcription Accuracy for Interviews for how to choose and apply that convention before coding starts, not partway through it.

Timestamps and traceability during adjudication

When two coders disagree on a unit, resolving it — through discussion, a calibration meeting, or a third reviewer — means going back to the exact moment in question. A timestamped transcript lets everyone involved jump straight to that point in the recording rather than re-reading and re-guessing which part of a long paragraph is actually under dispute. This matters more during reliability adjudication than during ordinary reading, because the whole point of the exercise is pinning down precisely where two independent readings diverged.

Speaker attribution and turn boundaries

Many unitization schemes use "one speaker turn" as the codable unit. If turn boundaries are inconsistent — two utterances merged into one, or one utterance split in two — coders are effectively working from different unit sets before a single code gets applied, which reintroduces the unitization problem described above. It's worth being direct about what this means for AI-generated transcripts specifically: YumiLM returns a transcript as sequential, timestamped text in the order it was spoken, without automatic speaker labels or turn segmentation. If your reliability design depends on speaker-turn units, budget time during transcript preparation to mark those turn boundaries yourself before independent coding begins — see Coding AI-Generated Interview Transcripts for the broader preparation checklist this fits into. Treat that preparation step as part of unitizing, not as separate from it.

Errors that change a coding decision, not just wording

A transcription error affects reliability differently than it affects a bare read-through. If only one coder happens to notice and mentally correct a misheard word — a dropped negation, a homophone substitution, a "fifteen" heard as "fifty" — while the other codes the error as written, their codes will diverge for a reason that has nothing to do with genuine interpretive disagreement. That kind of divergence gets folded into the same Kappa value as a real definitional disagreement, even though the fix is completely different: a definitional disagreement means revise the codebook; a transcript-fidelity disagreement means both coders need to be working from the same corrected text before the comparison is repeated.

Running an inter-coder reliability check, step by step

Finalize the codebook and the unitizing rule before selecting the reliability subset — changing either mid-check invalidates the comparison.
Select the reliability subset (commonly 10–20% of the corpus), ideally transcripts not used while the codebook was still being drafted, so agreement isn't inflated by coders' prior familiarity with those specific passages.
Prepare the subset transcripts identically — same verbatim/cleanup convention, same timestamps, same unit boundaries — for every coder involved.
Code independently, with no discussion between coders until both have finished coding the full subset.
Calculate percent agreement as a quick check, then Cohen's Kappa (or, with more than two coders or missing data, a statistic designed for that case, such as Krippendorff's alpha).
When a code's Kappa falls short, look at the actual disagreements before assuming it's interpretive: is it a codebook definition problem, a unitization/boundary problem, or a transcript-fidelity problem where coders were effectively working from different text?
Revise, recalibrate, and — if the codebook changed substantively — re-run the check on a fresh subset rather than treating the first run as final.
Document the whole process: subset size, per-code Kappa values, what was revised and why, and the date. This becomes part of your project's audit trail and is exactly what a peer reviewer or dissertation committee will ask to see.

Where AI transcription helps, and where it doesn't

Be specific about the boundary here, because it's narrower than it might sound. Starting from a shared, AI-generated transcript removes one real source of pre-analysis variation: if two coders had instead each transcribed the interview manually, their transcripts could already differ before coding even began. A single shared transcript — timestamped, exportable, and consistent across every coder who works from it — takes that variable off the table. YumiLM's Insight Guide can also give the coding team a quick structured overview (a premise, timestamped sections, highlighted moments) to orient everyone to a long interview before reliability work starts, the same way it's useful before ordinary first-cycle coding.

What it doesn't do: decide your unitizing rule, apply your codebook, judge whether two coders' decisions actually agree, calculate Kappa, or resolve a disagreement once one shows up. Those stay entirely the coding team's responsibility, same as with any transcript source.

One more honest caveat for multilingual teams: if coders on your team work in different languages, YumiLM's translation output can help everyone review a shared-language version of a transcript for orientation. But translation adds its own interpretive layer between the participant's words and the coded text — for in-vivo coding, or for any reliability check where exact phrasing matters, code from the original-language transcript, not a translated one.

Reporting inter-coder reliability in a paper or dissertation

Reviewers increasingly expect more than a bare "reliability was established" sentence. Report: which statistic you used and why (percent agreement alone is generally seen as insufficient on its own, per O'Connor and Joffe's review); the size and selection method of the reliability subset; the specific value(s) obtained, broken out by code if agreement varied meaningfully across the codebook rather than collapsed into a single overall number; how disagreements were resolved (discussion to consensus, a third arbitrating coder, majority rule); and whether the codebook was revised as a result, with dates. A single Kappa reported for an entire codebook can hide a codebook with three well-defined codes and one badly underspecified one — per-code reporting catches that in a way an overall average can't.

Common pitfalls

Running the reliability check only after all coding is finished. By then, a codebook problem the check would have caught is expensive to fix — every already-coded transcript needs revisiting.
Reporting percent agreement alone, without acknowledging that it doesn't correct for chance.
Selecting the reliability subset from transcripts the codebook was built on, which inflates agreement through familiarity rather than genuine consistency.
Treating one reliability run as final rather than as one iteration in a revise-and-recheck cycle.
Attributing every disagreement to "coder judgment" without first checking whether the actual cause was a unitization mismatch or a transcript-fidelity gap — the fix is different in each case, and lumping them together makes the codebook look worse than it is.

Privacy and confidentiality considerations

Running an inter-coder reliability check means more than one person now has direct access to the same raw transcript and, often, the underlying recording. Before sharing either with a second or third coder, confirm this is covered by the scope of consent participants actually gave, apply your institution's access-control and data-handling requirements to every coder on the team (not just the lead researcher), and anonymize identifying details before transcripts are shared beyond the immediate coding team. These are standard qualitative data-handling practices that apply regardless of how the transcript was produced.

Checklist

Codebook and unitizing rule finalized before the reliability subset is selected
Reliability subset (commonly 10–20%) drawn from transcripts not used to build the codebook
All subset transcripts prepared identically — same verbatim/cleanup convention, same timestamps, same unit boundaries
Coding done independently, with no discussion until both coders finish
Percent agreement and Cohen's Kappa (or an appropriate alternative) both calculated and reported
Disagreements traced to their actual cause — definitional, unitization, or transcript-fidelity — before revising the codebook
Reliability process (subset size, per-code values, revisions, dates) documented for the audit trail
Consent scope and access controls confirmed for every coder with transcript or recording access