Krippendorff has a paper (free and official versions) describing an adaptation of his measure of reliability α to continuous data, such as text. Specifically, it calculates the reliability of both unitizing (i.e., how the text is split into coded and uncoded units) and coding (i.e., which codes are assigned to what units).
I'm wondering if this has been implemented, say, in R. In his paper, Krippendorff mentions that a "computer program for calculating these [adaptations of α] is currently being developed." However, I could find no mention of or link to such software, neither on Krippendorff's site nor elsewhere. The paper is detailed enough that it would allow me to implement it myself, but if it's already been done no sense in reinventing the wheel.
Thanks in advance.
Best Answer
There is also now this new paper which extends Krippendorff's work on unitizing:
https://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00227 ("The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment" by Yann Mathet, Antoine Widl̈ocher, Jean-Philippe Ḿetivier in Computational Linguistics Journal (2015))
The code is in Java: https://gamma.greyc.fr/
Followup: https://www.mitpressjournals.org/doi/full/10.1162/COLI_a_00296 ("The Agreement Measure γcat a Complement to γ Focused on Categorization of a Continuum" by Yann Mathet in Computational Linguistics Journal (2017))
There are certain weakness(es) of alpha that were addressed in these two papers. I'm still reading them, so I don't know what these weaknesses were and how they were addressed. I'll hopefully be able to say more when I'm done reading.
Similar Posts:
- Solved – Inter-rater reliability with many non-overlapping raters
- Solved – Inter-rater agreement for Likert scale
- Solved – nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)
- Solved – Determining the number of raters for inter-rater reliability
- Solved – Determining the number of raters for inter-rater reliability