Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, lots of values < 5).
Contents
hide
Best Answer
Cohens Kappa is known to have limitations for skewed datasets.
Quoting an example from here
Consider following matrix :
+---+----+ | 1 | 6 | +---+----+ | 9 | 84 | +---+----+ The above example has an observed agreement of 0.85 but the Cohen' Kappa is 0.04.
The solution suggested in this article is to report two separate agreement metrics for positive and negative classes.