Solved – Limitations of Cohen’s kappa for sparse data

Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, lots of values < 5).

Cohens Kappa is known to have limitations for skewed datasets.

Quoting an example from here

Consider following matrix :

+---+----+ | 1 | 6  | +---+----+ | 9 | 84 | +---+----+ 

The above example has an observed agreement of 0.85 but the Cohen' Kappa is 0.04.

The solution suggested in this article is to report two separate agreement metrics for positive and negative classes.

Similar Posts:

Rate this post

Leave a Comment