Fleiss's kappa
Fleiss's kappa is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items. This contrasts with other kappas such as Cohen's kappa, which only work when assessing the agreement between two raters or the intra-rater reliability (for one appraiser versus themself).