What is the best way to interpret inter-scorer reliability (ISR)?

Abstract

Inter-scorer reliability for sleep studies typically use agreement for a measure of variability of sleep staging. This is easily compared between two scorers (with one as-gold-) using percent agreement, however this does not take into effect the differences in the-mis-scored' epochs (e.g. all mis-scored in the same way such as stage 2 for Stage 3). This requires a more stringent comparison between two scores which can be seen visually in the hypnogram as sleep stage transition differences between the scorer and-gold'. Aim: To use a combined measure of sleep stage transition and percent agreement for sleep stage concordance between two levels of scorers (in/experienced). Methods: A sleep study concordance was conducted with 200 epochs. 17 scorers took part (15 sleep technologists and 2 sleep consultants) with AASM 2012 guidelines. Experienced scorers were classed as scorers of >1 year (full-time equivalent) in PSG scoring. Sleep stage transitions were any change from one stage to another stage (W, N1, N2, N3, R). Percent agreement is calculated as number of epochs similar to a-gold' (determined by consensus agreement between three experienced scorers). Median and Mann-Whitney test compared experienced (N=9) with inexperienced group (N=8). Results:-Gold' had 16 sleep stage transitions, The experienced group median was 16 with difference from-gold' of 0 and inexperienced group median 10 with difference from-gold' of-6, which was statistically significant. Median (mean) percent agreement with-gold' for experienced scorers was 82.3% (79.5%) and inexperienced, 82.6% (84.8%), not statistically significant. Conclusion: Percent agreement is a general indicator of ISR for sleep staging. We have shown that despite level of experience, the percent agreement can be the same. Inexperienced scorers had significantly lower number of sleep stage transitions which may reflect imprecise application of AASM 2012 rules for scoring and identify more readily those that need training.L6246119832018-11-01 <br />