A New Method for Estimating the True Score of Subjects in the Assessment Based on the Binary Checklist

Abstract

Assessments of subjects by multiple raters are completed in several health research and practice arenas. As observed in interrater
comparisons of checklists during assessment, raters generally make mistakes in assessment. Despite these errors, the average
scores on all the checklists have been historically used to represent the subjects’ true scores. The effect of the assessment
rater error on these average scores has been seldom studied. In the present study, the effect of random rater error on subjects’
average scores was investigated mathematically. The analysis based on a mathematical model revealed that average scores were
truly representative only in special cases where raters made no assessment errors. In addition, average scores were poor estimates
of the true score due to random assessment error. A new, more precise formula to estimate subjects’ true scores was proposed.
The method to apply this formula to actual checklist assessment data was demonstrated.

Comments Off on A New Method for Estimating the True Score of Subjects in the Assessment Based on the Binary Checklist

Tags: ,

UA-25380860-1