CoderDrift: Design-aware temporal reliability auditing for human annotation studies
SOFTWAREX, cilt.35, 2026 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 35
- Basım Tarihi: 2026
- Doi Numarası: 10.1016/j.softx.2026.103010
- Dergi Adı: SOFTWAREX
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Compendex, INSPEC, Directory of Open Access Journals
- Ondokuz Mayıs Üniversitesi Adresli: Evet
Özet
Aggregate reliability can conceal changes across annotation order. CoderDrift is an open-source Python package for design-aware temporal audits of nominal and ordinal multi-rater data. It separates peer-relative and fallible-reference evidence and now provides prospective, design-specific calibration with immutable locked validation. Its family false-alert bounds are conditional on a declared simulation generator, not distribution-free guarantees. Without retuning the three built-in profiles, a 27-scenario robustness matrix produced support-qualified detection from 0-0.10 for weak abrupt changes and 1.00 for strong abrupt changes; combined stress produced 0-0.03. On the preserved MultiPref assignment graph, semi-synthetic abrupt-change recall was 0.74-0.98, gradual-change recall was 0.80, and two-change support-qualified recall was 0.40; 0 of 50 no-injection families alerted. MultiPref and WebCrowd25K are retained as diagnostic-sensitive observational audits, not external ground-truth validation. CoderDrift neither diagnoses causes nor establishes coder deterioration.