The reliability of clinical judgments in dermatology: a descriptive review on their role as a reference standard for technological diagnostic research

The expansion of dermatological mobile health (mHealth) and teledermatology has generated numerous diagnostic support tools, typically validated against traditional face-to-face assessments. Histopathological confirmation through skin biopsy is invasive, resource-intensive and, in many African settings, not readily available as a dermatological research reference standard. Yet, the reliability of clinical diagnoses used as such standards remains uncertain. The study objective was to assess the consistency (inter-rater reliability) of face-to-face clinical judgments among multiple dermatologists evaluating the same patient group. For this descriptive literature review, a comprehensive search of seven databases and information platforms was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guideline. In total, 1,700 studies were found after deduplication and screened using pre-set criteria. Eighteen studies were included: three compared clinical diagnoses directly and 15 assessed the reliability of dermatological scoring systems for disease severity and activity. All studies involved two or more dermatologists assessing the same patients. Reported agreement among raters ranged from 0.15 to 0.98 for studies evaluating clinical diagnosis and from 0.07 to 0.95 for studies assessing disease severity or activity scores, demonstrating wide variability in inter-rater reliability. In conclusion, clinical diagnosis alone, especially if made by one or a limited number of dermatologists, may not provide a sufficiently reliable reference standard for validating teledermatology, dermatological mHealth applications, or artificial intelligence (AI) tools. Strengthening evidence on the reliability of diagnostic reference standards is essential for evaluating emerging digital health innovations, particularly in regions with limited access to dermatologists and histopathology services. Future studies, with similarly experienced dermatological assessors, should have larger sample sizes, assess diagnostic agreement rather than disease severity outcomes, combine inter- and intra-rater assessments, and include diverse geographic contexts such as Africa, Asia and Latin America. The potential influence of assessors’ levels of experience on agreement should also be explored. A final recommendation is to come to a consensus about skin color classification.