TY - JOUR
T1 - Structured Assessment of AI-Assisted Systematic Reviews: Workflow and Rubric Insights
AU - Krzysztof Wiśniewski
AU - Małgorzata Nowak
AU - Tomasz Adamczyk
JF - Journal of Trustworthy and Explainable Intelligent Systems
JO - J Trustworthy Explain Intell Syst
Y1 - 2022
VL - 1
IS - 2
SP - 22
N2 - The application of Artificial Intelligence (AI) tools to assist with systematic literature reviews (SLRs) is expanding rapidly, yet researchers still lack structured frameworks for assessing and incorporating these tools into their research practices. This paper introduces and implements a rubric-based framework designed to evaluate AI toolchains across essential SLR stages, such as literature searching, screening, and data synthesis. A mixed-methods study was executed with 63 postgraduate student researchers, who represent the early-stage investigators that serve as primary adopters of these digital workflows. The results demonstrate that workflows structured around the rubric enhanced the participants' perceived utility of AI utilities, provided organized criteria for comparative analysis, and promoted greater transparency during tool selection. Although formal indicators of inter-rater reliability (such as Cohen’s or Fleiss’ Kappa) were not calculated due to the fact that participants did not evaluate identical benchmark sets concurrently, the consistency of the rubric was qualitatively validated through pilot evaluations, iterative updates, and structured focus group discussions. These efforts confirm interpretative alignment among users while recognizing that future validation via independent assessors is required. This research contributes a validated assessment rubric, an operational example of an AI-augmented SLR workflow, and a discussion regarding the consequences for research training, study reproducibility, and cross-disciplinary translation.
UR - https://tipublications.com/y367740241
ER -