Abstract

High-quality data is fundamental to the production of reliable official statistics and evidence-based decision-making. While deterministic data quality assessment frameworks provide transparent, consistent, and reproducible evaluations, they are limited in accommodating the intended statistical use of data and identifying complex quality issues beyond predefined validation rules. As administrative datasets continue to increase in volume, diversity, and complexity, there is a growing need for governance-oriented approaches that enhance existing quality assessment practices without compromising transparency, accountability, or consistency. This research proposes and evaluates an AI-assisted multi-layer enhancement of the DDI, a governance-oriented data quality assessment framework developed by SCAD. The proposed framework preserves deterministic assessment as its foundation while progressively integrating fitness-for-purpose contextual weighting, AI-assisted anomaly detection using the Isolation Forest algorithm, and AI-generated data quality justification through a Large Language Model (LLM). The objective is to investigate whether these complementary layers provide a more comprehensive, explainable, and decision-oriented assessment of data quality than deterministic assessment alone. A quantitative experimental research design was adopted using two publicly available UK Road Safety datasets comprising Accident and Vehicle information. Approximately 400,000 records from each dataset were analysed under controlled experimental conditions. The proposed framework was evaluated across three Statistical Value (SV) domains—Social, Economic, and Agriculture and Environment—and compared directly with the existing deterministic DDI using identical datasets, preprocessing procedures, and quality measurements. The findings demonstrate that each assessment layer contributes complementary evidence to the overall evaluation of data quality. Fitness-for-purpose contextual weighting enabled quality assessments to reflect differing statistical priorities across SV domains, while AI-assisted anomaly detection identified hidden multivariate quality issues that were not detected through deterministic validation rules. Furthermore, AI-generated justification translated analytical outputs into structured, human-readable reports that enhanced the transparency and interpretability of assessment outcomes. Collectively, the proposed framework produced a more comprehensive and governance-oriented assessment while preserving the transparency, consistency, and accountability of the existing deterministic DDI. This research contributes to the fields of official statistics and data quality management by demonstrating that deterministic quality assessment and artificial intelligence (AI) are complementary rather than competing approaches. The proposed AI-assisted multi-layer DDI framework provides a practical and explainable methodology for integrating intelligent technologies into governance-oriented quality assessment, supporting more informed decision-making and establishing a foundation for future research and operational implementation within National Statistical Offices (NSOs) and other public-sector organisations.

Publication Date

8-1-2026

Document Type

Thesis

Student Type

Graduate

Degree Name

Professional Studies (MS)

Department, Program, or Center

Graduate Programs & Research

Advisor

Zainab Al-Zanbouri

Comments

This thesis has been embargoed. The full-text will be available on or around 8/4/2027.

Campus

RIT Dubai

Download

Share

COinS