Abstract

As artificial intelligence (AI) becomes increasingly common in computational social science, \textit{inconsistency} has emerged as a key challenge. AI models often contradict themselves when given equivalent inputs, disagree with other models on the same data, and diverge from human judgments in seemingly opaque ways. Human annotators exhibit their own inconsistencies, both within individuals and across groups shaped by differing values and identities. Rather than treating these inconsistencies simply as noise, this dissertation argues that they contain meaningful signals that can be leveraged to improve learning efficiency, strengthen evaluation, and increase the reliability of large-scale social measurement. To study this phenomenon, we first develop a taxonomy of five forms of inconsistency: \textit{intra-model}, \textit{inter-model}, \textit{intra-annotator}, \textit{inter-annotator}, and \textit{human-AI}. Each of these is examined through empirical studies on real-world datasets from various social science domains. First, in intra-model inconsistency, we introduce \textit{inconsistency sampling}, an active learning strategy that treats a model's own contradictions as informative training signals, and apply this method to natural language inference-based analysis of cable news coverage of policing. Second, to study inter-model inconsistency, we conduct a noise audit of nine offensive speech classifiers on more than three million social media comments. We also use \textit{model inconsistency sampling} to analyze police coverage in local news across 10 U.S. cities, and in Indian parliamentary discourse on education. In intra-annotator inconsistency, we introduce \texttt{ARTICLE}, a framework that estimates annotator reliability through self-consistency rather than majority agreement, allowing minority perspectives to be preserved. To investigate inter-annotator inconsistency, we introduce the concept of \textit{vicarious offense}, a paradigm that reveals how political groups perceive, and often misperceive, what offends members of the other groups. Finally, in human-AI inconsistency, we show that generative AI systems and humans differ in how they weigh situational factors when making social decisions that entail moral responsibility and immediate personal risk. Our findings highlight the importance of inconsistency-aware methods for the responsible use of generative AI in socially consequential domains.

Publication Date

4-2026

Document Type

Dissertation

Student Type

Graduate

Degree Name

Computing and Information Sciences (Ph.D.)

Department, Program, or Center

Computing and Information Sciences Ph.D, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Ashiqur R. KhudaBukhsh

Advisor/Committee Member

Christopher M. Homan

Advisor/Committee Member

Daniel S. Nagin

Campus

RIT – Main Campus

Download

Share

COinS