Abstract

Many tasks can involve analyzing data points and assigning unbound numerical scores or labels. However, without a well-defined standard or point of comparison, such data annotation can require significant cognitive effort from human judges. To mitigate this issue, I propose the use of comparative judgments. Comparative judgments involve pairs of data points and a comparative label instead of a direct one. This process would only ask the human judge to compare the pair of data points and decide which one should have a higher label, but would not ask for the exact value. Moreover, ranking or sorting based tasks with data points based on comparative labels can benefit from a data annotation or formatting framework that includes the pairwise relative order. A machine learning model trained on such data could learn from this data more efficiently to preserve or predict such relative order. Although there have been reinforcement learning based approaches that make use of pairwise learning, these approaches focus on a goal of policy optimization, not the cognitive burden on human judges. I propose a task-agnostic comparative framework as an alternative method for data formatting and annotation. I explore three different case studies or tasks where I apply this framework to investigate if a machine learning model trained on comparative judgments can perform as well as, if not better than, traditional approaches. I also conduct human subject experiments to explore how these findings translate into practical settings for human judges. The primary case study here is on agile story point estimation for software development. I also explore the tasks of art evaluations for aesthetic preferences and of image-caption rating. My experiments show the proposed frameworks with on comparative judgments outperforming traditional regression-like approaches, while leaving a lower cognitive burden on the human judges.

Publication Date

7-23-2026

Document Type

Dissertation

Student Type

Graduate

Degree Name

Computing and Information Sciences (Ph.D.)

Department, Program, or Center

Computing and Information Sciences Ph.D, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Zhe Yu

Advisor/Committee Member

Yiming Tang

Advisor/Committee Member

Ashique KhudaBukhsh

Campus

RIT – Main Campus

Download

Share

COinS