Abstract

Human emotional communication involves subtle, low-intensity, and context-dependent expressions influenced by sociocultural and interpersonal factors. Their inherent ambiguity and variability create heterogeneous data distributions, making affective phenomena difficult to model in naturalistic settings where expressions differ from exaggerated affect behaviors in acted environments. Affective AI faces the fundamental challenge of balancing personalization, which captures individual nuances, with generalization, which ensures robustness of affective AI systems across individuals. Traditional centralized models emphasizing generalization often fail to capture fine-grained expressive heterogeneity, while fully personalized models improve system performance for individuals but lack transferability and robustness. The central task is thus to develop frameworks that balance personalization and generalization, establishing principled strategies for when to prioritize each, and designing models that maintain adaptability for individuals while preserving robustness across contexts. Despite advances in affective AI, systematic approaches for addressing personalization and generalization remain limited. Existing models, predominantly developed for generalization, underrepresent the subtle and low-intensity affective states that characterize natural interaction and provide limited scalable mechanisms for addressing heterogeneity in expressive and perceptual behaviors. Principled strategies for determining when to emphasize personalization or generalization are lacking, and current methods offer limited guidance for adaptation across a range of individuals, perspectives, and contexts. Moreover, interactive data label acquisition and federated active learning remain underexplored as mechanisms for examining how personalization and generalization interact under low-resource, subjective, and ambiguous conditions. Temporal affect-recognition models often overlook explicit interpersonal synchrony metrics as potential signals for personalized prediction and generalization. Addressing these gaps requires resources that capture subtle affective states in naturalistic settings and modeling frameworks that integrate individual, interpersonal, and shared information while balancing personalization with generalization. This dissertation addresses these challenges through multiple strands of work. First, it introduces a multimodal corpus of task-based dialogs capturing frustration and surprise, enabling the study of ambiguity and variability inherent in subtle emotional expressions and motivating the need for approaches that adaptively balance personalization and generalization. Second, it proposes FedSession, a personalized federated learning framework that incorporates individual modeling, session-level interpersonal dynamics, and global aggregation, achieving improved performance over standard personalized federated learning methods. Third, it develops an agreement-based taxonomy that leverages large language model-based agents, demonstrating how inter- and intra-agent agreement patterns can guide strategic decisions between personalization and generalization. Fourth, it shows that interactive machine learning paradigms can improve data efficiency and recognition performance in low-resource affective tasks, and it explores the effect of partner history and explicit synchrony for personalization and generalization to unseen speakers in dyadic interactions. Finally, it proposes FedTriage, a federated active learning framework for exploring and analyzing the personalization and generalization trade-off in perspectival contexts. Together, these contributions advance the study of personalization and generalization in affective AI by connecting data resources, personalized federated modeling, agreement-based decision frameworks, interactive machine learning, temporal dyadic affect modeling, and federated active learning. The dissertation shows that personalization can improve affective modeling under heterogeneous and low-resource conditions, while also suggesting a need for more advanced control mechanisms for preserving generalization. Collectively, this work contributes toward affective AI systems that are adaptive to individuals, robust across contexts, and attentive to the complexity, subjectivity, ambiguity, and privacy-sensitive nature of human emotional behavior.

Publication Date

8-2026

Document Type

Dissertation

Student Type

Graduate

Degree Name

Computing and Information Sciences (Ph.D.)

Department, Program, or Center

Computing and Information Sciences Ph.D, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Cecilia O. Alm

Advisor/Committee Member

Reynold Bailey

Advisor/Committee Member

Ashique Khudabukhsh

Campus

RIT – Main Campus

Download

Share

COinS