Abstract

Machine learning has evolved into advanced techniques for vision, language, and applications in different areas. However, human expertise is still essential in providing meaningful interpretations of the semantics for tasks in knowledge-rich domains, such as medicine, science, and security intelligence. It is beneficial to incorporate human knowledge into a machine learning system, and we consider human-in-the-loop an increasingly important area for future research. Human-in-the-loop machine learning aims to develop a reliable prediction model with minimal costs by integrating human knowledge and experience. In this thesis, we propose human-in-the-loop methods to address knowledge-rich data understanding challenges for different machine-learning tasks. Specifically, for multimodal data fusion tasks, leveraging human expertise to capture the cross-modal interactions is essential. We formulate domain knowledge from human experts via Bayesian graphical models. For segmentation tasks, we propose methods that leverage limited human annotation as weak supervision to make the model continuously encode the new knowledge. For vision-language tasks on image and video understanding, we develop methods that accept human input as side information to generate customized image or video descriptions and adapt the model to specific data types. To further enhance the efficiency of user annotation, we propose quantifying each information source's uncertainty to provide a holistic view of uncertainty. Such an uncertainty estimation can help the model effectively integrate different information sources, and select the most informative samples for human annotation. Our proposed models exhibit good performance in various machine-learning tasks by involving a human in the loop, facilitating human-machine interaction, and improving the interpretability of model predictions.

Library of Congress Subject Headings

Machine learning; Human-computer interaction; Knowledge acquisition (Expert systems)

Publication Date

12-2023

Document Type

Dissertation

Student Type

Graduate

Degree Name

Computing and Information Sciences (Ph.D.)

Department, Program, or Center

Computing and Information Sciences Ph.D, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Qi Yu

Advisor/Committee Member

Pengcheng Shi

Advisor/Committee Member

Rui Li

Campus

RIT – Main Campus

Plan Codes

COMPIS-PHD

Download

Share

COinS