Abstract

Deaf and hard of hearing (DHH) individuals face several barriers to communication in the workplace, particularly in small-group meetings with their hearing peers. The impromptu nature of these meetings makes scheduling sign-language interpreting or professional captioning services difficult. Recent advances in Automatic Speech Recognition (ASR) technology could help remove some of these barriers that prevent DHH people from becoming involved in group meetings. However, ASR is still imperfect, and it contains errors in its output text in many real-world conversation settings. This research investigates whether there are benefits in using ASR technology to aid understanding and communication among DHH and hearing individuals. This dissertation evaluates the effectiveness of using ASR in small group meetings, through empirical studies with both DHH and hearing participants, and develops guidelines for notification system design to encourage hearing participants to communicate and speak more clearly. This dissertation is divided into five main studies (I, II, III, IV, and V): Study I revealed that hearing conversational partners change their speaking behaviors when conversing with DHH individuals in the presence of ASR technology, and that their behaviors can be influenced by changing the design of ASR technologies. Showing that hearing speakers are capable of having their behaviors influenced provided a foundation for the rest of this dissertation to build upon. Study II presents research evaluating which behaviors the DHH community wants to see exhibited by their hearing conversational partners. Results from this study helps to create a core set of beneficial and desirable behaviors in hearing speakers that should be encouraged within ASR application design. Study III describes a novel, entirely virtual co-design approach with DHH and hearing pairs. This methodology was used to create prototype design features for ASR-supported communication technologies, in particular focusing on how to fix errors in ASR output and how to implement a notification system to influence speakers' behaviors, that will be tested in Study IV and Study V of this dissertation. Study IV describes a Wizard-of-Oz research study that implements three prototype designs for ASR notifications, informed by Study III, using an ASR-supported videoconferencing application designed to encourage certain beneficial and desirable changes in behaviors, as informed by Study II. This study investigates subjective preferences of both hearing and DHH participants and also evaluates how effectively the notification designs influence the participants' speaking behaviors. Study V describes an empirical research study investigating the effectiveness of our top two ASR notification prototypes in facilitating communication between DHH and hearing colleagues in natural conversational settings. This final study evaluates each prototype during natural conversation contexts primarily by measuring subjective ratings and interview questionnaire feedback from both DHH and hearing participants.

Library of Congress Subject Headings

Automatic speech recognition--Public opinion; Communication devices for people with disabilities--Design and construction; Meetings--Accessibility for people with disabilities; Deaf people--Means of communication; Hard-of-hearing people--Means of communication

Publication Date

7-2023

Document Type

Dissertation

Student Type

Graduate

Degree Name

Computing and Information Sciences (Ph.D.)

Department, Program, or Center

Computing and Information Sciences Ph.D, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Matt Huenerfauth

Advisor/Committee Member

Kristen Shinohara

Advisor/Committee Member

Garreth Tigwell

Campus

RIT – Main Campus

Plan Codes

COMPIS-PHD

Download

Share

COinS