Abstract

Vocational training programs aimed at improving the employability and autonomy of autistic individuals primarily focus on enhancing social communication skills. However, developing and implementing such training activities often demands significant time and effort from job coaches, who must adapt to various interview and workplace scenarios while customizing the training to meet the specific needs of autistic trainees. These challenges create barriers for job coaches to efficiently train autistic individuals and objectively evaluate their performance. To address these challenges, this dissertation explores the use of virtual reality (VR) and large language model (LLM)-driven agents to create customizable and interactive training platforms for autistic trainees and job coaches, especially for communication skills training. The research involved user evaluations with job coaches and autistic trainees in vocational training programs and comprises four studies.  In Part I, we explored the potential and usability of LLM-driven agents in VR for job-related communication skills training for autistic individuals. We developed a platform that utilizes LLM prompts to create customizable communication training scenarios. This platform allows trainees to independently and repetitively engage in communication training, while job coaches—regardless of technical expertise—can create immersive training simulations using prompts and 360-degree imagery. A preliminary study was conducted with autistic trainees and job coaches to evaluate the platform's usability and gather user feedback. Building on the platform from Part I, Part II focused on the experiences of job coaches as they developed communication training scenarios using LLM prompts. We conducted a longitudinal study in which job coaches learned to use this platform and integrated it into their routine training sessions, providing insights into their practices and the challenges they encountered when using this tool for training with autistic trainees. Part III, informed by feedback from job coaches and prior research, investigated the integration of environmental contexts and hand-based interactions into communication training. The goal was to enhance the realism of simulations and improve training outcomes. We developed an LLM-driven agent in VR capable of interpreting and responding to verbal inputs, environmental objects, and physical actions. Five everyday communication scenarios were created, and a study was conducted to assess agent behavior and user perceptions, using human evaluators to analyze the realism and effectiveness of the interactions. Part IV, the final study, evaluated the training experience and actual practice of trainees and job coaches in environment-grounded VR communication training. Building on the platform and prompting schema from Parts II and III, we extended the platform to incorporate environmental objects and grasp interactions. Its evaluation comprised a two-phase study. In Phase 1, we conducted a comparative study with autistic trainees and job coaches across three interaction modalities in VR communication training for autistic individuals: conversation-only (C), conversation with environmental objects (C+O), and conversation with environmental objects and grasp interactions (C+O+G), examining whether incorporating environmental objects and grasp interactions affected usability, user experience, and training perceptions. Next, in Phase 2, we integrated the most preferred C+O+G modality into the environment-grounded VR communication training platform and conducted a one-month longitudinal study with job coaches to evaluate how they developed and integrated environment-grounded VR communication training scenarios into routine practice, and to identify the opportunities and challenges in the process. Through these four studies, this dissertation contributes a customizable VR communication training platform driven by LLM-powered agents, a prompting schema for generating context-aware agent behaviors, and empirical findings on the usability, user experience, and practical adoption of environment-grounded VR communication training with autistic trainees and job coaches. This dissertation aims to support more independent and engaging role-play communication practice for autistic trainees in preparing for real-world job conversation settings and to empower job coaches to more easily create and deliver communication training scenarios that replicate real workplace interactions and are tailored to individual trainees' needs.

Publication Date

7-2026

Document Type

Dissertation

Student Type

Graduate

Degree Name

Computing and Information Sciences (Ph.D.)

Department, Program, or Center

Computing and Information Sciences Ph.D, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Roshan L. Peiris

Advisor/Committee Member

Khoo Eng Tat

Advisor/Committee Member

Ha-Kyung (Hidy) Kong

Campus

RIT – Main Campus

Download

Share

COinS