Abstract

Recent evidence in biology indicates crossmodal, which is to say information sharing between the different senses, influences in the brain. This helps to explain such phenomenon as the McGurk effect, where even though a person knows that he is seeing the lip movement “GA” and is hearing the sound “BA”, the person usually can’t help but think that they are hearing the sound “DA”. The McGurk effect is an example of where the visual sense influences the perception of the audio sense. These discoveries transition old feedforward models of the brain to ones that rely on feedback connections and, more recently, crossmodal connections. Although we have many software systems that rely on some form of intelligence, i.e. person recognition software, speech to text software, etc, very few take advantage of crossmodal influences. This thesis provides an analysis of the importance of connections between explicit modalities in a recurrent neural network model. Each modality is represented as an individual recurrent neural network. The connections between the modalities and the modalities themselves are trained by applying a genetic algorithm to generate a population of the full model to solve certain types of classification problems. The main contribution of this work is to experimentally show the relative importance of feedback and crossmodal connections. From this it can be argued that the utilization of crossmodal information at an earlier stage of decision making can boost the accuracy and reliability of intelligent systems.

Library of Congress Subject Headings

Intersensory effects--Computer simulation; Perception--Computer simulation; Neural networks (Computer science)

Publication Date

2007

Document Type

Thesis

Department, Program, or Center

Computer Science (GCCIS)

Advisor

Gaborski, Roger

Advisor/Committee Member

Butler, Zack

Advisor/Committee Member

Reznik, Leon

Comments

Note: imported from RIT’s Digital Media Library running on DSpace to RIT Scholar Works. Physical copy available through RIT's The Wallace Library at: QP441 .A5 2007

Campus

RIT – Main Campus

Download

Share

COinS