Description
Computer-based sign language recognition from video is a challenging problem because of the spatiotemporal complexities inherent in sign production and the variations within and across signers. However, linguistic information can help constrain sign recognition to make it a more feasible classification problem. We have previously explored recognition of linguistically significant 3D hand configurations, as start and end handshapes represent one major component of signs; others include hand orientation, place of articulation in space, and movement. Thus, although recognition of handshapes (on one or both hands) at the start and end of a sign is essential for sign identification, it is not sufficient. Analysis of hand and arm movement trajectories can provide additional information critical for sign identification. In order to test the discriminative potential of the hand motion analysis, we performed sign recognition based exclusively on hand trajectories while holding the handshape constant. To facilitate this evaluation, we captured a collection of videos involving signs with a constant handshape produced by multiple subjects; and we automatically annotated the 3D motion trajectories. 3D hand locations are normalized in accordance with invariant properties of ASL movements. We trained time-series learning-based models for different signs of constant handshape in our dataset using the normalized 3D motion trajectories. Results show significant computer-based sign recognition accuracy across subjects and across a diverse set of signs. Our framework demonstrates the discriminative power and importance of 3D hand motion trajectories for sign recognition, given known handshapes.
Date of creation, presentation, or exhibit
5-23-2016
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Document Type
Conference Paper
Department, Program, or Center
Information Sciences and Technologies (GCCIS)
Recommended Citation
Dilsizian, Mark; Tang, Zhiqiang; Metaxas, Dimitris; Huenerfauth, Matt; and Neidle, Carol, "The Importance of 3D Motion Trajectories for Computer-based Sign Recognition" (2016). Accessed from
https://repository.rit.edu/other/894
Campus
RIT – Main Campus
Comments
Presented at the 7th Workshop on the Representation and Processing of Sign Languages: Corpus Mining, The 10th International Conference on Language Resources and Evaluation (LREC 2016), May 23-28, 2016, Portorož, Slovenia