DESIGNING TRANSLATION TOOL: BETWEEN SIGN LANGUAGE TO SPOKEN TEXT ON KINECT TIME SERIES DATA USING DYNAMIC TIME WARPING

Zico Pratama Putera, Mila Desi Anasanti, Bagus Priambodo

Abstract


The gesture is one of the most natural and expressive methods for the hearing impaired. Most researchers, however, focus on either static gestures, postures or a small group of dynamic gestures due to the complexity of dynamic gestures. We propose the Kinect Translation Tool to recognize the user's gesture. As a result, the Kinect Translation Tool can be used for bilateral communication with the deaf community. Since real-time detection of a large number of dynamic gestures is taken into account, some efficient algorithms and models are required. The dynamic time warping algorithm is used here to detect and translate the gesture. Kinect Sign Language should translate sign language into written and spoken words. Conversely, people can reply directly with their spoken word, which is converted into literal text together with the animated 3D sign language gestures. The user study, which included several prototypes of the user interface, was carried out with the observation of ten participants who had to gesture and spell the phrases in American Sign Language (ASL). The speech recognition tests for simple phrases have therefore shown good results. The system also recognized the participant's gesture very well during the test. The study suggested that a natural user interface with Microsoft Kinect could be interpreted as a sign language translator for the hearing impaired.

Keywords


Human-Computer Interaction: Natural User Interface; Speech Recognition; 3D Animation; Sign Language

Full Text:

PDF


DOI: http://dx.doi.org/10.22441/sinergi.2018.2.004

Refbacks

  • There are currently no refbacks.


SINERGI
Fakultas Teknik Universitas Mercu Buana
Jl. Raya Meruya Selatan, Kembangan, Jakarta 11650
Tlp./Fax: +62215871335
p-ISSN: 1410-2331
e-ISSN: 2460-1217
http://publikasi.mercubuana.ac.id/index.php/sinergi

 

Creative Commons License
Journal by SINERGI is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

View My Stats

 

The Journal is Indexed and Journal List Title by: