Handspeaker: : Animation of the Mexican Sing Language Dactylology

Hand Speaker: Animation of the Mexican Sing Language Dactylology

G., JEREZ-GOMEZ*

Escuela de Ingeniería, Universidad Modelo; 15233939@modelo.edu.mx

A.C., CASTRO-CANCHOLA

Escuela de Ingeniería, Universidad Modelo; 15234419@modelo.edu.mx

P.I., ROSAS-PEREZ

Escuela de Ingeniería, Universidad Modelo; 19198549@modelo.edu.mx

R., NAVA-MARTINEZ

Escuela de Ingeniería, Universidad Modelo; rutilio.navam@modelo.edu.mx

The Mexican Sign Language Fingerspell Translator represents a critical tool for individuals attempting to learn Mexican Sign Language (LSM). This innovative tool offers a 3D visualization of each letter in LSM dactylology. Although assistive technologies such as hearing aids and closed captioning have significantly enhanced communication capabilities, a substantial communication gap persists between sign language users and non-users. To build up and provide a more interactive and affordable learning method for understanding and assimilate abstract concepts in LSM. This system has been developed to translate written text into Mexican Sign Language dactylology, moreover the system was created using Unity, a real-time 3D development platform, to facilitate real-time translation and visualization.

CCS CONCEPTS •

Software and its engineering

Medium Relevance

• Software creation and management
• Designing software
• Software design engineering

Additional Keywords and Phrases: Dactylology, Mexican Sign Language, Animation

1. Introduction

In Mexico, approximately 2.3 million people suffer from hearing disabilities [1]. Most of these individuals do not have access to adequate interventions, and when they do, the processes can take years and incur significant amount of money. This has an impact on the lives of affected individuals, as their limitations prevent them from having the same opportunities as others.[pic 1]

According to Guerrero-Balaguera & Pérez-Holguín [2], most people with hearing impairments interact with each other using sign language; otherwise, they often need an interpreter to communicate with those who do not understand their language. This situation creates a communication barrier and restricts the daily lives of these individuals. Marzo Peña, L. A., Rodríguez Fleitas, D. C., and Fresquet Pedroso, D. C. [3] state, “In the education of deaf people, there has been a historical debate regarding the methods, trends, and models used for their educational process. Some advocate for more hierarchical positions, while others support respect for their linguistic, communicative, and cultural diversity, such as sign language.

“Considering the context of hearing disabilities in Mexico, we offer a project involving technology to gap opportunities where communication is not a hindrance for either the receiver or the transmitter. Rojas Hernández, R., González Jaime, E.I., Trujillo Mora, V., López Chau, A., & González Morán et al. [4] mention that “The teaching of Mexican Sign Language (LSM) can be done through very short videos, as the positions and movements of the hand to represent a letter or word do not require much time.” Many users have taken it upon themselves to share videos showcasing the available vocabulary in LSM so that people can learn on their own without the need for an instructor. Utilizing technological tools, which are advancing rapidly, to develop support systems for people with disabilities is becoming increasingly common, as computers are interacting more and more with people and their daily lives.

According to Pathan RK, Biswas M, Yasmin S, Khandaker MU, Salman M. et al [5], many techniques have been developed using sensor data processing and motion detection through the application of various algorithms and dynamic methods such as machine learning and deep learning. Depending on the methodologies, researchers have proposed their ways of classifying sign languages, and based on this, the project can be developed. The overarching goal of this project is to design a translator from written text to the Mexican Sign Language alphabet to promote the inclusion of people with hearing disabilities.

The proposed design is validated only for the Mexican Sign Language alphabet, where only spelling will be shown. According to Cruz-Aldrete, M., DIELSEME [6] constitutes a valuable support resource to facilitate communication with people with hearing disabilities.

1. MATERIALS AND TOOLS

1. Mexican Sign Language Dactylology

According to Marzo Peña, Alejandro, et al. [3], "Sign languages are visual-gestural-dimensional languages based on the use of hands, eyes, face, mouth, and body. They are languages meant to be seen, not heard, and they fulfill the same functions as spoken languages for hearing people." These languages possess their own grammar, syntax, and lexicon, enabling people with hearing impairments in Mexico to express and communicate with each other. However, a significant percentage of the population is unfamiliar with it, complicating their interaction with the public.

Papatsimoul, M., Sarigiannidi, P., & F. Fragulis, G. [7] mentions that by incorporating sign language interpretation into such projects, we promote inclusion, providing equal access to information and entertainment while strengthening community bonds. Mexican Sign Language dactylology consists of both static and dynamic signs, the latter being those that exhibit certain movements indicated by a black arrow (see Fig 1).

[pic 2]

Fig 1. Mexican Sign Language Dactylology [9]. Photograph by Conadis. [Public domain], via. Gobierno de México. (www.gob.mx)

1. Unity 2022.3

Unity (game engine) was employed as the main tool since this software facilitates the development of 3D animations. Its graphics engine allows for the creation of realistic three-dimensional environments, which aids in visualizing data such as the animation of Mexican Sign Language dactylology. Additionally, its multiplatform support ensures that simulations can be run on a variety of devices, from desktop computers to mobile devices, making it an ideal platform for the project's development.

1. METHODOLOGY

Using Unity, the hand animation was created through a default template from the UNITY Assets. This 3D template covered the wrist to the fingertips (see Fig 2).

[pic 3]Fig 2. Wrist to fingertip template. Using 3D dimensions such as X, Y and Z to realize the animation

To edit the static letters, coordinates provided by UNITY in the "transform" tab were used. These position, rotation, and scale coordinates were visualized by selecting each bone and moving them along the X, Y, and Z axes. A set of coordinates for each finger in their respective positions was obtained, which together constitute the coordinates of a letter.

The animations for the dynamic letters were created using the "Animation" tab provided by Unity. Unity’s animation system is based on the concept of Clips which contain information about how certain objects should change their position, rotation, or other properties over time. Each clip can be thought of as a single linear This section works on a frame-by-frame basis, recording and moving some hand joints, which are then captured in the necessary frames for a letter in Mexican Sign Language dactylology. The hand's required position was recorded, stored in a Unity-compatible file executable only when the hand template is available.

Animation Clips are then organized into a structured flowchart-like system called a Controller. The Animator Controller acts as a “State Machine which keeps track of which clip should currently be playing, and when the animations should change or blend together. A very simple Animator Controller might only contain one or two clips, for example to rotate the wrist and move the fingertips.

The project was designed to function as follows: the user, through their mobile device, types of letters within the program, which then executes an animation of the written phrase in Mexican Sign Language dactylology (see Fig 3), forming a word that aids in communication and interaction between people with hearing impairments and others in their social circle. Chistol, Mihaela, Turcu, Cristina & Danubianu, Mirela [8] mention that creativity plays a fundamental role in developing software that is visually appealing and easy to use, while effectively solving challenges. Therefore, an attractive hand design was considered for the user.

...