A Sign of Things to Come: Russian Scientists Teach Sign Language to AI

Software engineers from SOL Surdo-Online, an innovative sign language translation service designed to allow deaf people to communicate with the non-deaf, have come up with plans to improve the service using artificial intelligence.

The SOL Surdo-Online service, a platform which interprets Russian sign language into speech in video format, allows the deaf and hard of hearing to communicate with the non-deaf in airports, pharmacies or anywhere else where an exchange of information is required.

Now, the company is working to introduce artificial intelligence (AI) into the equation in order to improve the system further.

Speaking to Russia's RIA Novosti, Surdo-Online project leader Alexei Melnik explained that "in Russia, there are approximately 200,000 deaf people, and one million who are hard of hearing, who use sign language. Their opportunities for communicating with the outside world are limited, while the government provides for only 40 hours of free assisted sign language interpretation [per year]."

"Using a platform where translators work online, it's possible to improve the efficiency at least fivefold. This is how the idea of project SOL Surdo-Online was born, and we launched it in September 2016. Now, we have decided to improve the system through the introduction of artificial intelligence," Melnik said.

At the moment, the service operates according to the following scheme: companies are given access to an online cloud service which allows users to contact sign language interpreters. These interpreters then relay the information over to cashiers, pharmacists, or whomever else the deaf person may be speaking to, and interpret speech back to the user. The service uses an ordinary Google Chrome browser, enabling it to be connected using an ordinary phone, tablet or computer.

Hearing impaired user using the "SOL Surdo-Online" service in a Russian pharmacy

© Photo : SOL Surdo-Online

However, developers are looking to streamline the service further. By late 2018, they plan to release a prototype video platform service where AI is used to recognize user gestures, effectively allowing the deaf and hard of hearing to communicate independently, without the help of human interpreters. 

The system is being trained to recognize gestures with the help of a neural network. At the moment, developers are assembling a database of gestures using video conferences between sign language users. It is estimated that the AI requires at least 1,000 demonstrations of gestures by different people to accurately figure out the patterns involved and their variations. 

The next step will be to synthesize these gestures to account for Russian sign language's 10,000+ gestures. The main difficulty, according to developers, will be the digitization of this '3D language' to enable it to become fully automated.

The SOL Surdo-Online team recently participated in EdCrunch, an international educational technology conference held in Moscow last week.

There, the project was evaluated by foreign experts. Peter Childs, director of the Innovation Design Engineering program in the UK, said that SOL Surdo-Online was a realistic project which already has a functional prototype of its platform. The project now requires investment and advice, but has great potential even at the international level, Childs said.