When she was younger, Anastasia Patricks would often sit down and watch dramas to relax – she didn’t expect to be called on to make a change. After watching the Japanese drama Orange Days, in which a character with hearing loss communicates with sign language, she found herself very interested in other shows about characters who were deaf or had hearing loss. It inspired her to learn some sign language herself.
“[Orange Days] was my first exposure to hearing loss and also sign language in general,” she recalls. “It sparked my interest in sign language. Over the years, I would watch different movies that featured people with hearing loss and how they use sign language to communicate.”
When she and her fellow Nanyang Technological University (NTU) students pursuing tech careers in Singapore thought about entering Microsoft’s Imagine Cup, they found themselves drawn to creating something that would help those who were deaf or had hearing loss. Anastasia had mentioned her interest in sign language to her friends and soon-to-be team members Richardson Qiu and Steven Rachman in the lunches after their NTU classes. It wasn’t long before sign language quickly became a shared interest for their Imagine Cup idea.
When the three students turned to existing sign language apps, they found that the majority lacked interactive and effective means of teaching. Existing apps taught people to finger spell and sign words or full sentences, but they didn’t allow users to practice sign language in a conversational manner – an important process for anyone to pick up a new language quickly and fluently.
The team thought that wasn’t enough and that they could do better. The desktop app, Sign2Sign – accessible via desktop and mobile browser, was their effort to bring better accessibility to sign language, one sentence at a time.
Team Sign2Sign, Assemble!
According to The Singapore Association for the Deaf (SADeaf), Singapore has an estimated 500,000 individuals with varying degrees of hearing loss – about 9% of the population. Of this proportion, 5,400 are registered with SADeaf. Of those registered, about a third know sign language. Globally, nearly 20% of the global population live with hearing loss (source: World Health Organization).
To create a more effective app, Team Sign2Sign asked a representative from SADeaf for insights – and learned that, while there was a set way of communicating within the community, that ability to communicate didn’t always extend to the families of those who were deaf or had hearing loss
“For people with hearing loss, interacting with their parents and their families can be challenging because their families don’t necessarily communicate in sign language,” says Richardson, an observation he made after an interaction with one of the middle-aged members of the association.
Inside the App
Armed with their research, Richardson, Anastasia and Steven started work on Sign2Sign, a hands-on learning app that takes a creative and inclusive approach to learning sign language. Designed for family members of people who have hearing loss and hearing people, Sign2Sign aspired to bridge the communication gap between hearing and hard of hearing individuals, one sentence at a time.
Available on desktop and mobile, the app puts a 3D avatar to work by demonstrating signs to help people learn signing vocabulary and practise it in conversation. With the help of the Microsoft Azure Learning Machine, the team integrated the AI model built using technology for sign recognition into Sign2Sign app effortlessly. The app provides real-time feedback for users to have two-way signing conversations through a screen, while watching themselves sign so they have a wider perspective on their progress.
To build the app, the team worked with Azure services, incorporating React. When it came time to train their AI, they also found that there isn’t enough data points and variation for each sign within the sign language. As a result, the team decided to generate their own data manually so they could increase the number of samples and variation of each sign – which helped improve the accuracy of the app.
To get their app up to speed, they adopted a quick experimentation approach, trying out several approaches and building several small prototypes for testing. They made use of Azure App Services for mobile and web, as well as Azure Machine Learning. Their agile approach also enabled them to make edits quickly for faster improvement. The team also learned to be proactive in consulting domain experts for advice on larger issues.
The Race to Imagine Cup
Team Sign2Sign rose to victory for the Lifestyle category out of tens of thousands of students from over 160 countries who registered for Imagine Cup 2022 and was one of 48 teams which advanced to the World Finals. They were awarded USD2,500 in cash and USD2,500 in Azure credits to continue building their app.
For two decades, Microsoft’s annual Imagine Cup has seen over 2 million students from 160 countries compete for prizes including training, mentorship, technology, publicity and cash.
Seeking to ignite a fire that inspires students to put their imagination and passion for technology to use, the competition aims to empower a new generation of developers and creators as they ignite new tech innovation and create solutions that tackle global problems.
Ahead on the Horizon
Team Sign2Sign plans to continue working with Microsoft Azure, while collaborating with more external organizations to further develop their sign language lesson plans. They also plan to expand their app to include other sign languages, including Singapore Sign Language.
“No one should be excluded in conversations, regardless of their ability to hear and speak. We hope that our app can provide a solution for users to learn sign language easily, anywhere and anytime, to create a more inclusive society,” says Richardson.
Held annually since 2003, the Microsoft Imagine Cup is the world’s premier student technology competition, affectionately known by participants as the “Olympics of student technology competitions”. This year’s Imagine Cup encouraged participants to “Dream it. Build it. Live it” and create applications that shape how individuals live, work and play and through their creativity, passion and knowledge of technology.
For more information about Imagine Cup, please visit https://www.imaginecup.com/.