NALscribe AR

Project Objectives

People with hearing loss often find it challenging to follow conversations, particularly in group settings or noisy environments. These difficulties can lead to increased effort, fatigue, and reduced participation in social and work situations. Live captioning apps can help by converting speech into text, but using a phone means looking away from the speaker, making it harder to see facial expressions and lip movements that many people rely on to support understanding.

The objective of this project was to explore whether live captions could be displayed directly within a person’s field of view using augmented reality, allowing access to captions without taking visual attention away from the conversation. 

Key Findings

The project delivered a working proof of concept, NALscribe AR, demonstrating that live captions can be displayed within a user’s field of view using augmented reality, rather than on a smartphone screen. This allows people to read captions while still looking at the person speaking, supporting more natural face-to-face communication, and addressing a key limitation of existing captioning apps.

The project also identified practical ways to improve caption quality in conversations. By using nearby smartphones as microphones, speech was captured more reliably, making it possible to present captions from different speakers separately. Captions could also be positioned in 3D space near each speaker, helping users identify who was speaking and follow the flow of conversation.

This work is unique in applying emerging augmented reality technology to a real-world communication challenge. It provides valuable insight into how future wearable technologies could better support people with hearing loss and demonstrates NAL’s role in exploring innovative, user-focused solutions for communication accessibility.

Next Steps and Potential Applications

This project was an early exploration of how emerging technologies could be used to improve access to conversation for people with hearing loss. It showed that augmented reality can be used to support hearing-related applications in new and creative ways, with live captioning as one initial example.

While the Apple Vision Pro is currently a high-cost and relatively heavy device, the project highlights what may be possible as wearable technology continues to evolve. The ideas explored in this work could be extended beyond captioning to support a range of future hearing-related applications, such as improved awareness of who is speaking, visual cues to support communication, and new ways of interacting with sound in everyday environments.

This exploratory work provides a strong foundation for further innovation and feasibility testing, helping to shape how future wearable technologies, such as smart glasses, could be used to improve communication and participation for people with hearing difficulties.