Researchers at Harvard University have developed Mobilio, a smartphone application designed to provide continuous navigation support for people who are blind or have low vision. The system combines artificial intelligence, built-in smartphone sensors and personalised spatial audio. In early tests, participants navigated faster and made significantly less contact with obstacles than when using Google Maps together with a white cane.
Independent mobility remains a daily challenge for people who are blind or have low vision. Traditional mobility aids such as white canes, guide dogs and electronic travel devices can each address specific parts of that challenge, but few solutions combine reliable turn-by-turn navigation, continuous path guidance and obstacle avoidance within a single system. Engineers at Harvard University’s John A. Paulson School of Engineering and Applied Sciences have developed Mobilio to address that gap using hardware that many potential users already carry: a standard smartphone. The application combines machine learning, sensor fusion and personalised audio feedback. Instead of simply providing directions from one waypoint to another, Mobilio continuously analyses the user’s surroundings and provides guidance on where to walk and how to avoid obstacles. The research was led by PhD student Raymond Liu in collaboration with Patrick Slade, Assistant Professor of Bioengineering and Associate Faculty Member at the Kempner Institute. The findings were published in Nature Biomedical Engineering.
Three Core Requirements Shape the System
Before developing the technology, the researchers surveyed more than 100 blind or low-vision people to understand what they considered essential in a navigation system. Three capabilities emerged as particularly important: reliable turn-by-turn navigation, continuous guidance along pavements and paths, and obstacle detection and avoidance. Existing tools typically cover only part of this requirement. GPS-based mapping systems can provide general navigation instructions, for example, but they are not sufficiently precise to guide a blind user safely along a specific pedestrian route. Mobilio therefore goes beyond conventional waypoint-based navigation by continuously assessing the immediate environment.
Turning the Smartphone into a Mobile Sensor Platform
Mobilio uses components already integrated into modern smartphones. These include the camera, GPS and the inertial measurement unit that detects movement and device orientation. If the handset supports it, the system can also use LiDAR. One of the central technical advances is a specially trained computer-vision model. It analyses the live camera feed from a pedestrian perspective and identifies surfaces and structures relevant to safe navigation. These include pavements, pedestrian crossings, roads and other walkable or non-walkable areas. Patrick Slade compares the principle with the route planning of a small autonomous vehicle. The system combines smartphone sensor data, GPS information and the user’s actual movement to continuously calculate how the person should move from their current position towards the destination.
Why Vehicle-Based Training Data Are Not Enough
A major challenge was the visual-recognition model itself. Many publicly available street-scene datasets are captured from cars. That perspective is useful for autonomous driving, but it is poorly suited to pedestrian navigation because infrastructure that matters to a person walking may appear very differently. Standard models can therefore misclassify pedestrian-relevant structures. Liu addressed this by training a semantic segmentation model using images captured from a pedestrian perspective. The model is designed to distinguish more reliably between surfaces, routes and obstacles that matter to blind users. The work also highlights a broader issue in AI development: there is comparatively little training data collected from the perspective of people moving through public spaces on foot. A further challenge was computing power. All algorithms and models had to operate in real time on a conventional smartphone, which offers substantially less processing capacity than a laptop or specialised AI hardware. The researchers therefore designed Mobilio from the outset as a system that can run entirely on the phone and could eventually be distributed as a standalone application.
Spatial Audio Replaces Constant Spoken Instructions
Mobilio also differs from conventional navigation software in the way it communicates with the user. Rather than relying solely on spoken instructions, the app provides continuous audio signals. These beeps appear spatially from directions such as the left or right, indicating how the user should steer. The feedback is also personalised. As the person walks, the software measures how accurately they respond to the audio signals. Directional cues and pitch patterns are then adjusted automatically. The researchers describe this approach as human-in-the-loop optimisation. The aim is to avoid a one-size-fits-all navigation system. Instead, Mobilio adapts its guidance to the movement and response patterns of each individual user.
Early Tests with 14 Participants
For the first evaluation, the research team recruited 14 volunteers through the Carroll Center for the Blind in Newton, Massachusetts. Each participant completed several navigation tasks. The baseline consisted of Google Maps turn-by-turn navigation used together with a white cane. Participants then repeated the tasks using Mobilio in combination with the cane. The tests included a predefined outdoor route and an indoor obstacle course designed to reproduce common navigation challenges.The results favoured the Harvard system. Participants using Mobilio and a white cane completed the outdoor route around 13% faster than those using Google Maps with the cane. The difference was even more significant in obstacle avoidance. Contact with obstacles on the indoor course fell by approximately 41%. In terms of reliability — defined as how consistently the system guided users to their destination without major navigation errors — Mobilio performed at a level comparable with a human guide, according to the researchers.
An Addition to Existing Mobility Aids, Not Yet a Replacement
The early results also make clear what Mobilio has not yet demonstrated. The system was not tested as a standalone replacement for existing mobility aids. Participants continued to use a white cane during the trials. At this stage, the technology therefore adds a digital layer to tactile navigation, providing route guidance, environmental recognition and obstacle warnings alongside the immediate physical feedback of the cane. This combined approach may ultimately prove important. A cane can identify obstacles directly in front of the user, while the smartphone can provide information about direction, route layout and structures beyond immediate reach.
Personal Experience Shaped the Research
The project also has a personal dimension for Raymond Liu. His older brother is blind, and the experience of supporting his independent mobility strongly influenced the objectives of the research. The team therefore placed considerable emphasis on involving real users rather than developing the system purely around technical performance metrics. Liu argues that practical user testing is essential because it is difficult to predict how people will respond to an assistive device, particularly within the highly diverse spectrum of blindness and visual impairment.
Moving Towards Real-World Deployment
The current results represent an early stage rather than a final validation of the technology. The researchers plan to test Mobilio in a wider range of realistic environments around Boston. These trials will examine how well the system performs under changing conditions and how naturally it integrates into users’ everyday routines. The project is being supported by the Harvard Grid Accelerator, with the aim of further developing the technology and making it broadly accessible as a future product. Harvard University’s Office of Technology Development has already protected the intellectual property associated with the research and is exploring commercialisation opportunities.
Smartphone Hardware as an Assistive Platform
The significance of Mobilio lies less in any single sensor than in how existing technologies are combined. Camera, GPS, motion sensing, optional LiDAR, computer vision and adaptive audio guidance are integrated into a mobile assistance system intended to run on standard consumer hardware. If the approach proves reliable in larger and more diverse trials, it could lower the barrier to advanced digital navigation support. Instead of requiring dedicated specialist hardware, users could potentially access such functionality through software running on a smartphone they already own. The current study, involving 14 participants, remains too small for a definitive assessment. Nevertheless, the results indicate that the combination of AI-based environmental recognition, smartphone sensing and personalised audio guidance has the potential to measurably improve independent navigation for people who are blind or have low vision.
