SecondSight
A phone-based assistant that helps blind and low-vision people avoid everyday hazards — spotting obstacles on the ground in real time and describing the world around them on request.
Overview
Traditional aids like canes and guide dogs are invaluable, but they can miss small, close-range hazards. SecondSight complements them — a wearable-style safety layer that watches the ground and speaks up before you reach the danger.
It runs two modes. Point the camera at the ground and it continuously detects common hazards — rocks, bottles, potholes, poles, stairs — and warns you through sound and vibration on the phone or a paired Apple Watch. Lift the camera and it switches to describing the scene around you. The whole safety loop runs on the device, so it keeps working with no internet connection.
This is a proof-of-concept built for research and demonstration — not a certified assistive or medical device.
Demo
How it works
- 1
On-device hazard detection
A compact YOLO 11 Nano model runs directly on the iPhone, flagging hazards in under a third of a second — fast enough to warn before the next step.
- 2
Automatic mode switching
The gyroscope reads camera angle: pointed down (within ~30° of the ground) it watches the path; lifted, it switches to describing the scene — no buttons to press.
- 3
Offline scene description
An on-device visual language model (FastVLM) generates spoken descriptions of the surroundings, fully offline for privacy and reliability.
- 4
Hands-free, wearable alerts
A standalone Apple Watch companion delivers haptic and speech alerts on the wrist — it can even be clipped to a cane and controlled with a thumb.
- 5
Trained by an MLOps pipeline
The detection model comes from an end-to-end CI/CD pipeline (EnigmaAI) covering data preparation, training, hyper-parameter optimisation, validation and deployment — served to the app through a stateless FastAPI microservice.
Architecture
Three independent components with clear separation of concerns: an MLOps pipeline that produces and validates the model, a FastAPI inference service, and the iOS/watchOS app that runs the on-device experience.
