AMMA
Most guidance breaks the moment you do. Miss a step and the instructions march on without you. AMMA adapts the interface to the person, switching between HUD, object highlighting, and speech from a live user model. I built the multimodal pipeline and the engine bridge that made it run inside a commercial VR game.
Guidance that breaks the moment you do
Step-by-step instructions assume a perfect user: right order, no mistakes, one way of understanding. Real people burn the salmon, work out of sequence, and read a HUD where someone else needs to hear the step. When guidance can’t adapt, it stops being help and starts being noise.
The research question: how does guidance move past a fixed script to adapt to mistakes, differing physical abilities, and individual communication preferences, in real time?
Adapt the interface, not just the instruction
AMMA generates the interface from a step list by tracking what the user has actually done and learning a personal model of how they work, then choosing which modality to fire and when. The hard part was making it run inside a real VR task, where the system watches every grab, pour, and seasoning and reacts before the moment passes.
Building the adaptive layer
I owned the layer that turns a tracked state into the right cue at the right time, and the bridge that let research logic drive a closed commercial game.
Component details
| Component | Detail |
|---|---|
| C#/BepInEx plugin | Injects research logic into the closed game, relaying every grab, pour, and seasoning to the Python side with minimal latency. |
| Research instrumentation | Pickle-based logging for perfect post-study replay, plus a study CLI to start and stop sessions, reset positions, and manage continuous speech recognition. |
| HUD · highlight · speech | The logic that switches between a HUD, 3D object highlighting, and speech, deciding by task type and the user's learned preference, prioritizing time-critical cues like Flip or Stop so instructions never stack. The speech pipeline parses raw recipe data (0/300mg) into natural speech (300 milligrams) to hold immersion. |
Faster, lighter, and tuned to the person
I joined AMMA as a Stanford CURIS undergraduate researcher in Summer 2022 and am third author on the IEEE VR 2024 paper. I built the adaptive multimodal pipeline (HUD/highlight logic, the Azure TTS pipeline, and the language parsing) and the research instrumentation (logging + study CLI), and co-developed the C#/BepInEx engine bridge. The state tracker and the study design were team work, led by Jackie (Junrui) Yang and advised by Monica S. Lam and James A. Landay.
The lesson that stuck wasn’t about adaptivity. It was about restraint. The fastest way to overwhelm someone in VR is to give them more guidance, so the real engineering was deciding what not to surface: suppressing redundant cues, interrupting cleanly when a time-critical step arrived, and keeping speech natural enough that it didn’t break immersion. And making research logic drive a closed commercial game (C# to Python over a socket, hooking low-level events in real time) taught me that the unglamorous bridge is often what decides whether an idea can be studied at all.