Aadit KrishnaAnsh KrishnaArmaan AgrawalJonathan Wang

Medkit

First-aid guidance in your ear, hands-free

An AI first-aid coach for Meta smart glasses. Voice-guided, hands-free emergency support with real-time visual aids.

Features

Voice-first coach

Short, direct replies. One step at a time. Designed for hands-busy use.

Wake word & hands-free

Say 'Medkit' to start. Stays active in critical scenarios.

Scenario-aware

AI infers situation type and severity from conversation.

On-screen checklists

Step-by-step cards and checklists on your phone so you know exactly what to do next.

Live tools

Metronome for CPR, countdown timers, on-screen checklists.

Scene understanding

Camera analyzes hand placement and position to refine guidance.

Architecture

System flow from glasses to AI

INPUTπŸ•ΆοΈMeta Ray-BanMeta Wearables DAT SDKπŸ“ΉCamera FeedStreamSession video frames🎀MicrophoneHFP Bluetooth audioπŸ“±iPhone FallbackBack camera + micπŸ”ŠSpeakerAI voice via HFPBLE + HFP BluetoothπŸ–ΌοΈJPEG CompressSampled every 3 secondsπŸ«₯Face BlurPrivacy filter on all facesbase64πŸ”„PCM16 Convert24kHz mono, int16πŸ’¬Wake Word GateSFSpeechRecognizerSays "Medkit" β†’ activatebase64WebSocketwss://ModalON-DEVICE PROCESSING (iOS)DEDALUS ORCHESTRATION LAYER🧠K2 ThinkOrchestrating AgentRoutes inputs to the right model, coordinates responsesframesaudioVOICE MODELπŸ—£οΈOpenAI Realtimegpt-4o-realtime-previewServer VAD β€’ Whisper-1PCM16 in/out β€’ Tool callingDirect WebSocket to OpenAIspeechVISION MODELπŸ‘οΈGPT-4o Visionvia AsyncDedalus SDKmodel: "openai/gpt-4o"JPEG β†’ 1-2 sentence scene descriptionDAuth: keys never on our serverblurred frames[VISUAL CONTEXT][SCENE UPDATE]AI voice responseTOOL CALLSset_scenariotype + severity + regionstart_metronomeCPR rhythm (110 BPM)start_timerCountdown (switch, hold)show_uiChecklist / alert cardβ†’ Routed to iOS ToolExecutor via WebSocketiOS TOOL EXECUTION🎡Metronome880Hz via HFP⏱️TimersCountdown UIπŸ—ΊοΈAED / HospitalMapKit searchπŸ“‹UI CardsChecklist / alertπŸ“ΉRecordingH.264 + AACπŸ“„PDF ReportsSession exportDAuth Credential FlowπŸ”1. EncryptAPI key encrypted on ModalπŸ›‘οΈ2. Sealed EnclaveRust, isolated β†’ OpenAIβœ…3. Secure ResponseBack to backendπŸ”’ Raw OpenAI keys never exist on our serveruses DAuthData Flow LegendSpeech: Mic β†’ PCM16 β†’ wake word β†’ WebSocket β†’ K2 β†’ RealtimeVision: Camera β†’ JPEG β†’ face blur β†’ WebSocket β†’ K2 β†’ GPT-4o (Dedalus)Response: Realtime generates voice β†’ back through WebSocket β†’ glasses speakerTools: K2 calls tools β†’ routed to iOS β†’ metronome / timers / maps / UI cards

User walkthrough

First-time user journey from connecting glasses to completing a session

Setup

Connect glasses

Open the app, tap 'Connect Glasses,' complete Meta's pairing/registration (and any OAuth in-browser).

Setup

Start session

When the glasses are connected, tap 'Start Session'; grant mic, speech, and camera permissions if prompted.

Ready

See the feed

Video from the glasses and an audio visualizer show that the session is live; status shows 'Say Medkit to start' (or 'Listening' when active).

Active

Activate

Say 'Medkit' (or a close phrase the app accepts); status switches to 'Listening' and audio is sent to the coach.

Active

Describe the situation

e.g. 'My friend fell and isn't breathing.' The coach responds by voice and may set a scenario (e.g. CPR); the phone shows checklists, metronome, and timers.

Active

Follow guidance

User follows voice steps and glances at the phone for rhythm, timers, or checklist; if the coach gets a scene update, it may say something like 'Your hands look good β€” keep that pace.'

Active

Check-ins

If the user is silent for a while, the coach may ask a short check-in; user can answer or keep going.

Complete

End session

User taps stop and/or disconnects the glasses when done.

Decision support only β€” not a substitute for professional medical help. Always call emergency services when appropriate; follow local first-aid guidelines and training.