Building a Language Simulator With World Labs
This past weekend I teamed up with my wife and three year old for a World Labs hackathon and built a rough language simulator on the web.
The idea is simple: practice a language by completing a real task in a real-feeling place.
The first scenario is a Taiwanese boba tea shop. You stand at the counter, hear what you need to order, speak in Mandarin, answer the cashier’s follow-up questions, and get a scored receipt at the end.
It lets you rehearse and go again and again with variations until you feel comfortable.
Ordering a drink sounds small, but it contains a lot of real language pressure:
• choosing the right drink • saying the size • adjusting sweetness and ice • fixing mistakes • confirming the final order • understanding the cashier • doing it fast enough that it feels natural
This is where World Labs gets interesting. We can swap the environment, and swap the scenario. Today it is a boba shop in Mandarin. Tomorrow it could be a French bakery where you need to buy bread, ask about pastries, and understand the total. Maybe our agents just build these for us?
Tech Stack Overview
• World Labs Marble for creating the 3D environment • Spark.js for rendering the Gaussian splat scene • Google DeepMind Gemini Live for voice input and spoken responses • Gemini TTS as a fallback for generated speech • Three.js for bringing the scene to the web • React, Vite, and TypeScript for the app layer • Codex for helping wire the prototype together quickly • Local parsing logic for fast order understanding before asking an LLM
The current demo is rough, but the shape feels right. The learner is trying to successfully complete a real task.
Welcome your thoughts on this hack!