Building a Language Simulator With World Labs

This past weekend I teamed up with my wife and three year old for a World Labs hackathon and built a rough language simulator on the web.

Very proud that my three year old (our lead tester) remembered to order me a bobatea too. I’m going to cherish that clip for a long time!

The idea is simple: practice a language by completing a real task in a real-feeling place.

The first scenario is a Taiwanese boba tea shop. You stand at the counter, hear what you need to order, speak in Mandarin, answer the cashier’s follow-up questions, and get a scored receipt at the end.

It lets you rehearse and go again and again with variations until you feel comfortable.

Ordering a drink sounds small, but it contains a lot of real language pressure:

• choosing the right drink
• saying the size
• adjusting sweetness and ice
• fixing mistakes
• confirming the final order
• understanding the cashier
• doing it fast enough that it feels natural

This is where World Labs gets interesting. We can swap the environment, and swap the scenario. Today it is a boba shop in Mandarin. Tomorrow it could be a French bakery where you need to buy bread, ask about pastries, and understand the total. Maybe our agents just build these for us?

Tech Stack Overview

• World Labs Marble for creating the 3D environment
• Spark.js for rendering the Gaussian splat scene
• Google DeepMind Gemini Live for voice input and spoken responses
• Gemini TTS as a fallback for generated speech
• Three.js for bringing the scene to the web
• React, Vite, and TypeScript for the app layer
• Codex for helping wire the prototype together quickly
• Local parsing logic for fast order understanding before asking an LLM

The current demo is rough, but the shape feels right. The learner is trying to successfully complete a real task.

Welcome your thoughts on this hack!

---