# World Labs Web Based Language Sim

> A web-based language simulator prototype built with World Labs, Spark.js, and Gemini to practice real spoken tasks in a Taiwanese boba shop.

URL: [Original article](https://mattcool.tech/posts/world-labs-web-based-language-sim)

Published: 2026-05-12T13:12:00.000Z

Tags: notes, spatial-web, webxr, chinese

# Building a Language Simulator With World Labs

This past weekend I teamed up with my wife and three year old for a World Labs hackathon and built a rough language simulator on the web.

Very proud that my three year old (our lead tester) remembered to order me a bobatea too. I'm going to cherish that clip for a long time!

The idea is simple: practice a language by completing a real task in a real-feeling place.

[Image: World Labs boba shop language simulator scene]

Caption: The first prototype drops you into a Taiwanese boba shop and asks you to order in Mandarin.

The first scenario is a Taiwanese boba tea shop. You stand at the counter, hear what you need to order, speak in Mandarin, answer the cashier’s follow-up questions, and get a scored receipt at the end.

It lets you rehearse and go again and again with variations until you feel comfortable.

Ordering a drink sounds small, but it contains a lot of real language pressure:

- choosing the right drink
- saying the size
- adjusting sweetness and ice
- fixing mistakes
- confirming the final order
- understanding the cashier
- doing it fast enough that it feels natural

This is where World Labs gets interesting. We can swap the environment, and swap the scenario. Today it is a boba shop in Mandarin. Tomorrow it could be a French bakery where you need to buy bread, ask about pastries, and understand the total.
Maybe our agents just build these for us?

*Tech Stack Overview*

- World Labs Marble (https://docs.worldlabs.ai/) for creating the 3D environment
- Spark.js (https://sparkjs.dev/docs/) for rendering the Gaussian splat scene
- Google DeepMind Gemini Live (https://ai.google.dev/gemini-api/docs/live) for voice input and spoken responses
- Gemini TTS (https://ai.google.dev/gemini-api/docs/speech-generation) as a fallback for generated speech
- Three.js (https://threejs.org/) for bringing the scene to the web
- React (https://react.dev/), Vite (https://vite.dev/), and TypeScript (https://www.typescriptlang.org/) for the app layer
- Codex (https://openai.com/index/introducing-codex/) for helping wire the prototype together quickly
- Local parsing logic for fast order understanding before asking an LLM

The current demo is rough, but the shape feels right. The learner is trying to successfully complete a real task.

[Image: Scorecard showing the final boba order results and points]

Caption: The scorecard breaks down your performance.

Welcome your thoughts on this hack!