Building on my last Transformers.js experiment in a browser game, Local WebGPU Agent Lab asks: what happens if I give a local, in-browser model a playground in my blog?

Once the models are downloaded (about 1.9–2.3 GB with the current quantization choices), agent inference, speech recognition, and optional speech synthesis run in the browser. You can then start building in the lab with the agent.

Build Together in the Lab

The agent has a sprite kit and a set of reusable nodes to build things with. These stations occupy spaces in the room, and you can click a station to use its controls. The agent can also make decorative changes and other interesting combinations as you play. I lazily reused a character sprite from my GTC San Jose web game for Matt.

You can give the agent a direction like “build a place where we can make music.” It inspects the open lab spaces, chooses one specific output, and selects nodes that might work. I hope to swap in smaller and more capable models in the future and continue to test.

The lab can also use parts of my actual blog, including post metadata, excerpts, images, and references to components used by other posts.

Try the Lab

Click the lab below to open it and try things out!

If WebGPU, its adapter, the worker, or a core model fails, then the lab should open with a compatibility dialog that explains the problem. If it doesn’t - let me know!

Agent Appendix

Agent’s Building Tools

• Inspect the current stations and their state.
• Choose from a 26-piece semantic sprite kit and place, move, rotate, or remove lab equipment and construction pieces.
• Extend the constructed room shell when a directive needs more floor space.
• Create a station around one persisted output contract, assign every reusable node a concrete effect on that output, arrange its node canvas, and choose a compact 1 × 1 or large 2 × 2 persistent footprint in the room.
• Build a cleanup robot that can remove persisted junk piles and small fires when those incidents are intentionally added to the lab.
• Modify, connect, focus, or remove existing stations.
• Change station state through the same persisted lab state and session undo history used by the user controls.
• Search the blog’s same-origin image path catalog without downloading the images themselves.
• Build Site library and Resource output nodes from compact overview excerpts, post metadata, lazily loaded photos, metadata-only component references, and full-blog excerpts fetched only when requested. Astro and Svelte source stays metadata-only instead of entering the React bundle.
• Run known useful machines through deterministic, schema-valid tool recipes; the blog printer creates or reuses a capable station, then runs print_blog_image without depending on a lucky second model call.
• Give a build a capability such as image printing, then create or remove visible outputs tied back to that build.
• Finish by explaining the build’s concrete purpose, functional nodes, and exact bound blog resource when it used one.

A Web Worker keeps inference responsive. The page still owns the state and turns every accepted tool call into a visible event. Build requests retain a small functional intent instead of collapsing into one generic creative path: music requests are repaired toward playable keyboards, pads, sequencers, waveforms, and different oscillator/scale state, while writing, signal, light, play, and visual requests draw from their own node recipes. Open-ended visual directions can still use gradients, auroras, orbits, ribbons, mosaics, waves, colors, motion, and controls, but that grammar is no longer the default for every machine. A resource-backed music build can bind an exact post or excerpt and deterministically derive its rhythm, tempo, tone root, scale, and oscillator from that source. For open-ended creative builds, the lab records prior titles and design signatures; the host rejects repeats, and the safe fallback excludes them.

Notes for Coding Agents

• Ownership: LocalAgentLab.tsx owns the UI, intent routing, host recipes, event stream, and commits. labState.ts owns parsing, bounded persistence, and pure lab transitions. agentTools.ts is the exhaustive model-callable schema. stationOutput.ts, stationLayout.ts, and labSprites.ts enforce output, geometry, and sprite-crop invariants. labShare.ts renders state-driven share images. The worker owns model I/O.
• Mutation boundary: model text never touches React state or localStorage directly. Parse and allowlist an AgentAction, normalize it for the active goal, run applyAgentAction, and commit only a returned state with a new identity. Rejected and no-op actions must not create undo or storage writes.
• Core invariants: every station has one concrete output contract, and every node must produce, control, visualize, or measure that output. Station footprints stay 1 × 1 or 2 × 2. Tiles, resources, and image paths are allowlisted, and station state is capped at 24 keys.
• Lazy resources: /blog-images.json contains safe image paths and component metadata, not image bytes or component source. Compact text and post metadata load separately; /llms-full.txt and image bytes load only when a request or open station needs them. Do not import arbitrary blog components into the React island to extend the catalog.
• Worker protocol: the host and worker share WORKER_PROTOCOL_VERSION, and the host validates worker:ready before loading pinned models. When a message shape changes, bump both protocol constants. Keep model progress, errors, token metrics, and binary audio out of persisted lab state.
• Safe edits: change the smallest owning module, update schemas and reducer validation together, add round-trip coverage for persisted-state changes, then run npm run test:local-agent-lab, Prettier on touched files, and npm run build. The harness is repository-only and is not shipped to visitors.

Local Persistence and Offline Scope

• Local storage: matt-local-agent-lab-storage saves Matt’s position, stations and their layouts, loose tiles, connections, positioned printed outputs, and creative history in the current browser profile. matt-local-agent-token-usage separately stores the cumulative local token total. Conversation, activities, and undo history are session-only; failed writes leave the open lab usable in memory.
• Offline boundary: once the tab, worker, models, and any needed blog text or image bytes are loaded, local inference, speech recognition, optional speech synthesis, and lab edits can continue if connectivity drops. Completed model files use Transformers.js’s browser cache, not localStorage. The lab does not install an offline-cache service worker; Partytown’s production worker is unrelated to offline caching, so offline reloads and uncached worker, runtime, model, blog, or image assets are not guaranteed.

Build breakdown

• Bundle: about 951 KiB of feature-owned production assets: 574 KiB of sprites and media plus 377 KiB of UI JavaScript, CSS, and worker code; the UI, styles, and worker compress to about 107 KiB over gzip.
• Code: 31 changed files and about 1.54 MB in the branch; about 30,100 added lines of code and styles, or 24,100 without the 5,979-line repository-only test harness.
• Astro: the client:only="react" island loads only on this post route, mounting while the article is open so model downloads can run in the background.
• Models: about 1.9–2.3 GB in total, depending on the agent quantization, including Whisper Tiny and the optional local voice.