WorldTUI: a 360M model that pretends to be a terminal — the full build notes

What the source published

Developer SimEdw published WorldTUI on his personal blog: a 360M-parameter language model whose only job is to pretend to be a terminal. There is no real shell and no filesystem behind it — every keypress goes in, and the model predicts how the screen changes. It can create files, run vim and tmux, play a game of snake, and usually remembers what you wrote a few minutes earlier. The author says the two biggest wins were teaching the model to repair its own mistakes and decoding screen rows in parallel.

The idea started when the author saw Google's Genie 3 and began imagining applications that are learned rather than written: instead of a game engine rendering the scene, a model predicts the next most likely frame from the user's inputs. Without a GPU cluster or a pile of video data, he scaled the ambition down — not a game world, a terminal. A terminal turned out to be a good fit: it is just a grid of characters rather than pixels, perfectly labelled training data can be generated by running real shells, and a pretrained model already knows what ls or cat should print. The real question was whether the model could maintain a consistent world, not just generate convincing terminal output.

There is a playable web demo: the model is quantized to int4, 208MB, and runs entirely in the browser tab. A desktop browser with WebGPU and a physical keyboard is recommended — Chrome or Edge works best, while a modern phone loads it but painfully slowly. The author's invitation is to go try to break it, starting with typing pythoon on purpose to confirm it is not a real shell.

Sources: [1] [2]

Try it: break the fake terminal in this order

1. Open the demo link in the original post (Try it) and wait for the 208MB download — everything runs locally in the tab, nothing is uploaded. Use desktop Chrome or Edge.
2. Test its memory: echo a random string into a file, run a few other commands in between, then cat the file back and see whether it remembers.
3. Push into fullscreen apps: switch vim modes with esc, save and quit; open two tmux panes and jump between them; eat a few pellets in snake.
4. Probe its temper: type pythoon instead of python to check the typo tolerance; nest a few directories and cd back out to see whether it gets lost; combine two commands it has probably never seen together and watch it break.
5. To replicate the recipe: record real programs keypress by keypress inside Docker → model the terminal as a character grid with a 333-token vocabulary → start with supervised fine-tuning → run three rounds of self-correction training → decode rows in parallel for speed → quantize to int4 for WebGPU.

Sources: [1]

The full build: data, training, speed, shrink

Data is the foundation of the whole thing. Finding no existing dataset with precise keystroke-to-screen alignment, the author built his own: run real programs inside Docker and record every keypress plus the screen after each one. The recipe: randomize usernames, hostnames, prompt styles and terminal sizes; generate directories and files from fictional personas (developer, sysadmin, and so on); pick 6 to 24 weighted activities per round (editing files, vim, python, tmux, and more); track state so sessions mostly open files that actually exist; type like a person, one keypress at a time with tab completion, at a 1.2% typo rate; start 35% of sessions partway through, forcing the model to take over from any state. The final dataset was 336k episodes, roughly 2B tokens — for comparison, the base model saw around 18T tokens in pretraining.

For modeling, the author treats the terminal as a grid of characters. Every session starts with a snapshot: screen size, contents and cursor position; the size is part of the input (mostly 32x96, with some 24x80 and 40x120), and the demo even allows a custom starting screen. Each keypress then goes in as the raw bytes a real terminal would send (Enter as a newline byte, left arrow as ESC [ D), and the model answers with which rows to rewrite and where the cursor lands; with no input it gets a WAIT token with a duration, which is how snake keeps moving on its own. A COPY operation saves tokens: instead of rewriting every row on scroll (about 3,000 tokens), it issues copy instructions like old row 1 becomes new row 0, costing three tokens per row. The whole vocabulary is 333 tokens — 256 byte tokens, 64 row tokens and a handful of special operations — with grammar-constrained decoding so every prediction is a valid screen update.

Training came in two big steps. First, supervised fine-tuning on Qwen2.5-Coder-0.5B (a coding model that had already seen plenty of CLI usage): given screen history plus a keypress, predict the screen change a real terminal would produce. Ordinary typing was learned quickly, but errors compounded in long sessions — one wrong letter in a filename poisoned everything after. So the author borrowed from DAgger for self-correction: pick a point in a recorded session, let the model predict 1 to 4 events ahead, compare the resulting screen with the real recording, and when the difference is small enough, construct a correction that repairs the model's screen on the next keypress, then keep training on the corrected frame and subsequent real events — three rounds, each time re-sampling mistakes from the latest model. The key restriction: words introduced by a correction must have appeared on an earlier real screen or in the user's typing, never invented by the model, or training would reward hallucinations. Together with extra data, this lifted PresetBench from 24/47 to 38/47, at a total of roughly 75 RTX 4090 GPU-hours.

Speed was the other battle. A full redraw is about 3,000 tokens, which took 12 seconds on an RTX 4090 when generated one at a time. Noticing that most rows do not depend on each other, the author switched to parallel row decoding: one batch element per row, all sharing the same history (including the KV cache), each generating only its own update — cutting the cost from the sum of all rows to the longest row, about 100 steps, and dropping large-frame p95 latency from 6.7s to 0.87s. Rows that cannot coordinate produce weirdness (vim opening on row 3 instead of row 0, snake growing several food pellets at once), fixed with speculative decoding using the same model: the parallel rows draft, the model verifies sequentially, keeping tokens up to the first disagreement. In the browser on an M4 Mac, typing takes about 60ms per character and a big update like man drops from 2.5s to 0.9s. For the shrink: keeping only the embeddings for the 333 used tokens (the original 152k-token table cost 136M parameters) took the model from 494M to about 360M parameters, then int4 quantization brought it to 208MB. The web demo uses a hand-written JavaScript Qwen decoder with WGSL compute shaders on WebGPU, unpacking 4-bit weights at inference time with 32-bit activations, all local to the tab. The distillation experiment was instructive: removing layer 7 barely moved next-token accuracy (99.3% to 98.9%) but crashed the memory test from 64/64 to 0/64 — the student looked fine while quietly losing its memory, so distillation was abandoned.

Three test suites: ShellBench has 106 commands across 24 handwritten bash sessions scored by exact output; PresetBench has 47 scripted cases collected from demo failures; the memory test has 32 worlds, each writing two random values to two files and reading them back in separate sessions. WorldTUI scored 80/106 on ShellBench (prompted baselines got 52 and 60) and 9/13 on memory readbacks. The author is equally frank about the limits: the Docker containers had no internet access, so tools like lynx are boring; Python handles only simple operations and hallucinates GPT-3-style answers on harder ones; nested directories can leave it lost; and the demo runs about 20x slower in Firefox than in Chrome.

Sources: [1] [2]

Key takeaways

• WorldTUI is a 360M-parameter model that plays terminal: no real shell, keypresses in, screen changes out; it creates files, runs vim and tmux, and plays snake.
• All data is synthetic: keypress-by-keypress recordings of real programs in Docker, 336k episodes and about 2B tokens, with a 1.2% typo rate and 35% mid-session starts.
• The training trick is three rounds of self-correction (DAgger-style): predict ahead, compare with the real screen, learn to repair on the next keypress; PresetBench rose from 24/47 to 38/47.
• Parallel row decoding cut large-frame latency from 6.7s to 0.87s, with same-model speculative decoding fixing cross-row coordination; int4 quantization gives a 208MB model that runs locally in the browser.
• Distillation lesson: removing one layer barely touched token accuracy yet crashed the memory test from 64/64 to 0/64 — a small model's memory lives in specific layers, so test memory before pruning.
• Limits: no network access, Python only handles simple tasks, nested directories can confuse it; the demo is fastest on desktop Chrome/Edge and about 20x slower in Firefox.

Sources: [1]