LQ
RE
ChatGBA
PRESS ANY KEYor click to power on
POWER
TURBO
T
W
A
D
S
SELECT
START
SPACE
ENTER
BJ
AK

At real Game Boy speed, ChatGBA takes about 2 seconds per letter. If you would rather not wait four minutes for a paragraph, flip the TURBO switch (T) at the top left. Try JavaScript inference here.

Downloads

The cartridge
4.03 MiB
Quantized weights
4.01 MiB
Full weights
15.3 MiB

By the numbers

To be era authentic, a model like this could be trained on 120 PlayStation 2s in 60 days. That training would take 8,700 kWh of electricity ($1,600 at today's rate). To put that into context, $1,600 is 2,800 Chicken McNuggets.

Parameter count
4,003,264
Model size
4.01 MiB
Context length
512 bytes
Corpus size
139 GiB

The model, layer by layer

ChatGBA reads and writes bytes. There is no tokenizer: each UTF-8 byte is one token. A conversation is plain text with three special markers: <|u|> starts a user message, <|a|> starts an assistant message, and <|e|> ends either one. A conversation might look like <|u|>Hi<|e|><|a|>Hello!<|e|>, and a reply ends when the model writes <|e|> itself.

normalization mixing nearby positions attention MLP residual
Byte in
one of 256 values
Embedding
look up row [byte] of a 256 × 192 table → x, feature dimension: 192
BLOCK × 10
RMSNorm
a = x / √(mean(x²) + ε) × γ
Canon A
a += w₀·a[t] + w₁·a[t−1] + w₂·a[t−2] + w₃·a[t−3], one 4-tap filter per feature
Q, K, V projections (grouped-query attention)
q = a·WQ: 6 query heads × 32. k = a·WK, v = a·WV: 2 key/value heads × 32. 3 q heads share 1 kv head
QK-norm
RMSNorm on every head's query and key, each with its own learned 32 scales
RoPE
rotate each pair (2i, 2i+1) of q and k by position × 10000−2i/32: position enters here, as an angle
Attention, grouped
softmax(q·k / √32) over earlier positions; 3 query heads share each key/value head. Blocks 1–4 and 6–9 look back 64 bytes, blocks 5 and 10 the whole 512
Output projection
6 heads × 32 → 192 (WO)
Residual add
x = x + attention's output
RMSNorm
m = RMSNorm(x) × γ₂
Canon C
another 4-tap causal filter, on the MLP's input
SwiGLU MLP
silu(m·Wgate) × (m·Wup), 192 → 512, then ·Wdown 512 → 192. No biases
Residual add
x = x + the MLP's output, and on to the next block
Final RMSNorm
x / rms(x) × γ
Unembedding (tied)
scores = x · Embeddingᵀ
Next byte
temperature 0.7

Why each piece is there

PieceWhat it buys a model this small
Pre-norm (RMSNorm before each half)Keeps the numbers in a predictable range going into every matrix, which deep stacks and fixed-point math both need. RMSNorm skips the mean subtraction of LayerNorm: cheaper, and it works as well.
Canon layersBytes are tiny tokens: a word is 5 or 6 of them. A 4-tap filter lets every position see its last few neighbours for almost no parameters (768 per block), so attention does not have to spend heads on "what letter came before me".
QK-normNormalizing queries and keys keeps attention scores from growing without bound during training, and on the GBA it bounds the numbers the integer engine has to handle.
RoPEPosition as a rotation means attention depends only on the distance between two bytes. That is what lets the context slide: past 512 bytes the oldest byte falls out and nothing else changes.
Grouped-query attention6 query heads but only 2 key/value heads: the cache that remembers the conversation is a third the size. On a machine with 256 KB of work RAM, that is the difference between fitting and not.
Short attention windows8 of the 10 blocks only look 64 bytes back; 2 see everything. Local blocks are cheap and their cache is small; the 2 global ones carry the long-range conversation.
SwiGLUA gated MLP learns more per parameter than the classic two-matrix one; 512 hidden units keeps the total near 4M.
Tied embeddingsThe 256 × 192 table is used twice, in and out: 49,152 parameters instead of 98,304.

Where the 4,003,264 parameters are

TensorShapePer blockAll 10 blocks
Attention norm, MLP norm2 × [192]3843,840
Canon A, Canon C2 × [4, 192]1,53615,360
WQ, WO[192, 192], [192, 192]73,728737,280
WK, WV[192, 64], [192, 64]24,576245,760
Query norm, key norm[6, 32], [2, 32]2562,560
Wgate, Wup, Wdown[192, 512] × 2, [512, 192]294,9122,949,120
Block total395,3923,953,920
Embedding (tied)[256, 192]49,152
Final norm[192]192
Total4,003,264

How it fits on a Game Boy Advance

The GBA has a 16.78 MHz ARM7TDMI, no floating point, no divide instruction, 32 KB of fast RAM and 256 KB of slow RAM. The model has to live in the cartridge and run in integers.

ProblemWhat the cartridge does
16 MB of float weightsEvery matrix becomes int8 with one scale per output row (a 4× shrink, 4.0 MB), read straight from cartridge ROM. Each byte is stored offset by 128 so one AND instruction unpacks it.
No floating pointEverything is fixed point: activations are 32-bit integers with 16 fractional bits ("Q16"). Inputs to a matrix are rounded to 13 bits with a shared power-of-two shift, so int8 × 13-bit products add up safely in 32 bits.
exp, sigmoid, sin, cosLookup tables in the ROM: 2−i/256 for softmax, the sigmoid for SiLU, and RoPE's angles for 512 positions. Past position 511, a per-frequency "anchor" rotation turns by 512 steps at a time, so the context can slide forever.
Remembering the conversationAn int8 key/value cache with one scale per vector, in rings: 64 positions for the windowed blocks, 512 for the two global ones. 196 KB of the 256 KB.
KV cache 192 KB
canon 22.5 KB
text
free

The 256 KB of external work RAM: the cache, the canon layers' last three inputs per block, the chat screen's text, and what is left.

The result costs about 0.001 bits per byte of accuracy against the float model, and the cartridge in an emulator gives byte-for-byte the same replies as the same integer engine compiled for a Mac. Each byte takes about 37 million cycles: 2.2 seconds.

How it was trained

StageDataHow
PretrainingAll of Cosmopedia-v2: 39.1 million synthetic textbook pages, stories and articles, 147 GB. One pass.512-byte windows, 128 per step, 2.2 million steps. Documents shuffled; 1 in 10,000 held out for validation and another for the test.
Fine-tuningsmol-smoltalk: 458,000 conversations.512-byte windows starting every 256 bytes, so every conversation is also seen cut off at the top, the way a sliding context shows it. Only the assistant's bytes are graded.

The optimizer is Muon for the attention and MLP matrices (it orthogonalizes each update, so every direction of a matrix learns at the same rate) and AdamW for the rest. The learning rate warms up, stays flat for most of the run, then falls in a straight line to zero over the last 10%. Weights are float32; the GPU multiplies in TF32.