Downloads
By the numbers
To be era authentic, a model like this could be trained on 120 PlayStation 2s in 60 days. That training would take 8,700 kWh of electricity ($1,600 at today's rate). To put that into context, $1,600 is 2,800 Chicken McNuggets.
The model, layer by layer
ChatGBA reads and writes bytes. There is no tokenizer: each UTF-8 byte is one token. A conversation is plain text with three special markers: <|u|> starts a user message, <|a|> starts an assistant message, and <|e|> ends either one. A conversation might look like <|u|>Hi<|e|><|a|>Hello!<|e|>, and a reply ends when the model writes <|e|> itself.
Why each piece is there
| Piece | What it buys a model this small |
|---|---|
| Pre-norm (RMSNorm before each half) | Keeps the numbers in a predictable range going into every matrix, which deep stacks and fixed-point math both need. RMSNorm skips the mean subtraction of LayerNorm: cheaper, and it works as well. |
| Canon layers | Bytes are tiny tokens: a word is 5 or 6 of them. A 4-tap filter lets every position see its last few neighbours for almost no parameters (768 per block), so attention does not have to spend heads on "what letter came before me". |
| QK-norm | Normalizing queries and keys keeps attention scores from growing without bound during training, and on the GBA it bounds the numbers the integer engine has to handle. |
| RoPE | Position as a rotation means attention depends only on the distance between two bytes. That is what lets the context slide: past 512 bytes the oldest byte falls out and nothing else changes. |
| Grouped-query attention | 6 query heads but only 2 key/value heads: the cache that remembers the conversation is a third the size. On a machine with 256 KB of work RAM, that is the difference between fitting and not. |
| Short attention windows | 8 of the 10 blocks only look 64 bytes back; 2 see everything. Local blocks are cheap and their cache is small; the 2 global ones carry the long-range conversation. |
| SwiGLU | A gated MLP learns more per parameter than the classic two-matrix one; 512 hidden units keeps the total near 4M. |
| Tied embeddings | The 256 × 192 table is used twice, in and out: 49,152 parameters instead of 98,304. |
Where the 4,003,264 parameters are
| Tensor | Shape | Per block | All 10 blocks |
|---|---|---|---|
| Attention norm, MLP norm | 2 × [192] | 384 | 3,840 |
| Canon A, Canon C | 2 × [4, 192] | 1,536 | 15,360 |
| WQ, WO | [192, 192], [192, 192] | 73,728 | 737,280 |
| WK, WV | [192, 64], [192, 64] | 24,576 | 245,760 |
| Query norm, key norm | [6, 32], [2, 32] | 256 | 2,560 |
| Wgate, Wup, Wdown | [192, 512] × 2, [512, 192] | 294,912 | 2,949,120 |
| Block total | 395,392 | 3,953,920 | |
| Embedding (tied) | [256, 192] | 49,152 | |
| Final norm | [192] | 192 | |
| Total | 4,003,264 |
How it fits on a Game Boy Advance
The GBA has a 16.78 MHz ARM7TDMI, no floating point, no divide instruction, 32 KB of fast RAM and 256 KB of slow RAM. The model has to live in the cartridge and run in integers.
| Problem | What the cartridge does |
|---|---|
| 16 MB of float weights | Every matrix becomes int8 with one scale per output row (a 4× shrink, 4.0 MB), read straight from cartridge ROM. Each byte is stored offset by 128 so one AND instruction unpacks it. |
| No floating point | Everything is fixed point: activations are 32-bit integers with 16 fractional bits ("Q16"). Inputs to a matrix are rounded to 13 bits with a shared power-of-two shift, so int8 × 13-bit products add up safely in 32 bits. |
| exp, sigmoid, sin, cos | Lookup tables in the ROM: 2−i/256 for softmax, the sigmoid for SiLU, and RoPE's angles for 512 positions. Past position 511, a per-frequency "anchor" rotation turns by 512 steps at a time, so the context can slide forever. |
| Remembering the conversation | An int8 key/value cache with one scale per vector, in rings: 64 positions for the windowed blocks, 512 for the two global ones. 196 KB of the 256 KB. |
The 256 KB of external work RAM: the cache, the canon layers' last three inputs per block, the chat screen's text, and what is left.
The result costs about 0.001 bits per byte of accuracy against the float model, and the cartridge in an emulator gives byte-for-byte the same replies as the same integer engine compiled for a Mac. Each byte takes about 37 million cycles: 2.2 seconds.
How it was trained
| Stage | Data | How |
|---|---|---|
| Pretraining | All of Cosmopedia-v2: 39.1 million synthetic textbook pages, stories and articles, 147 GB. One pass. | 512-byte windows, 128 per step, 2.2 million steps. Documents shuffled; 1 in 10,000 held out for validation and another for the test. |
| Fine-tuning | smol-smoltalk: 458,000 conversations. | 512-byte windows starting every 256 bytes, so every conversation is also seen cut off at the top, the way a sliding context shows it. Only the assistant's bytes are graded. |
The optimizer is Muon for the attention and MLP matrices (it orthogonalizes each update, so every direction of a matrix learns at the same rate) and AdamW for the rest. The learning rate warms up, stays flat for most of the run, then falls in a straight line to zero over the last 10%. Weights are float32; the GPU multiplies in TF32.