One algorithm, five programs
Python: microgpt.py
A scalar autograd class, a 1-layer, 4-head GPT, Adam and a sampler, over lists of floats.
149 code lines · 64.4 s · loss 2.28
Rust: microgpt-rs
The same algorithm and shape: a tape-based scalar autograd, no crates, compiled.
390 code lines · 0.589 s · loss 2.36
microgpt.mlpl
microgpt.py line by line with hand-written layers. With --rs-parity it replays
microgpt-rs's random stream and its output is byte-identical to the Rust.
230 code lines · 0.718 s · loss 2.37
aline, garien, anisn, alilia, thayn
microgpt-idiomatic.mlpl
The same data regime, built from the Model DSL: embed, causal_attention,
residual, chain, adam over models, sample.
Faster than compiled Rust.
48 code lines · 0.468 s · loss 2.47
jiafini, tamesin, jali, kair, tasen
microgpt-compact.mlpl
The smallest honest version: the corpus as one token stream in 16-token windows, one DSL chain, KV-cached sampling.
32 code lines · 0.539 s · loss 2.56*
an, cren, yn, arialilin
Wall time for the whole program (load, train 1000 steps, sample 20 names) on an Apple M1 Max, median of 7 runs (CPython: one run). Loss: mean over the last 100 training steps. *The compact variant trains on 16-token windows, so its loss is not strictly comparable. Code lines exclude comments, blank lines and docstrings.
Results
The idea in one expression
Where microgpt.py spends a class on autograd and loops over scalars, the idiomatic MLPL model is one expression, and training is one call per step:
body = chain(rms_norm(d),
residual(chain(rms_norm(d), causal_attention(d, 4, 3))),
residual(chain(rms_norm(d), linear(d, 4 * d, 4), relu_layer(), linear(4 * d, d, 5))),
linear(d, V, 6));
train 1000 {
adam(cross_entropy(u:logits(inp), tgt), [tok, pos, body], 0.01 * (1 - step / 1000), 0.85, 0.99, 1e-8)
};
Literate programs
Each is an Org document run through ob-mlpl: every block's output is real, each section
states its math, and the model's functions carry their equations as @formula annotations. The
program blocks tangle to a script whose output is checked against that variant's baseline, so the prose
cannot drift from the code.
What we learned
- Array primitives are the efficiency. The idiomatic MLPL interpreter run beats compiled Rust because a DSL layer is one native array op, while the Rust port keeps microgpt's per-scalar tape.
- Exact cross-language parity is possible. microgpt-rs's SplitMix64 RNG, reimplemented in pure MLPL with 16-bit limbs, makes the faithful port's output byte-identical to the Rust.
- Readability and speed trade off case by case. Hand-written equations vs DSL layers, a mask built inside the loss vs passed in, per-step reads vs pre-encoding: each choice is measured, so it can be made on the numbers (comparison, section 5).
- The interpreter shapes the code. Every u:call copies the globals and large reads copy arrays, so the data is pre-encoded and the corpus expunged before training.
- What would make MLPL shorter still: format,gather/slice, destructuring, param records, copy-on-write values, a cacheable position layer, and compiled inference. See the requests and the idioms guide.