notes

one project per post — architecture, bugs, numbers

Eight A100s, $61, and 124M parameters

Full reproduction of GPT-2 124M on rented multi-GPU hardware. Val loss 3.40 vs OpenAI's 3.29 (97% match), HellaSwag 27% vs 29.45%, in 2.5 hours of training.

18 min read · 2026

Birkhoff in 8.7 KB

An 8.71 KB prompt for SAIR's equational-theories competition (Tao + Davis, follow-up to Honda-Murakami-Zhang 2025). Replace free-form LLM reasoning with a 9-magma Birkhoff-sound decision procedure. A 31B model running this prompt beat a 120B one on the hardest set.

16 min read · 2026

micrograd: a scalar-valued autograd engine

A 150-line autograd engine that supports +, *, **, tanh, exp, and a tiny MLP on top.

6 min read · April 01, 2026

2026 · autograd backprop · deep-learning
A transformer that reads C++ and writes Python

16.4M-parameter encoder-decoder transformer for C++ → Python code translation, trained on XLCoST on a 4 GB GPU. val_loss 2.0474.

15 min read · March 01, 2026

2026 · transformer translation code · deep-learning