I spent 131 runs tuning a 9-layer language model under a hard 600-second cap on a single RTX 4000 Ada. The biggest gains did not come from clever regularizers. They came from making the model simpler, faster, and more decoder-heavy.
OpenAI's Parameter Golf is simple to describe and nasty to optimize. Train a tiny language model, keep the compressed artifact under 16MB, and finish inside a ten-minute wallclock budget. The leaderboard metric is bits per byte. Lower is better.
I was not training on the official 8xH100 submission setup. I used a much cheaper proxy: one RTX 4000 Ada on RunPod, with the same 600-second cap. That gave me a baseline 9L/512D model at 1.6117 BPB. From there I ran 131 experiments and tracked every one.
That constraint changed how the whole search behaved. In ordinary model design, you ask what adds capacity. In Parameter Golf, the first question is harsher: what buys you more useful steps before the clock runs out?
The first half of the project was a graveyard of tricks that sounded smart and mostly made the model worse.
Z-loss, token dropout, embed mixup, logit penalties, and label smoothing either did nothing or made BPB worse. Standard cross-entropy was already hard to beat.
Canopy scaling, mycorrhizal gradient sharing, thermal vent boosting, predator-prey schedules. These were the romantic ideas. They were also the fastest route to terrible runs.
Deeper models, wider models, and 3x MLP widths all lost training steps. On a fixed clock, that erased the theoretical gain before the run even finished.
Z-loss 1e-4 1.6282 Cosine 2-cycle 1.7383 Canopy Light 1.8587 Predator-Prey LR 1.7981 Weight decay schedule 0.04 1.6569 3x MLP + pure decoder 1.5474 # slower, fewer steps, worse overall
That pattern kept repeating: if a trick made optimization noisier, or the model slower per step, it usually lost. The challenge was quietly telling me to stop decorating the baseline and start removing friction.
The best changes did not feel random. They all made the model spend more of its ten minutes on useful learning instead of overhead.
| Change | What it did | Best local result |
|---|---|---|
| Parallel blocks | Attention and MLP ran in parallel, which made the model both faster and better. This was the first big jump. | 1.5600 BPB |
| Decoder-heavy splits | Moving layers out of the encoder helped monotonically. 1/8 beat 2/7, 2/7 beat 3/6, and pure decoder ended up best locally. | 1.5377, then 1.5207 |
| GQA with 2 KV heads | This was a double win: better BPB and faster training, which raised the run from 265 to 276 steps under the same cap. | 1.5329 BPB |
| Untied embeddings | The late surprise. Separating input and output embeddings gave one of the biggest single gains after the structural changes were already in place. | 1.5211 BPB |
| Tidal LR, SC15, RoPE 5000, Matrix LR 0.11 | These were the tuning moves that kept the faster architecture pointed in the right direction. | Part of final best run |
The cleanest interpretation is that this tiny model did not want to be more elaborate. It wanted to be more direct. Fewer encoder layers. Cleaner routing. Less parameter sharing between input and output. More time spent taking optimizer steps.
The progression looked less like a fireworks show of new tricks and more like peeling away parts that were not earning their keep.
By the end, the best recipe in this study looked less like a bag of hacks and more like a coherent answer to the constraint.
ITERATIONS=400 TIDAL_LR=1 LOGIT_SOFTCAP=15.0 ROPE_BASE=5000 PARALLEL_BLOCK=1 MLP_ACT=silu2 HEAD_DIVERSITY=1e-4 EMBED_LR=0.8 MATRIX_LR=0.11 ENCODER_LAYERS=0 NUM_KV_HEADS=2 TIE_EMBEDDINGS=0 # Result on RTX 4000 Ada proxy: # 1.5207 BPB in 600 seconds
That run finished at roughly 14MB after int8 + zlib compression, which left around 2MB of headroom under the 16MB budget. The model did not improve by becoming huge, exotic, or fragile. It improved by wasting less of its budget.
If you only remember one thing from this project, make it this: under a hard training cap, systems decisions and architecture decisions collapse into the same problem. A change that makes each step cheaper can be as important as a change that makes each step smarter.
That is why so many normal intuitions flipped. More layers were not obviously better. Fancier optimization was usually harmful. The best tiny model in this study kept improving as it became more decoder-heavy, more parallel, and less tied together.
Parameter Golf looks like a compression contest from the outside. In practice it feels more like a lesson in restraint. The model got better when I stopped asking it to do everything.
Experiment repo →