OpenAI Parameter Golf · 131 Experiments

A tiny model got better
by doing less.

I spent 131 runs tuning a 9-layer language model under a hard 600-second cap on a single RTX 4000 Ada. The biggest gains did not come from clever regularizers. They came from making the model simpler, faster, and more decoder-heavy.

131 runs RTX 4000 Ada 600s wallclock 1.6117 → 1.5207 BPB ~14MB final artifact
The Setup

This challenge turns speed into model quality.

OpenAI's Parameter Golf is simple to describe and nasty to optimize. Train a tiny language model, keep the compressed artifact under 16MB, and finish inside a ten-minute wallclock budget. The leaderboard metric is bits per byte. Lower is better.

I was not training on the official 8xH100 submission setup. I used a much cheaper proxy: one RTX 4000 Ada on RunPod, with the same 600-second cap. That gave me a baseline 9L/512D model at 1.6117 BPB. From there I ran 131 experiments and tracked every one.

That constraint changed how the whole search behaved. In ordinary model design, you ask what adds capacity. In Parameter Golf, the first question is harsher: what buys you more useful steps before the clock runs out?

Key finding: Under a hard wallclock cap, faster training is not just an engineering convenience. It is part of the architecture. Extra steps often beat extra complexity.
1.6117Baseline BPB
1.5207Best BPB
-0.091Gain vs baseline
276Steps in 600s
What Failed

Most of the clever ideas were dead ends.

The first half of the project was a graveyard of tricks that sounded smart and mostly made the model worse.

Regularization mostly hurt

Z-loss, token dropout, embed mixup, logit penalties, and label smoothing either did nothing or made BPB worse. Standard cross-entropy was already hard to beat.

Gradient tricks were worse

Canopy scaling, mycorrhizal gradient sharing, thermal vent boosting, predator-prey schedules. These were the romantic ideas. They were also the fastest route to terrible runs.

Bigger often meant slower, then worse

Deeper models, wider models, and 3x MLP widths all lost training steps. On a fixed clock, that erased the theoretical gain before the run even finished.

selected misses from the log
Z-loss 1e-4                  1.6282
Cosine 2-cycle               1.7383
Canopy Light                 1.8587
Predator-Prey LR             1.7981
Weight decay schedule 0.04   1.6569
3x MLP + pure decoder        1.5474  # slower, fewer steps, worse overall

That pattern kept repeating: if a trick made optimization noisier, or the model slower per step, it usually lost. The challenge was quietly telling me to stop decorating the baseline and start removing friction.

What Worked

The best changes all pushed in the same direction.

The best changes did not feel random. They all made the model spend more of its ten minutes on useful learning instead of overhead.

ChangeWhat it didBest local result
Parallel blocks Attention and MLP ran in parallel, which made the model both faster and better. This was the first big jump. 1.5600 BPB
Decoder-heavy splits Moving layers out of the encoder helped monotonically. 1/8 beat 2/7, 2/7 beat 3/6, and pure decoder ended up best locally. 1.5377, then 1.5207
GQA with 2 KV heads This was a double win: better BPB and faster training, which raised the run from 265 to 276 steps under the same cap. 1.5329 BPB
Untied embeddings The late surprise. Separating input and output embeddings gave one of the biggest single gains after the structural changes were already in place. 1.5211 BPB
Tidal LR, SC15, RoPE 5000, Matrix LR 0.11 These were the tuning moves that kept the faster architecture pointed in the right direction. Part of final best run

The cleanest interpretation is that this tiny model did not want to be more elaborate. It wanted to be more direct. Fewer encoder layers. Cleaner routing. Less parameter sharing between input and output. More time spent taking optimizer steps.

The Story Arc

The project only started moving when the architecture got simpler.

The progression looked less like a fireworks show of new tricks and more like peeling away parts that were not earning their keep.

Step 1
Fix the schedule
Tidal LR was the first clear win. A longer warmup helped the model use the short run better.
1.6117 → 1.5906
Step 2
Make blocks parallel
This was the turning point. Parallel blocks improved quality and increased step count at the same time.
1.5906 → 1.5600
Step 3
Bias hard toward decoding
Every shift away from the encoder helped. 1/8 beat 2/7, and 2/7 beat 3/6. The trend was monotonic.
1.5600 → 1.5377
Step 4
Finish with efficient asymmetry
GQA with 2 KV heads, untied embeddings, and finally a pure decoder squeezed out the last gains without blowing the time or size budget.
1.5341 → 1.5329 → 1.5211 → 1.5207
Best Run

The final configuration is not magical. It is just disciplined.

By the end, the best recipe in this study looked less like a bag of hacks and more like a coherent answer to the constraint.

run_best.sh
ITERATIONS=400
TIDAL_LR=1
LOGIT_SOFTCAP=15.0
ROPE_BASE=5000
PARALLEL_BLOCK=1
MLP_ACT=silu2
HEAD_DIVERSITY=1e-4
EMBED_LR=0.8
MATRIX_LR=0.11
ENCODER_LAYERS=0
NUM_KV_HEADS=2
TIE_EMBEDDINGS=0

# Result on RTX 4000 Ada proxy:
# 1.5207 BPB in 600 seconds

That run finished at roughly 14MB after int8 + zlib compression, which left around 2MB of headroom under the 16MB budget. The model did not improve by becoming huge, exotic, or fragile. It improved by wasting less of its budget.

What This Means

If you only remember one thing from this project, make it this: under a hard training cap, systems decisions and architecture decisions collapse into the same problem. A change that makes each step cheaper can be as important as a change that makes each step smarter.

That is why so many normal intuitions flipped. More layers were not obviously better. Fancier optimization was usually harmful. The best tiny model in this study kept improving as it became more decoder-heavy, more parallel, and less tied together.

Parameter Golf looks like a compression contest from the outside. In practice it feels more like a lesson in restraint. The model got better when I stopped asking it to do everything.

Experiment repo →
Full 131-run experiment log →
OpenAI Parameter Golf PR #1434 →