← ranausmanai
Mechanistic Interpretability

Can a 9B Model Run Twice as Fast?

I spent three days trying to make a 9-billion-parameter model run twice as fast on a MacBook Air. Here is what the wall looks like when you hit it.

MLX Apple Silicon Qwen3.5 Mostly Failed
SCROLL
The Process

How We Ran the Experiments

Every experiment followed the same loop. Idea, code, measure, compare to baseline, accept or reject. Most rejected.

THE EXPERIMENT LOOP - 22 ITERATIONS, 1 SUCCESS

We started with a clear baseline: Qwen3.5-9B at 4-bit quantization, running through mlx_lm with a proper KV cache. That gave us 16.35 tokens per second. Anything that did not beat that number - or that destroyed quality - went in the reject pile.

Finding 1

The Sparsity Gradient

Not all layers work equally hard. Early layers are almost asleep. Late layers are wide awake. We measured this directly.

The profiler runs prompts through the model and counts how many MLP neurons output near-zero for each token at each layer. The result is a heatmap with a clear pattern.

LAYER × TOKEN SPARSITY HEATMAP - DARKER = MORE NEURONS ASLEEP

Layer 0 shows 99.75% sparsity at threshold 0.1. Layer 31 shows 5.58%. The gradient is smooth and monotonic. The model does almost nothing in its first layers and everything in its last layers.

Here is the catch: this is activation sparsity, not weight sparsity. The neurons are quiet for this particular input, but the weights still have to be loaded from memory. You cannot skip a layer just because its neurons are asleep.

mlp_sparsity.py
Layer  0: 99.75% sparsity (threshold 0.1)
Layer  5: 85.37% sparsity
Layer 10: 66.42% sparsity
Layer 15: 60.49% sparsity
Layer 20: 31.82% sparsity
Layer 25: 22.23% sparsity
Layer 31:  5.58% sparsity

Global average: 52.11%
Finding 2

Every Layer Matters

We deleted each layer one at a time and watched what happened. Almost nothing survived.

The profiler replaces a single layer with an identity pass, runs the full model, and compares the output to baseline. If a layer were redundant, the output would barely change.

LAYER ABLATION - EACH BAR SHOWS OUTPUT QUALITY WHEN THAT LAYER IS REMOVED

Only layer 0 is truly sacred. Remove it and the model outputs gibberish. But even the "most expendable" layers only preserve about 60% of tokens when removed. There is no layer you can safely drop without paying a quality cost.

This rules out depth-adaptive inference for Qwen3.5. The model is tightly packed. Every layer contributes something. There is no fat to trim.

Finding 3

Predictions Don't Stabilize Until Layer 29

If you could exit early when the model is "sure," you would save time. But the model is never sure until the end.

TOKEN PREDICTIONS AT EACH LAYER - COLORS SHOW WHEN EACH TOKEN LOCKED IN

For each token generated, we sampled the model's prediction at layers 3, 7, 11, 15, 19, 23, and 27. Most tokens do not stabilize until layer 23 or 27. Some bounce around until the very last layer.

This is why adaptive early-exit failed. We set checkpoints at layers 8, 16, and 24. The model was almost never confident enough to exit early. It just ran full depth every time, plus the overhead of checking.

The Scorecard

What We Tried

Every experiment, measured honestly. The baseline is 16.35 tok/s.

BASELINE 9B
16.4 tok/s
SELF-SPECULATIVE
4.9 tok/s
MEDUSA V2
4.9 tok/s
ADAPTIVE GATES
15.2 tok/s
35B MOE
0.01 tok/s
CASCADE
~45 tok/s

The cascade routes easy prompts to a 0.8B model and hard prompts to the 9B model. It is the only technique that delivered a real speedup. Everything else either broke even or made things worse.

A Cautionary Tale

How Medusa Fooled Us

Train small heads to predict future tokens from an intermediate layer. In theory, you verify drafts in parallel and go fast. In practice, the heads learn to memorize.

MEDUSA DRAFTING - GREEN DRAFTS ACCEPTED, RED REJECTED

We trained two versions. The first used 600 examples from 20 prompts. Validation accuracy looked good at 47%, but real acceptance was only 56% because the heads had memorized the training distribution.

The second version used 2,000 examples. Validation accuracy rose to 59%. Then we tested on held-out prompts and saw 14% real acceptance. The model was generating drafts, verifying them, and rejecting almost all of them. The overhead of verification made it slower than the baseline.

medusa_benchmark.py
Baseline:  16.35 tok/s
Medusa v1: 18.32 tok/s (1.12×) - 56% drafts accepted
Medusa v2:  4.91 tok/s (0.30×) - 14% drafts accepted

The published Medusa papers use 50,000+ diverse prompts. We used 20. The gap between academic scale and laptop scale is the difference between a technique that works and one that does not.

The Surprise

The 35B Model That Runs Slower

A Mixture-of-Experts model has 35 billion parameters but only activates 3 billion per token. It should be faster. It is 850 times slower.

MOE ROUTING - 256 EXPERTS, 8 ACTIVE PER TOKEN

I downloaded the Qwen3.5-35B-A3B model. It has 256 experts across 40 layers and activates only 8 per token. In theory, that means less memory bandwidth per forward pass. In practice, the routing overhead dominates.

0.01
TOKENS / SECOND (35B MoE)

Five tokens took six minutes. The routing computation and scattered memory access destroyed performance.

8.5
TOKENS / SECOND (9B DENSE)

The dense model reads one contiguous block of weights. The MoE jumps between 8 different blocks per layer.

On Apple Silicon, scattered memory access kills you. The dense model reads one contiguous block of weights. The MoE model jumps between 8 different expert blocks per layer, and the routing computation itself is not fused efficiently in the MLX implementation.

The Physics

Memory Bandwidth Is a Hard Ceiling

Every token requires reading almost all 5.5GB of model weights from memory. The M4 Air has about 100 GB/s of bandwidth. Do the division.

THE WALL - EVERY EXPERIMENT HITS THE SAME MEMORY BANDWIDTH CEILING

The math is brutal. 5.5 gigabytes of weights divided by 100 gigabytes per second equals 18 tokens per second in the absolute best case. Real overhead brings that down to 14–16 tok/s. No algorithmic trick can change how fast electrons move through a memory bus.

This is why every experiment hit the same ceiling. Not because the code was wrong. Because the hardware is the hardware.

What the Wall Taught Us

We did not find a secret speedup. But we mapped the exact shape of the problem. Qwen3.5-9B has no expendable layers, no early-exit points, and no dead weights. It is a tightly packed architecture where every component earns its keep.

The real contribution is the tooling. Three profilers that can analyze any MLX model: MLP sparsity, prediction stability, and layer importance. They are not glamorous, but they turn guesswork into measurement. That is where the next breakthrough will come from. Not from skipping layers, but from understanding them.

Code on GitHub →