I spent three days trying to make a 9-billion-parameter model run twice as fast on a MacBook Air. Here is what the wall looks like when you hit it.
Every experiment followed the same loop. Idea, code, measure, compare to baseline, accept or reject. Most rejected.
We started with a clear baseline: Qwen3.5-9B at 4-bit quantization, running through mlx_lm with a proper KV cache. That gave us 16.35 tokens per second. Anything that did not beat that number - or that destroyed quality - went in the reject pile.
Not all layers work equally hard. Early layers are almost asleep. Late layers are wide awake. We measured this directly.
The profiler runs prompts through the model and counts how many MLP neurons output near-zero for each token at each layer. The result is a heatmap with a clear pattern.
Layer 0 shows 99.75% sparsity at threshold 0.1. Layer 31 shows 5.58%. The gradient is smooth and monotonic. The model does almost nothing in its first layers and everything in its last layers.
Here is the catch: this is activation sparsity, not weight sparsity. The neurons are quiet for this particular input, but the weights still have to be loaded from memory. You cannot skip a layer just because its neurons are asleep.
Layer 0: 99.75% sparsity (threshold 0.1) Layer 5: 85.37% sparsity Layer 10: 66.42% sparsity Layer 15: 60.49% sparsity Layer 20: 31.82% sparsity Layer 25: 22.23% sparsity Layer 31: 5.58% sparsity Global average: 52.11%
We deleted each layer one at a time and watched what happened. Almost nothing survived.
The profiler replaces a single layer with an identity pass, runs the full model, and compares the output to baseline. If a layer were redundant, the output would barely change.
Only layer 0 is truly sacred. Remove it and the model outputs gibberish. But even the "most expendable" layers only preserve about 60% of tokens when removed. There is no layer you can safely drop without paying a quality cost.
This rules out depth-adaptive inference for Qwen3.5. The model is tightly packed. Every layer contributes something. There is no fat to trim.
If you could exit early when the model is "sure," you would save time. But the model is never sure until the end.
For each token generated, we sampled the model's prediction at layers 3, 7, 11, 15, 19, 23, and 27. Most tokens do not stabilize until layer 23 or 27. Some bounce around until the very last layer.
This is why adaptive early-exit failed. We set checkpoints at layers 8, 16, and 24. The model was almost never confident enough to exit early. It just ran full depth every time, plus the overhead of checking.
Every experiment, measured honestly. The baseline is 16.35 tok/s.
The cascade routes easy prompts to a 0.8B model and hard prompts to the 9B model. It is the only technique that delivered a real speedup. Everything else either broke even or made things worse.
Train small heads to predict future tokens from an intermediate layer. In theory, you verify drafts in parallel and go fast. In practice, the heads learn to memorize.
We trained two versions. The first used 600 examples from 20 prompts. Validation accuracy looked good at 47%, but real acceptance was only 56% because the heads had memorized the training distribution.
The second version used 2,000 examples. Validation accuracy rose to 59%. Then we tested on held-out prompts and saw 14% real acceptance. The model was generating drafts, verifying them, and rejecting almost all of them. The overhead of verification made it slower than the baseline.
Baseline: 16.35 tok/s Medusa v1: 18.32 tok/s (1.12×) - 56% drafts accepted Medusa v2: 4.91 tok/s (0.30×) - 14% drafts accepted
The published Medusa papers use 50,000+ diverse prompts. We used 20. The gap between academic scale and laptop scale is the difference between a technique that works and one that does not.
A Mixture-of-Experts model has 35 billion parameters but only activates 3 billion per token. It should be faster. It is 850 times slower.
I downloaded the Qwen3.5-35B-A3B model. It has 256 experts across 40 layers and activates only 8 per token. In theory, that means less memory bandwidth per forward pass. In practice, the routing overhead dominates.
Five tokens took six minutes. The routing computation and scattered memory access destroyed performance.
The dense model reads one contiguous block of weights. The MoE jumps between 8 different blocks per layer.
On Apple Silicon, scattered memory access kills you. The dense model reads one contiguous block of weights. The MoE model jumps between 8 different expert blocks per layer, and the routing computation itself is not fused efficiently in the MLX implementation.
Every token requires reading almost all 5.5GB of model weights from memory. The M4 Air has about 100 GB/s of bandwidth. Do the division.
The math is brutal. 5.5 gigabytes of weights divided by 100 gigabytes per second equals 18 tokens per second in the absolute best case. Real overhead brings that down to 14–16 tok/s. No algorithmic trick can change how fast electrons move through a memory bus.
This is why every experiment hit the same ceiling. Not because the code was wrong. Because the hardware is the hardware.
We did not find a secret speedup. But we mapped the exact shape of the problem. Qwen3.5-9B has no expendable layers, no early-exit points, and no dead weights. It is a tightly packed architecture where every component earns its keep.
The real contribution is the tooling. Three profilers that can analyze any MLX model: MLP sparsity, prediction stability, and layer importance. They are not glamorous, but they turn guesswork into measurement. That is where the next breakthrough will come from. Not from skipping layers, but from understanding them.
Code on GitHub →