I served Qwen3.6-27B five ways on a Mac Studio (and my first numbers were wrong)

A few weeks back, Claude Code and I benchmarked which local model to use for coding and the rubric lied to me halfway through. This is the sequel, and it has the same shape: a careful test, a wrong first answer, and a correction that turned out to be the most useful part.

This time the question was narrower. For one model, Qwen3.6-27B, what is the fastest honest way to serve it on Apple Silicon, and does the speculative-decoding hype survive on a machine with real memory bandwidth?

The setup, and who actually built it

The machine was a Mac Studio M3 Ultra with 96GB of unified memory. The contenders were five serving configurations of the same 27B model.

I should be straight about how this got made, the same way I was last time. Claude Code wrote the benchmark harness. I set the direction, picked the tests, and called the methodology when a result looked too clean to trust. That split matters later, because the best finding came out of a correction, not the raw numbers.

Why 8-bit, and not the speed build

Here is a choice that separates this from most MTPLX writeups you will find. The headline speculative-decoding numbers floating around, like the M4 Pro test that hit 18 tok/s, are usually run on the 4-bit speed build. That build is tuned to maximize throughput and gives up some precision to get there.

I did not want the throughput headline. I wanted to know what it costs to run this model well. So every configuration here is an 8-bit quality build. Same precision class across the board. That makes the comparison fair, and it makes the speed numbers more conservative than the ones you usually see, because I am not trading accuracy for a bigger tokens-per-second figure.

What the benchmark actually measured

Two prompts, both with extended reasoning turned off so the model answered instead of monologuing.

The speed prompt was a small, self-contained task, averaged over three warm runs:

Write a complete Python function implementing a binary search tree
with insert, delete, search, and in-order traversal. Include type hints.

The quality prompt was an LRU cache, graded by actually running the output:

Implement a Python class `LRUCache` with:
- __init__(self, capacity: int)
- get(self, key: int) -> int   (return -1 if missing)
- put(self, key: int, value: int) -> None
O(1) time for both. No functools. Output ONLY the class definition.

I did not eyeball the code. The harness executed each model's class against five cases and only counted a pass if all five were green:

1. The canonical LeetCode 146 operation sequence.
2. Evicting the least-recently-used key when capacity is exceeded.
3. Returning -1 for a missing key.
4. Overwriting the value of an existing key.
5. Updating recency on get, so the right key gets evicted later.

The results

Every configuration passed all five cases, so this is a speed and efficiency story, not a quality one. Four findings came out of it.

Finding 1: MTPLX won, but not by the margin smaller Macs report

MTPLX quality led at 34 tok/s, which is what that project is known for. But if you are primed by the popular writeup where a 27B model jumped from 7 to 18 tok/s on an M4 Pro with native MTP, you are expecting something like a 2.6x leap. I did not get one, and remember that number is usually the 4-bit speed build, not the 8-bit I was running.

On the M3 Ultra the speculative-decoding gain was far smaller, and the reason is bandwidth. Speculative decoding buys you fewer round trips to memory per accepted token. On a machine that already feeds the model fast, there is less waiting to remove, so the win compresses. The technique is real everywhere. How big the win feels depends on how starved your hardware was to begin with.

Finding 2: same weights, two runtimes, five tokens per second apart

Run the identical plain MLX 8-bit build on two different servers and one returns 27.3 tok/s while the other returns 22.7. Same model, same quantization, same machine. The serving runtime alone moved the result by almost five tokens a second.

That is a wider swing than the format debate usually admits. People will argue MLX versus GGUF for an afternoon. Far fewer will check whether their MLX runtime is the thing quietly costing them speed, even though here the runtime mattered as much as the format did.

Finding 3: speculative decoding is free speed

On the dedicated MLX server, switching on the MTP draft heads moved the same 8-bit weights from 27.3 to 30.8 tok/s. No accuracy change, both passed all five cases. A clean, controlled 13 percent for flipping one setting with the weights held constant. Free speed is rare. When it shows up this cleanly, take it.

Finding 4: the run I threw out

Here is the correction I promised. My first pass had every configuration loaded in one app at once, and the memory pressure dragged the MLX scores down. It also capped output at 2,000 tokens, which sounds generous until you remember Qwen3.6 thinks out loud before it answers. On that run the model spent its whole budget inside a reasoning block and got cut off before writing a single line of code, so a working model scored as a failure.

Claude, which wrote the harness, flagged the truncation. I flagged the other one, when a configuration that looked far more efficient than the rest turned out to have a thinking budget quietly switched on. We threw the first run out, gave each model the machine to itself, turned off the reasoning detour, and measured warm. Everything in the table is that second pass.

The tokens-to-solve column is the fingerprint of the whole problem. One configuration finished in about 1,000 tokens while the others needed roughly three times that. Not because it was smarter, but because it spent far less of its budget thinking before it spoke.

The part most benchmark posts leave out

I did not write the harness, and pretending I did would be the easiest lie in this post. I set direction and made the judgment calls. Claude built the thing and pushed back when my framing was loose. The thinking-budget catch was mine. The truncation catch was its. Neither of us produces that clean a table alone, and I have stopped treating that as something to hide. The interesting story was never that I ran a benchmark. It is that the benchmark was wrong first, and getting it right took both of us arguing about why.

So what should you run, and what should you trust?

If you want the top of the table, MTPLX quality at 34 tok/s. If you want the lesson that stuck, read Finding 2 again, because it was not about which format won.

Here is the stance I will plant. Most local-model benchmarks you read are measuring cold, crowded, truncated runs and reporting the number as if it were the model's true speed. I no longer trust a tok/s figure unless I know the model had the machine to itself, ran warm, and was allowed to finish. If a benchmark cannot tell me those three things, it is not measuring the model. It is measuring the mess around it, and quietly handing you the mess as a fact.