Chapter 27 — Why disaggregate prefill and decode

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 1 (the bandwidth wall and the four levers), Ch 7 (NVFP4), Ch 22 (KV bytes per token), Ch 26 (delta handoff). This is the first chapter of Part VI.

27.1 Where we are: KV as a movable asset

Ch 26 ended with the delta handoff: with 32,768 of 65,536 prompt tokens already held, the wire moved 54.2851 % of the full payload and the decoded output was exactly the full-handoff reference [claims #12]. The lesson generalized quietly: the KV cache is not a private data structure of the machine that computed it — it is an asset that can be moved, stored, and resumed, bit-exactly, somewhere else.

This part of the book takes that lesson to its logical end. If KV can move, then the machine that computes prefill and the machine that uses the result need not be the same machine. That is the disaggregated lane. This chapter is the argument for why you would bother — an argument that returns, at systems scale, to the memory-bound case Ch 1 built for one token.

The book’s standing question — what does one token cost, where does the time go, and what may be moved without breaking the exactness contract? — now has a fourth clause in play: moved across the wire. Everything in Part VI is about paying for that clause honestly.

27.2 Two regimes, stated precisely

Before the wire, before the second machine, one question has to be settled precisely, because every argument in this Part leans on it: why should a machine be good at one half of inference and bad at the other, when both halves push the same weights through the same kernels? You have met the two regimes before, briefly (Ch 1 §1.2); here they are as the engineering facts the whole lane rests on.

Decode — generating tokens one at a time. Each token does one matvec against every weight matrix (Ch 13 is the hero example). One multiply-add per weight byte read; ~53 GFLOP of arithmetic against ~16.76 GB of reads; arithmetic intensity ~3.2 FLOPs per byte (derived in Ch 1 §1.3). Decode is bandwidth-bound, serial, and proportional to the number of generated tokens. You cannot make it faster by having more arithmetic units sitting idle.

Prefill — processing the whole prompt before the first token can be emitted. Here the same weight matrices are multiplied by many rows at once: the engine chunks prompts into 512-position batches (PREFILL_BATCH_TOKENS = 512, [crates/muser-engine/src/decode.rs:53]), and each weight byte read from DRAM is reused across the whole chunk — the module doc says it plainly: “prefill of T tokens ≈ one token’s DRAM traffic” [crates/muser-engine/src/prefill.rs:6-8]. The arithmetic per byte therefore scales with the batch. Prefill is compute-bound, parallel, and proportional to the prompt length.

Same weights, same kernels’ worth of math, opposite bottleneck. That asymmetry is the entire subject of this chapter.

The roofline, with both workload points

Ch 1 put decode on one side of the machine’s roofline — the “balance point” where a workload’s FLOPs-per- byte exactly matches the machine’s FLOPs-per-byte of bandwidth. With the ~800 GB/s memory class [ledger L0] and the ~2.6 TFLOP/s of FP32 that keeps ALUs busy at decode intensity (both derivations from Ch 1 §1.3), the balance point sits at ≈ 3.2 FLOPs/byte. Now put both regimes on the same chart:

   arithmetic intensity (FLOPs per byte read), log scale

   10^4 ┤                                            ● prefill, 512-chunk
        │                                            │ ≈ 1,619 F/B
        │                                            │ (derived below)
   10^3 ┤
        │
   10^2 ┤
        │                     roofline: compute ceiling
        │                    ╱  (need ~1.3 PFLOP/s FP32 to feed a
        │                   ╱   512-chunk at 800 GB/s — no Mac has that)
    10 ┤                  ╱
        │                ╱     ← machine balance point ≈ 3.2 F/B
     4 ┤ ─ ─ ─ ─ ─ ─ ─╱─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
        │            ╱
     1 ┤ ● decode ≈ 3.2 F/B
        │ │      bandwidth-bound      compute-bound
        │ └ the entire token time is the weight read   (intensity → right)
      └─┴────────┴──────────────────────────────────────────────
        1 B/token        ~5 B/token                ~1,000+ B/token

Figure 27.1: The roofline flip, ported from the Ferrite book’s Ch 23 device [ferrite-book Ch 23] and recomputed for Muse Glimmer on the M3 Ultra. The ancestor’s marked points were decode ~3.6 vs prefill ~455 FLOPs/byte at batch 128 on an A18 Pro — Ferrite-lineage numbers that do not transfer; the method does.

Derive the prefill point yourself, the way Ch 1 derived the decode one. Per token, the matmul parameter count is 26.5 × 10⁹ and per-token FLOPs ≈ 2 × 26.5e9 ≈ 53 GFLOP (the arithmetic in Ch 1 §1.3 step 4). A 512-token chunk therefore costs:

FLOPs per chunk  ≈ 53.0e9 × 512        ≈ 2.714e13 FLOP
bytes read       ≈ 16.757e9            (each weight byte read once, reused)
intensity        ≈ 2.714e13 / 16.757e9 ≈ 1,619 FLOPs / byte

Roughly 500× further right than decode. On the compute side of the roofline, the only way to go faster is more arithmetic per second — or cheaper arithmetic per FLOP. A Mac’s FP32 units are what they are; a GPU with FP4 tensor cores (matrix-multiply units that consume 4-bit operands natively; see Ch 7 for the NVFP4 format) offers both. Hold that thought for two more sections.

The measured shape of the two regimes on one Mac

The roofline predicts the measured behavior. Local decode is 35.440 tok/s (kquant, CV 0.037 %, its full scope in Ch 1 §1.3) — 28.22 ms per token, dominated by the weight stream. Local prefill per prompt token is much cheaper — weight reads amortize across the chunk — but it is pinned against the machine’s compute ceiling instead. Do the arithmetic with the deep cell: 570.122 s mean for a 131,008-token prompt [claims #6, local baseline] is

570.122 s / 131,008 tokens ≈ 4.35 ms per prompt token
53.0e9 FLOP/token / 4.35e-3 s ≈ 12.2 TFLOP/s sustained

Twelve TFLOP/s of sustained arithmetic on this machine’s FP32-class throughput — the compute-bound regime, hit at depth, on one Mac.

Be careful about what that number is, though, because we were. Nobody instrumented the machine’s arithmetic here. The 12.2 figure is what falls out of dividing the measured 570.122 s wall-clock cell by its token count and multiplying by a per-token FLOP estimate, so it is a derivation and we label it one rather than dressing it up as a compute measurement. We keep it because it does not need to be tight to carry the argument: the derivation would have to be wrong by two orders of magnitude before prefill stopped landing on the far side of the roofline from decode.

Now look at what the user experiences in that regime. Nothing. For nine and a half minutes, nothing. That is the TTFT cliff.

27.3 The TTFT cliff at depth

So put a number on that silence — and name it first, because the silence is the quantity this whole Part exists to shorten.

TTFT — time to first token — is the latency from “prompt submitted” to “first generated token visible.” At shallow depth on one Mac it is fine. At depth it is catastrophic, and it is catastrophic precisely because prefill is proportional to prompt length and compute-bound on silicon shaped for bandwidth-heavy decode.

We walked the cliff ourselves rather than model it: the local lane, five exact-token repetitions at every depth, the prompt growing by powers of two as we climbed. There is nothing subtle about the shape that came back.

Table 27.1: Local prefill TTFT by depth — linear growth in the prompt, compute-bound on one Mac [docs/benchmarks.md §3].

Every row there is a retained run [docs/benchmarks.md §3] [ledger "Phase 4 disaggregated GX10→Mac context matrix", 2026-08-20]. Read the bottom row as a person waiting rather than as a table cell, and the engineering problem states itself. An agent workload that refreshes a 100k-token context is not an edge case of this table; it is the table’s whole reason to exist. This is the demand side. The supply side is the next section’s question: what would it cost to get that KV from somewhere else?

27.4 What the wire would have to carry

Here is the quiet fact that makes disaggregation plausible for this model specifically: Muse Glimmer’s KV is small. Ch 22 derived 1,024 bytes per token per layer (2 KV heads × head_dim 128 × 2 bytes × K+V). If all 52 layers shipped in full, that would be 52 KiB per token. But only the 13 NoPE full-attention layers (Ch 14) grow with context; the 39 SWA layers (Ch 23) live in a 2,048-token ring. What crosses the wire at depth is therefore:

NoPE:  13 layers × 1,024 B/token × position
SWA:   39 layers × 1,024 B/token × 2,048 tokens (the window, not the past)

At the 130,815-token cell that product is 1,823,184,896 B (≈ 1.82 GB). That is not an estimate we are asking you to take on faith. It is what the wire actually carried, and the per-class arithmetic above reconciles to it byte for byte — a reconciliation Ch 22 §22.7 walks through in full. Getting the two sides to agree took two conventions that are easy to trip over. The receiver holds back the boundary token, so NoPE rows ship for prompt − 1, and the last of them is decoded locally; the SWA rings travel as three 13-layer groups rather than as one block. Both conventions are written down, and the client-side record of the run that moved those bytes is retained: [docs/kvpack-merge-handoff-20260820.md §3 D1], [receipt phase4-disagg-20260820/130815-g900091/out-p4/ f-p4-text-g900091-client.json].

Now hold the two costs of one token side by side, because that comparison is the argument for this whole Part: ≈ 13.9 kB (decimal; 13.6 KiB) shipped per token versus ~53 GFLOP recomputed per token. Shipping the answer is absurdly cheaper than computing it again. Nor is that special pleading for our model. The sealing plan’s external-research section makes the same point against the literature: ~52 KiB/token effective full-model footprint versus ~2.2 MB/token for DistServe’s OPT-66B — a 42× smaller artifact class — and DéjàVu’s viability rule, that a transfer must cost less than the recompute it replaces, holds here with room to spare [docs/disaggregated-prefill-sealing-plan-20260818.md §4].

And the wire is fast relative to that payload. Our raw reference ceiling was taken on the direct 10GbE link the lab ran before the migration: ~9.4 Gbps single-stream [ledger T0] [docs/disaggregated-prefill-sealing-plan-20260818.md §W0]. Then, on 2026-08-23, we moved the fabric behind a switch — an improvement for everything except our confidence in that ceiling, which was now a measurement of a topology that no longer existed. So we re-probed instead of assuming the number travelled with us. The product direction came back at 9.256 Gbps, close enough to call the class intact. The reverse direction came back at 6.161 Gbps, which was not what we wanted and is retained as a deviation anyway [ledger GX10 return 2026-08-23, attempts 3–4 + readiness entries]. That reflex — when the ground moves, re-measure, and keep the half of the result you did not like — is the one habit that makes the rest of this Part’s numbers worth anything.

Even at the release floor, the shape holds. That floor is a 3.0 Gbps installed-payload median ([crates/muser-cluster/src/lib.rs:14-15]), where installed payload means the handoff’s payload bytes divided by the kernel’s measured send busy-time — the wire clock Ch 31 defends. Held to that rate, the transfer floors are ~224 ms at 2k, ~1.06 s at 32k, ~3.75 s at 131k [docs/disaggregated-prefill-sealing-plan-20260818.md §4]. Chapter 31 owns the full wire-discipline story (pacing, EEE, why the product rate sits below raw); the point here is the shape: the wire cost is seconds; the local recompute cost is minutes.

So the economics write themselves — if you have a prefill machine whose compute is up to the job, and if moving the KV does not break the exactness contract. Everything after this section is about those two ifs.

27.5 The measured payoff

So much for the argument on paper. Paper is free; what decides this Part is whether the machines agreed with it.

Here is the lane we actually built and measured, as qualified: a resident vLLM NVFP4 producer on one GX10 node prefills the prompt and hands the KV to Mac Metal decode over authenticated Handoff V2. The next two chapters are the producer and the transport; the measurement is here.

The shallow, final-image cell — the one the claims register scopes carefully — at 2,048 prompt / 256 output tokens on the final image, with one uncounted warmup handoff then five counted reps:

1.493 s median TTFT, 0.22 % counted CV, ≥ 6.23 Gbps installed payload, deterministic output — [claims #6], receipt nvfp4-pacing8g-20260818/p4-wrapper23/.

The counted-warmup convention is part of the claim (rep 0 is ~8 % hot from CUDA warmup; the ruling that made it uncounted is [docs/disaggregated-prefill-sealing-plan-20260818.md §7.3]). A post-router re-qualification on the switched fabric reproduced the class: 1.535889499 s median, CV 0.322 %, payload 6.4592–7.2065 Gbps [ledger "Post-router GX10 lane requalification"].

The deep cell — the headline — at 130,815 tokens, EEE-off arm, same night, same producer, same fixture, one warmup + five counted:

137.405 s median remote TTFT, CV 0.576 %, ≥ 6.995 Gbps per-rep payload floor, deterministic output, versus 570.122 s local 131,008-token mean: 4.149× — [claims #6], receipts under kvpack-ladder-20260820/stage2-130815-rerun/.

Scope discipline, both sides, because the number is useless without it: the remote side is a median over five counted reps at 130,815 tokens with EEE disabled (the enrolled link invariant, Ch 31); the local side is a mean at 131,008 tokens from the same claims row. The local baseline is 0.15 % deeper than the remote cell, so the payoff is, if anything, understated [ledger "EEE A/B at 130815"]. The earlier Phase-4 matrix — five reps per depth, an earlier packet lineage — put the whole band at 3.75–4.26× across 2,048 → 130,815 [docs/benchmarks.md §3]:

Table 27.2: The disaggregated payoff band [docs/benchmarks.md §3] [ledger "Phase 4 disaggregated GX10→Mac context matrix"]. Payoff here is local ÷ remote TTFT — note this is the one ratio family in the book that is not the llama ÷ muser convention.

Those are the cells that survived. Two earlier numbers did not, and the claims register requires that they travel with the survivors — so here they are, in the order we lost them.

The first we lost as a claim rather than as a fact. Running the integrated lane cold, we got a 3.881 s disaggregated TTFT for a 2,048-token prompt — 1.87 s of it native producer compute — against ~6.5 s of local serving prefill, on a wire paced at 3.925 Gbps. We wanted that to be the headline for the lane. Then we tried to move it into the claims register, which asks of every number the one question this cell could not answer: what was the spread across repetitions? There had been a single run. Nothing about the packet is false, and it is retained; what it is not is a stability claim, and the register files it as exactly that — a dated single-cell packet from 2026-08-17, “operator-accepted engineering headline, not a five-repetition stability claim” [docs/nvfp4-fast-lane-evidence-20260817.md §Measured product numbers]. The lesson cost us only a headline: one cold cell is an anecdote until it repeats.

The second we lost outright. An early comparison set the exact Spark producer against a 275 s Mac exact mirror and came out at 5.83× — a much prettier ratio than the band above, and a different denominator than the one the claims row scopes. It is retired, and must never be cited [docs/nvfp4-fast-lane-evidence-20260817.md §Measured product numbers] [claims #6]. We leave the retraction visible instead of quietly deleting the figure, because a reader who meets that ratio on an old slide deserves to know it was ours and that we withdrew it. Both packets remain non-notarial; the release lock governs what may be said publicly [docs/launch-claims.md §Ground rules].

And reuse — the Part V machinery — collapses the bill further on repeat traffic: warm first token 0.6132 s at 65,536 and 1.0566 s at 130,815, bit-identical text, no producer drive at all [claims #11]; delta handoff 54.2851 % of bytes [claims #12]. Disaggregation and reuse compose.

27.6 Why v0.1 is honestly ONE Mac + ONE producer

A payoff table invites a natural question: how big is the thing that produced it? Readers who have met other disaggregated systems will be picturing a pool of prefill workers and a scheduler in front of them. That is not what we have. Here is a place where the architecture document is blunt, and the book will not soften it:

“The v0.1 topology is one Mac decoder and one Spark/GX10 producer. … Multi-producer scheduling and node discovery are not implemented.” [docs/muser-architecture.md §Durable and remote KV]

The claims register puts the launch-side form: “1× Mac + 1× GX10 today”; scale-out is roadmap, and no wording may imply a multi-GX10 cluster is running [claims #8]. The receiver admits one producer at a time — one control endpoint, one HMAC key id, and a replay ledger keyed per key id ([crates/muser-cluster/src/lib.rs:9-12]). Onboarding a second node registers it; it does not create a second concurrent producer [docs/one-button-onboarding.md §v1 limits].

Why this is the right v0.1, not a cop-out: the roles are the architecture; the placement is a technicality. The producer/consumer split is defined between processes, not vendors or hosts — the handoff protocol, identity binding, and kvpack state format hold for a colocated producer over loopback just as for a remote one [docs/disaggregated-prefill.md §Roles, not machines]. What the single-remote-producer placement adds is exactly the one thing this part of the book has to teach: tensor-core NVFP4 prefill and unified-memory decode each win on different silicon [docs/disaggregated-prefill.md §The idea]. Colocated producers remain unqualified, not unarchitected [docs/release-todo-20260823.md §10].

The failure model follows from the topology and is worth stating as a design principle, because it decides what the producer is allowed to be:

The producer is a single point of failure for TTFT, never for correctness.

If the GX10 is down, the lane falls back to local Mac prefill — the engine keeps a complete local path for exactly this reason ([crates/muser-engine/src/prefill.rs:10-12]: prefill is “the Mac-local fallback path when GX10 disaggregated prefill (muser-cluster) isn’t available”). Output tokens stay exact; TTFT degrades to Table 27.1’s column. The claims register’s own boundary language: “one producer is a TTFT SPOF with local-prefill fallback, never a correctness SPOF” [claims #13]. A disaggregated system whose correctness depends on a remote node’s uptime has failed at system design; Muser’s contract keeps every correctness gate on the Mac side of the wire (that is Ch 32’s subject in full).

27.7 The decision, as a tradeoff

Every chapter in this book owes a tradeoff with measured consequences. This one has three:

Why split at the prefill/decode boundary and not somewhere else? Because that is where the roofline flips (Figure 27.1) — the two regimes want opposite machines, and the artifact that crosses the boundary (KV, ~14 KiB/token at depth) is three orders of magnitude cheaper to move than the work it saves (~53 GFLOP/token). The measured consequence is the 3.75–4.26× band [docs/benchmarks.md §3]; the counterfactual is the 570.122 s local cell [claims #6].

Why not just make local prefill faster? The obvious alternative — better local batch kernels — attacks the compute-bound side on silicon whose computing budget is already spent elsewhere. The measured consequence of not having that option is visible in the six-depth plain matrix: local prefill means 1.0139–1.0397× versus llama [claims #2] — competitive, not transformative; no local lane turns 570 s into 137 s. The FP4-tensor-core route exists on the GB10, not on the Mac ([docs/disaggregated-prefill-sealing-plan-20260818.md §4], the W4A4 / FlashInfer CUTLASS notes — Ch 28 gets the details).

What does the split cost? A wire, with everything a wire brings: a pacing story (Ch 31 — pacing is the sender-side rate cap on its own sockets), a durability story (the replay ledger’s fsync dance, Ch 30), a security story (mTLS + HMAC, Ch 30), and a precision story (trusting someone else’s prefill, Ch 32). Every one of those costs is a chapter in this Part, and every one of them earned its chapter by biting us during the campaign. We paced the sender to be kind to the link, then spent effort explaining a rate we had imposed ourselves: 3.9 of 9.4 Gbps was our own pin [ledger T1]. A latency tail we would gladly have blamed on the network lived in our own commit path instead, in the replay ledger’s fsync ([docs/disaggregated-prefill-sealing-plan-20260818.md §W1]). And energy-efficient Ethernet’s retransmission blackouts proved to be a property of our own burst schedule rather than of a sick link [ledger "EEE link ruling — operator decision (2026-08-20)"]. Read those three together and a pattern falls out that is worth carrying into the next five chapters: the wire was hardly ever the villain — our own defaults were. The lane survived them because it fails closed, not because it was lucky.

27.8 What comes next

The argument is done: prefill is a throughput job that wants tensor cores and FP4; decode is a latency job that wants unified memory close to the user; Muse Glimmer’s KV is small enough to move; and the measured payoff at depth is a 4.149× TTFT reduction with deterministic output — under scopes this book will keep restating. To disaggregate you need a prefill machine. Ours is one ASUS GX10 — an NVIDIA GB10 — running a resident vLLM NVFP4 producer in a docker container, with a fail-closed culture all its own. That machine, its producer process, and its exit code 75 are the next chapter.

References

• [crates/muser-engine/src/decode.rs:53] — PREFILL_BATCH_TOKENS = 512, the chunk size behind the prefill intensity arithmetic.
• [crates/muser-engine/src/prefill.rs:6-12] — “prefill of T tokens ≈ one token’s DRAM traffic”; the local-fallback role of the same driver.
• [crates/muser-cluster/src/lib.rs:9-22] — 1× Mac + 1× GX10 launch config, one-producer-at-a-time admission, 3.0 Gbps release floor.
• [docs/benchmarks.md] — §Methodology (repetition and floor conventions), §3 (the disaggregated payoff table and the EEE-off 130,815 row).
• [docs/muser-architecture.md §Durable and remote KV] — v0.1 topology, lane matrix, multi-producer-not-implemented.
• [docs/disaggregated-prefill.md] — roles-not-machines; the two-jobs argument; honest limitations (single producer, link dependence).
• [docs/disaggregated-prefill-sealing-plan-20260818.md] — §4 (KV-size math vs DistServe/DéjàVu, transfer floors, GB10 tensor-core notes), §W0 (raw 9.4 Gbps), §W1 (pacing and the fsync-tail lesson), §7.3 (the counted-warmup ruling).
• [docs/nvfp4-fast-lane-evidence-20260817.md §Measured product numbers] — the dated 3.881 s / 1.87 s / ~6.5 s integrated cell and the retired 5.83×.
• [docs/launch-claims.md] — #2 (local matrix), #6 (the disaggregated claims and their scope language), #8 (topology wording), #11/#12 (reuse and delta), #13 (SPOF boundary), §Ground rules.
• [ledger …] — “Phase 4 disaggregated GX10→Mac context matrix” (the payoff band), “EEE A/B at 130815” (the 137.405 s / 4.149× cell), T0/T1 (raw ceiling, pacing ladder), “Post-router GX10 lane requalification”.
• [receipt phase4-disagg-20260820/130815-g900091/out-p4/f-p4-text-g900091-client.json] — payload_bytes = 1,823,184,896, the deep wire payload.
• [docs/kvpack-merge-handoff-20260820.md §3 D1] — the payload reconciliation (NoPE + SWA arithmetic).
• [ferrite-book Ch 23] — the roofline-flip device this chapter ports; its A18 Pro points (~3.6 vs ~455 F/B at batch 128) are Ferrite-lineage and do not transfer to Muser measurements.
• glossary — terms introduced this chapter: TTFT, disaggregated prefill, producer, consumer (receiver), tensor core, TTFT cliff, local-prefill fallback.