How to Write an Inference Engine

The Muser book — Muse Glimmer on Apple Metal, kvpack, and disaggregated prefill

Scope of this book. This book explains, end to end, how the Muser inference engine generates text from the pinned Muse Glimmer model (52 layers, ~30B parameters) on Apple Silicon, with a kquant DFlash speculative lane, a native NVFP4 lane, and a disaggregated lane where a remote NVIDIA GB10 (“GX10”) node prefills NVFP4 under vLLM and hands the KV cache to the Mac over the authenticated Handoff V2 transport. Every line of Metal and Rust quoted here is read from the pinned source tree; every number is tagged with its measurement source.

This is a zero-to-hero book. The first time a concept appears — a softmax, a SIMD group, a nibble, a residual stream, a rotary embedding, a KV cache page, an NVFP4 block — it is defined in place, with a diagram and a worked example, before it is ever used to explain Muser code. A reader who knows Rust but has never written a GPU kernel, and who has never read a transformer paper, should be able to finish this book understanding the math of LLM inference, the Metal that makes it fast on Apple Silicon, the precision discipline that makes remote prefill trustworthy, and the evidence culture that keeps all of it honest.

We wrote it the way we worked. When a chapter reaches a real decision, it does not hand you the winning answer and move on. It sets up the fork, tells you what we tried and what we expected, lets the attempt fail on the page, names what the failure taught us — and only then shows what shipped. Put the other way round: the dead ends are not confessions tucked into an appendix, they are where the reasoning is visible, and they are the whole reason the shipped answer deserves your trust instead of merely your patience. Wherever we tell you one of those stories, the run that proved it is retained and cited, so you can go and check us.

How to read this book

You are holding eight parts, and they are not all the same kind of reading. Which order, then, and how much do you have to swallow before you are allowed to skip around?

Linearly, the first time. The chapters build on each other:

The appendices (A–F) hold the glossary, the kernel dispatch table, the lane and flag surface, the bibliography, the pin record, and the writing contract.

As a reference, afterwards. Each kernel chapter is self-contained: it states the math, the access pattern, the Rust dispatch, the Metal shader, the measured tradeoffs, and its own bibliography. You can jump to any one chapter once you have read Part I.

The measured reality

A book about performance is worth exactly as much as its numbers, so the fair question to ask before reading another sentence is: which machine made them, under what load, and can anyone else check? We would rather answer that here, at the front, than have you wonder about it for eight parts.

All Muser numbers in this book come from the retained evidence of the 2026-08 campaigns — five-repetition synthetic matrices, the kvpack ladder, the pacing ladder, and the qualification wizard runs — recorded in the campaign ledger (docs/goal-parity-ledger-2026-08.md in the Muser repo) and cited here with receipt paths under muser-receipt://. The decode hardware is one Apple Silicon Mac with an M3 Ultra and 96 GB of unified memory; the prefill hardware is one ASUS GX10 (NVIDIA GB10) node on the same wired 10GbE fabric; the comparator is pinned llama.cpp. The shipped lane matrix at the pin [docs/muser-architecture.md]:

Read that table for its gaps as much as for its throughputs. Only the kquant lane carries a speculative number; the fast product lane’s speculative cell says rejected (fail-closed), which is this engine’s way of saying that we tried the combination, it did not hold, and the code now refuses it outright rather than quietly serving you something weaker under the same name. That refusal has a chapter of its own. It is the shape of most stories in this book: an attempt we believed in, a measurement that disagreed, and a piece of machinery that now says no on purpose.

The book’s recurring question: what does one token cost, where does the time go, and what may be moved — into a draft model, into a cache, or across the wire — without breaking the exactness contract? Every kernel chapter returns to the dispatch-gap accounting that keeps Muser’s decode at parity-within-noise while rejecting fusions that would change logprobs [docs/decode-dispatch-gap-20260815.md]; every systems chapter returns to what the evidence actually permits us to believe.

The pinned source of truth

Every book written against living code goes stale. The only choice an author has is whether it goes stale visibly or invisibly, and a book whose line numbers drift silently is worse than no book at all. So we pinned it.

This book is written against one pinned revision of Muser (see PINNED.md). When the book and the code disagree, the code wins. Every quoted line is tagged file:line relative to the Muser repository root so you can verify it. The canonical paths:

• Metal shaders: crates/muser-engine/src/shaders/ (including the ferrite/ lineage directory)
• Engine and forward pass: crates/muser-engine/src/
• Benchmarks and harness: crates/muser-bench/
• kvpack adapter: crates/muser-kvpack/
• Server and sessions: crates/muser-server/
• Cluster/transport: crates/muser-cluster/
• Vendored kvpack: third_party/kvpack/
• Evidence: muser-receipt:// (append-only; cited by path)

Lineage and attribution

Muser is a from-scratch Rust workspace with no Ferrite runtime dependency, but its CPU text-extraction lineage and select shaders/kernels were adapted, with attribution, from the private Ferrite research tree (docs/extraction-manifest.md, NOTICE). This book has its own ancestor too: the Inference Book on Apple Metal written against Ferrite and Qwen2.5-1.5B on an A18 Pro. This book keeps that book’s pedagogical spine — zero-to-hero, define-everything-on-first-use, cite-everything — and rebuilds the content for Muser’s model, lanes, and measured reality. Where a Ferrite lesson survives on Muser’s live path, the book says so and cites it.

The reason that genealogy matters while you read, rather than only in the credits: the ancestor ran a far smaller model on a phone-class chip. Its numbers describe a different machine and do not transfer to this one. So when an A18 Pro figure appears in these pages it is labelled as ancestry, never as a Muser result, and the two never share an unmarked sentence.

What this book is not

Some promises are easiest to keep as prohibitions. Each of the three below is a genre this book could have slid into without anyone noticing, so we named them early and held each other to them.

• It is not marketing. Every number states what was measured and under what scope — device, model, quantization, reps — and cites its receipt. If you want a sales pitch, this is the wrong book.
• It is not a survey of dead ends for their own sake. Rejected designs (the linear distributed-speculative lane, native NVFP4 speculative decode under Fallback B, the ANE route for v0.1) appear where their failure teaches a tradeoff that survives. When one does appear, it is told as what it was — a fork we walked down, an expectation we were holding, the measurement that ended the argument — and not as a verdict in a list, because a list of verdicts teaches nobody how to make the next decision. The path is as important as the destination.
• It is not speculation. Where a “why” cannot be cited from source or a measurement, it is marked [unverified] rather than smoothed over.

Bibliography convention

Each chapter ends with a References section of its own, because a reader finishing one kernel chapter should not have to hunt the back of the book. You will usually meet these tags at the end of a paragraph, or gathered in a section’s evidence trail, rather than wedged into the middle of a sentence: the evidence is not optional, but neither is being readable, and a claim you cannot follow is no better cited than one you cannot check. Citation tags:

• [crates/.../file.rs:LINE] — Muser source, pinned revision.
• [docs/<file>.md] — Muser engineering documents.
• [ledger §N] — docs/goal-parity-ledger-2026-08.md, the campaign ledger.
• [claims #N] — docs/launch-claims.md row N.
• [receipt <path>] — evidence under muser-receipt://.
• [ferrite-book Ch N] — the ancestor book (pedagogical lineage).
• [Metal-SS §N] / [Metal-PG §…] — Apple Metal Shading Language Specification / Programming Guide.
• [CUDA §…] / [PTX …] — NVIDIA CUDA documentation, where the disaggregated lane demands the comparison.
• [arxiv:XXXX.YYYYY] — the paper.
• [vLLM …] / [llama.cpp …] — upstream projects Muser interoperates with.

Status of each chapter

The book is written in passes, and we would rather admit which pass a chapter is in than let you assume it has been checked. So every chapter declares itself before it says anything else — a status line at the top:

• status: polished — reviewed, citable, ready to read.
• status: draft — written, not yet through review passes.
• status: stub — placeholder, to be filled.

Building

The book lives in src/ as plain Markdown with an mdBook-style SUMMARY.md. To read linearly, start at Ch 1. With mdbook installed, mdbook build and mdbook serve work out of the box (book.toml pins the metadata and the mermaid preprocessor; the site is built without the repository’s _research/ working artifacts, which sit outside src/ by design). GitHub Pages deploys on every push to main via .github/workflows/pages.yml.

That is the apparatus — the scope, the machines, the pin, the tags. The rest of the book is the walk itself, and Part I starts where the work started: with the question of where a token’s time actually goes, and the discovery that the answer has almost nothing to do with arithmetic.

Chapter 1 — The problem: why inference is a memory problem

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: none. This is the first chapter of the book. It assumes you have never run a transformer and have never thought about memory bandwidth.

1.1 The promise of this book, and the question that runs through it

This book explains, end to end, how the Muser inference engine generates text from one pinned model — Muse Glimmer, 52 layers, roughly 30 billion parameters — on one Apple Silicon Mac with an M3 Ultra GPU and 96 GB of unified memory. Every line of Metal and Rust quoted in these chapters is read from the pinned Muser source tree; every number carries a tag that says where it was measured. When the book and the code disagree, the code wins.

Here is the thesis, in one sentence, before any definitions:

Generating a single token of text means streaming sixteen gigabytes of model weights through the GPU, and the GPU finishes the arithmetic long before the bytes have finished arriving.

That is the whole book in one line. LLM inference — the act of a model producing text — looks like a computation problem. It is not. It is a bandwidth problem wearing a computation costume. This chapter derives that fact on the back of an envelope, from the model’s own geometry, so that by the end you can reproduce every number yourself. We will do the arithmetic together, in roughly the order we did it ourselves — including the step where we went looking for a number we assumed existed, did not find it, and had to change how we asked the question.

And here is the question that recurs in every chapter after this one — the book’s standing question:

What does one token cost, where does the time go, and what may be moved — into a draft model, into a cache, or across the wire — without breaking the exactness contract?

Every kernel chapter in Part IV returns to that question (“is this kernel where the time goes?”), every systems chapter in Parts V–VII returns to it (“what may be moved?”), and Part VIII is about the evidence culture that keeps every answer honest. Keep the question in your pocket. We will collect answers as we go.

1.2 The cast: a few words you need first

Before the arithmetic, eight one-sentence definitions. We are about to count two quantities and divide them by two speeds, and every word below is load-bearing in one of those counts. Each is expanded in the glossary and, where it matters, in a later chapter.

• A parameter is one learned number inside the model — one knob tuned during training. A “30B model” holds on the order of 30 billion of these numbers.
• Weights are the parameters collectively — the giant tables of learned numbers stored on disk. “Reading the weights” means loading those tables so the math can use them.
• A token is one unit of text — roughly a piece of a word. “The cat sat” is about four tokens. The model emits tokens one at a time.
• Decode is the act of generating those tokens one by one, after the prompt has been read. Reading the prompt is called prefill; the two regimes have opposite bottlenecks, and the contrast becomes important in Ch 36.
• A matvec (matrix × vector) is the one operation decode does, over and over: multiply a big table of weights by a single vector representing the current token. The hero kernel of this book, Ch 13, is a matvec.
• A FLOP (Floating-point OPeration) is one unit of arithmetic — one multiply or one add. A matvec over a matrix with N elements costs about 2N FLOPs.
• A GGUF is the on-disk model-file format Muser reads: a small header, then the weight tensors packed end to end.
• Quantization is storing each learned number in fewer bits than a full 32-bit float — four-ish bits per weight instead of sixteen or thirty-two. Part II (Ch 5) is devoted to it; here you only need the fact that it shrinks the file.

The argument below is just: count the FLOPs, count the bytes, divide each by the machine’s speed at that thing, compare the two times.

1.3 One token, costed by hand

This is the arithmetic the rest of the book rests on. We cost out generating one token of Muse Glimmer on the M3 Ultra. No hand-waving: every number is derived, and the ones that are measurements say so.

Step 1 — how many parameters?

The label on the box says thirty billion parameters. Labels round, and we are about to divide bytes by this number, so we would rather read the model’s true shape off the artifact than off the marketing.

Happily, we do not have to take anyone’s word for the shape. Muse Glimmer’s geometry is parsed fail-closed from the GGUF at load time, and a test asserts every parsed field against the pinned release artifact. The table below is therefore not a transcription from a model card — it is the geometry the engine refuses to start without [crates/muser-engine/src/config.rs:169-181] [crates/muser-engine/tests/muse_golden.rs:97-108]:

Table 1.1: Muse Glimmer geometry, read from the pinned GGUF contract.

Now count parameters from the shapes. The loader asserts a per-tensor shape contract at config.rs:294-318, so the dimensions multiplied out below are the ones the engine itself checks on the way in: a wrong figure here fails the load rather than quietly producing a wrong book. Per layer:

attention projections:
  q      6,656 × 4,096 = 27,262,976
  k      6,656 ×   256 =  1,703,936
  v      6,656 ×   256 =  1,703,936
  gate   6,656 × 4,096 = 27,262,976
  o      4,096 × 6,656 = 27,262,976
                              85,196,800
feed-forward (gate, up, down):
  3 × (6,656 × 19,968)  = 3 × 132,907,008 = 398,721,024
norms (4 × 6,656) + per-head QK norms (2 × 128) = 26,880
                                           per layer: 483,944,704
× 52 layers                             = 25,165,124,608
+ token embedding 6,656 × 202,048       =  1,344,831,488
+ LM head         6,656 × 202,048       =  1,344,831,488
+ final norm                             =          6,656
                                          ─────────────
total                                   = 27,854,794,240

So Muse Glimmer is 27.85 billion parameters by hand — the “~30B” on the label is rounded up. The exact count matters less than the habit: every bandwidth number in this book divides through a total like this one, and you should be able to re-derive it.

Step 2 — how many bytes on disk?

Parameters are not bytes, and it is bytes that the memory system has to move. So the next question is what all those learned numbers actually weigh.

Full-precision f32 would be 4 bytes each: 27.85e9 × 4 ≈ 111 GB. That does not fit the plan. The pinned kquant GGUF stores most weights in 4-to-6-bit blocks instead (Part II explains the formats), and its exact size is asserted by the release-gate test:

#![allow(unused)]
fn main() {
// crates/muser-server/src/chat_template.rs:250
assert_eq!(metadata.len(), 16_756_681_056, "release GGUF byte size");
}

That assertion is not decoration. The size is written down in two independent places that have to agree, and the second is the engine’s own crate doc, which states it in words: “The pinned target artifact is 16,756,681,056 bytes on disk.” [crates/muser-engine/src/lib.rs:14] If a rebuild ever changed the artifact, the gate would fail loudly rather than let this chapter’s arithmetic quietly drift.

Units, because they will bite you otherwise:

16,756,681,056 bytes
  = 16.757 GB   (decimal, 10⁹ bytes — the convention this book uses)
  = 15.61  GiB  (binary, 2³⁰ bytes)

Cross-check against Step 1: 16,756,681,056 × 8 bits ÷ 27,854,794,240 parameters ≈ 4.81 bits per parameter — quantized blocks plus their per-block scale bytes plus the two big tables, averaged out. A 30B model at ~4.8 effective bits is what makes a 96 GB Mac a plausible host at all.

Step 3 — how much do you read to produce ONE token?

This is the insight that surprises people. To generate one token, every weight matrix is used exactly once: the matvec for each projection touches every row of every weight table a single time. The weights are 16.76 GB and used once per token, so they cannot live in any cache — caches are megabytes. They must be streamed, in full, out of main memory, for every single token.

Say it the other way round, because this is the part that trips people up. The weights are not a dataset the GPU loads once and then keeps close. They are a river. One token is one complete pass of that river past the ALUs, and the next token starts the river again from the top, byte for byte the same.

1 token   → read ~16.76 GB of weights, once.
10 tokens → ~167.6 GB.
500-token answer → ~8.4 TB of reads.

Not because the model grows — because every token re-reads all of it.

A reader who already knows what a KV cache is will be objecting by now: the weights are not the only thing decode reads. Quite right, and the reason it matters here is that our envelope is only honest if the other traffic is negligible — if it were comparable, every ratio in this chapter would be wrong. So it is worth spending a paragraph proving that it is not.

The non-weight traffic exists but is small at shallow context. The KV cache (a per-layer memory of past tokens; Ch 15 is its chapter) costs 1,024 bytes per layer per cached token — the formula 2 KV heads × 128 × 2 bytes × (K + V) is derived in [docs/memory-footprint.md §KV formula]. At the benchmark cell we use below (a ~98-token context), attention reads 52 × 98 × 1,024 B ≈ 5.2 MB per token — 0.03 % of the weight stream. At the full 131,072-token context it grows to ~1.83 GB per token [docs/memory-footprint.md] — still 9× smaller than the weights; Ch 22 costs it out properly. The objection is real, then, but it does not change the shape of the answer: at the depths this chapter reasons about, the weight stream is the traffic.

Step 4 — how much arithmetic is that?

Bytes counted. Now the other side of the comparison, the side everyone assumes is the expensive one: how much arithmetic does a single token actually demand?

A matvec y = W·x over a matrix with N elements costs ~2N FLOPs — one multiply-add pair per element. Summing only the matrices (norms and the embedding lookup contribute ~nothing):

matmul parameters = 52 × (85,196,800 + 398,721,024)   = 25,163,726,824
                  + LM head 6,656 × 202,048           =  1,344,831,488
                                                     ─────────────
                                                     26,508,558,312
FLOPs per token ≈ 2 × 26.5e9 ≈ 53.0 GFLOP

(The embedding table contributes 0 FLOPs — the lookup is a gather, not a multiply.) That is the entire compute budget for one token: ~53 GFLOP.

Step 5 — what does the machine actually do?

We have the bytes and we have the FLOPs. To turn either into a time we need the machine’s own speeds — and this is where the book departs from the ancestor text it descends from. The fork is worth walking slowly, because what we chose here governs every bandwidth claim in the rest of the book.

The Ferrite book, written for a small phone-class chip, could quote a measured DRAM ceiling for its hardware, and we expected to do the same: run a read microbenchmark, take the ceiling, divide the weight bytes by it, done. We could not. For this M3 Ultra, no Muser document records a measured pure-read DRAM ceiling. That left two roads. We could borrow a specification figure from a datasheet and let the reader assume it was ours — fluent, authoritative, and unearned. Or we could work the equation backwards from a number we had genuinely measured, and say plainly that it is a derivation.

We took the second road, and it is the reason the number below carries a label everywhere it appears: this book derives the effective read rate from measured decode throughput rather than asserting a bus speed it never observed.

The measurement we do have is the decode throughput itself. The kquant lane’s headline decode number — five repetitions after warmup, synthetic fixture, F16 KV, 66-token prefix / 32 teacher-forced tokens (teacher-forced: the harness feeds known prior tokens rather than model-generated ones) — is:

35.440 tok/s (35.439527527, CV 0.037 %) — kquant plain decode [ledger P1.3], echoed in [docs/benchmarks.md §1]

(“CV” is coefficient of variation, std/mean across the repetitions — the lower, the steadier the cell.)

Two derived numbers fall out, and both are derivations, not measurements:

token time   = 1 / 35.439527527        = 28.22 ms per token

effective read rate
  = 16,756,681,056 bytes × 35.439527527 tokens/s
  = 593.85 GB/s   (decimal; 553 GiB/s)

This 594 GB/s is an effective rate derived from measured throughput — the machine’s demonstrated average weight-stream rate at this operating point. It is not a specification number and not a measured pure-read ceiling. For orientation only: the campaign ledger refers to the M3 Ultra’s “~800 GB/s” memory class when discussing kernel occupancy [ledger L0] — a class label, again not a measured ceiling. Against that class, decode runs at ≈ 74 % (593.85 / 800). Where this book needs an honest bandwidth reference, it uses the derived effective rate and says so.

Step 6 — compute time vs. memory time

Now the punchline. At the derived effective rate, just reading the weights for one token costs:

16.756681056 GB / 593.85 GB/s = 28.22 ms

That is the entire measured token time — 28.22 ms, from Step 5. The weights alone, streamed at the rate the machine actually sustains, consume the whole token. The 53 GFLOP of arithmetic has no room of its own; it must hide completely underneath the byte stream (Figure 1.1).

Put in plainer words: the GPU is not slow, and it is not busy. It is waiting. The whole engine described in this book is an argument about what to do with a processor that finishes early.

Sanity-check the roofline direction. The workload’s arithmetic intensity — FLOPs per byte read — is:

53.0e9 FLOP / 16.757e9 bytes ≈ 3.2 FLOP / byte

A machine whose memory moves ~800 GB/s [ledger L0, class reference] needs only ≥ 800 × 3.2 ≈ 2.6 TFLOP/s of sustained FP32 to keep its ALUs busy on this workload. The M3 Ultra GPU’s exact FP32 peak is not recorded in any Muser document [unverified], but any GPU paired with an ~800 GB/s memory system clears 2.6 TFLOP/s many times over. Decode sits deep on the memory side of the roofline — and, more directly: the measured token time equals the weight-stream time. There is nothing left over for compute to own.

     DRAM (one unified 96 GB pool)                 GPU (M3 Ultra)
   ┌────────────────────────────────────┐      ┌────────────────────────────┐
   │  mapped GGUF   16.76 GB (kquant)   │      │  ~53 GFLOP of matvec math  │
   │  ════════════════════════════════  │ ───▶ │  hides entirely under the  │
   │  ════════════════════════════════  │ 594  │  byte stream; the token    │
   │  ════════════════════════════════  │ GB/s │  time IS the read time     │
   │  read ALL of it, ONCE per token    │      │  (28.22 ms measured)       │
   └────────────────────────────────────┘      └────────────────────────────┘
     52 layers × {Q,K,V,gate,O, gate,up,down}
     + final norm + LM head (202k-wide)

Figure 1.1: One token = stream ~16.76 GB once. The bytes leave DRAM at the ~594 GB/s effective rate (derived from the measured 35.440 tok/s [ledger P1.3]); the math finishes underneath. The GPU is byte-starved, not ALU-starved.

1.4 The bandwidth wall

The consequence is brutal and it dictates the entire engine.

If a workload is compute-bound, you speed it up with more ALUs, higher clocks, a bigger GPU. If it is memory-bound, none of that helps — the ALUs you already have are waiting for bytes most of the time; adding more adds more waiting. Muse Glimmer decode does 3.2 FLOPs per byte read; the machine’s balance point is an order of magnitude higher. It is on the wrong side of the roofline, and no kernel cleverness moves it to the right side — the matvec is one multiply-add per weight byte; its intensity is fixed by the model’s format.

So only three things can make decode faster:

1. Read fewer bytes per token — smaller weights. This is why Part II exists: the kquant blocks that get Muse Glimmer to 4.81 effective bits/param, and the NVFP4 native lane. It is also where the argument’s first tempting expectation died. Fewer bits per weight ought to mean fewer bytes per token, and fewer bytes per token ought to mean faster decode; we ran the two lanes against each other expecting the newer format to pull ahead. NVFP4 decode landed at 35.491 tok/s against kquant’s 35.440 — parity within noise, never claimed faster [ledger P1.3] [docs/benchmarks.md §1]. At batch-1 decode both lanes read nearly the same bytes at nearly the same rate, so the win we actually bought was capacity, not decode speed. That distinction — a lane that buys you room is not a lane that buys you time — is one this book refuses to let slide, in either direction.
2. Read bytes faster — a different memory system. Not a lever you have on a fixed Mac; the ceiling is the machine’s.
3. Avoid re-reading — reuse computed results. The KV cache exists so attention does not recompute the past every token (Ch 15); kvpack (the durable cache format of Part V) warm reuse collapses a 68.6 s cold prefill at 65,536 tokens to a 0.6132 s warm first token, bit-identical text — a single-sample cell, not a distribution [claims #11] [ledger "Kvpack ladder stage-5 isolated-depth verdict"].

Muser’s era adds a fourth lever the ancestor book did not have: move the work somewhere else.

• Move it into a draft model: DFlash speculative decoding drafts cheap tokens and verifies a whole window of them in one batched pass against the full model — the verify matvec reintroduces weight-reuse into a loop that had none. This lever also produced the book’s most instructive retraction. An early headline put the speedup at 1.3273×; then the measurement window itself turned out to be wrong, and requalifying against the fixed window brought the honest figure down to 1.23692× vs the pinned llama.cpp comparator at 2,048 depth [claims #15]. The old headline is superseded and must not be cited as a current result, and the famous 107.9 tok/s figure survives only as the kquant spec bar [ledger "Spec-prefill funded-fix requalification"]. Both were kept in the ledger rather than quietly deleted, which is the habit Part VIII is about: a number you have to explain is worth more than a number you have to trust.
• Move it across the wire: disaggregated prefill runs the compute-friendly half of the job on a remote NVIDIA GB10 node and hands the KV back — 4.26× faster to first token at 2,048 depth [docs/benchmarks.md §3] [ledger "Phase 4 disaggregated GX10→Mac context matrix", 5 reps].
• Move only what’s new: delta handoff ships just the suffix — 54.2851 % of full bytes with bit-exact output at the 32,768-of-65,536 cell [claims #12] [receipt kvpack-ladder-20260820/attempt-10-…-stage6-delta/ stage6-delta-65536/stage6-verdict.json].

Every chapter of this book is about one of these four levers, or about the evidence culture that decides whether a lever actually worked.

1.5 “96 GB is still a budget”

The ancestor Ferrite book opened with a provocation: on its 8 GB phone-class chip, “8 GB is a lie” — capacity was never the binding constraint, bandwidth was [ferrite-book Ch 1]. On this machine the provocation flips sign but keeps its shape: 96 GB is real, and it is still a budget — capacity answers “does it fit, and what else fits”; bandwidth answers “how fast”. You need both, and the second one governs decode.

What must fit in the 96 GB [docs/memory-footprint.md]? The arithmetic, all of it derivable:

Table 1.2: The 96 GB budget [docs/memory-footprint.md] — with that document’s own caveats: these are on-disk/topology-derived numbers, “summing artifacts + KV is only a lower bound,” and no smaller-memory configuration may be advertised as supported.

Two budget facts worth internalizing now. First, the release contract is four full-context slots on this one machine [docs/memory-footprint.md §Release requirement] — KV (7.3 GB at the ceiling) plus weights (~16.8 GB) plus optional draft/vision (~3 GB) plus activations is what “four slots” costs; the KV term, not the weights, is what decides how many slots fit (Ch 22 derives this). The engine shares one mapped weight arena across all slots precisely so the 16+ GiB target is loaded once, not once per slot ([crates/muser-engine/src/decode.rs:954-957]). Second, capacity and bandwidth fail differently: run out of bandwidth and you get 28 ms tokens; run out of capacity and you get nothing.

1.6 How to read every number in this book

One thing remains before the descent: how to read a number when you meet one. None of the rules below is an abstract principle. Each was learned by watching a figure that was perfectly true of a particular run turn into a claim about the product, and then having to take it back. Stated once here, obeyed everywhere after:

• Ratios are llama ÷ muser — above 1.0 means muser is faster [docs/benchmarks.md §Methodology]. Absolute tok/s drifts with machine state; the same-session interleaved ratio is the trustworthy cross-engine statistic.
• Counted cells are five repetitions after the stated warmup convention, means with coefficient of variation; single-sample cells say so (Ch 38 covers the protocol).
• Synthetic vs. natural is load-bearing. The spec-decode ratios above are synthetic-fixture numbers; on natural text, cross-engine outputs diverge, and spec decode loses on high-acceptance shallow text (rust at 2,048: 0.931) [ledger "Spec re-measurement at the fixed window"]. Never let a synthetic number become a workload claim.
• Never cite the all-accept 110.59 tok/s control as serving performance, the retired 5.83× remote-prefill figure, the superseded 1.3273× spec headline, or the barred 1.64960× decode-at-131k accounting [docs/launch-claims.md]. Every figure on that list was once the true result of a real run; each became misleading the moment it was quoted outside the cell that produced it. They are retained, and retained visibly, so nobody has to rediscover on their own why they were pulled.
• Ferrite-lineage numbers (the ancestor lab’s A18-class measurements) are labeled as lineage when they appear, never as Muser results.

1.7 Where the book goes from here

You now hold the book’s central derivation. The rest is a guided descent into the machinery that lives under it.

• Part I (this part) teaches the Metal compute model — Ch 2 (devices, queues, command buffers, threads, threadgroups, SIMD groups), Ch 3 (unified memory and the buffer substrate that maps 16.76 GB of weights with zero copies), and Ch 4 (how .metal source becomes runnable kernels — from three distinct sources).
• Part II is quantization: how 27.85 billion parameters fit in 16.76 GB, and what each lane pays for it.
• Parts III–IV build the model and walk the decode loop kernel by kernel — the place where “one token = one pass over the weights” becomes fifty-two layers of named dispatches.
• Part V is the KV cache as an asset; Part VI is the disaggregated lane — the fourth lever, taken seriously; Part VII is serving; Part VIII is measurement and the evidence culture.

By the end, the sentence at the top of §1.1 will be something you have proven, kernel by kernel, from source — not something you were told.

But there is a prerequisite. Everything in this book after this page happens on the GPU, and to follow it you must speak Metal: what a device is, what a command buffer records, what a threadgroup is, why the SIMD group is the unit that actually matters on Apple Silicon. That language is the next chapter.

References

• [crates/muser-engine/tests/muse_golden.rs:97-108] — the pinned-artifact geometry assertions (52 layers, hidden 6,656, heads 32:2, head_dim 128, FFN 19,968, vocab 202,048).
• [crates/muser-engine/src/config.rs:169-181] — fail-closed GGUF metadata parsing of the same fields; :294-318 the per-tensor shape contract used in the parameter count.
• [crates/muser-engine/src/lib.rs:14] — “The pinned target artifact is 16,756,681,056 bytes on disk.”
• [crates/muser-server/src/chat_template.rs:237-261] — the release_gguf test: byte size, chat-template length (7,167 B), and the three SHA-256 identities.
• [docs/memory-footprint.md] — KV formula, the 96 GB budget table, the four-slot release contract, and its own “lower bound only” caveat.
• [docs/benchmarks.md] — §1 (35.44/35.49 parity-within-noise), §Methodology (ratio and repetition conventions), §3 (disaggregated payoff band).
• [ledger P1.3] — docs/goal-parity-ledger-2026-08.md, kquant/NVFP4 plain-decode table: 35.440 (CV 0.037 %) / 35.491 (CV 0.130 %), five reps, 66-token prefix / 32 teacher-forced tokens, F16 KV.
• [ledger L0] — same ledger, “microbenchmark-first apparatus and the occupancy bound”: the “~800 GB/s M3 Ultra” memory-class reference.
• [claims #11], [claims #12], [claims #15] — [docs/launch-claims.md]: warm reuse at depth, delta handoff, current spec-decode restatement.
• [receipt kvpack-ladder-20260820/attempt-10-…-stage6-delta/ stage6-delta-65536/stage6-verdict.json] — the 54.2851 % delta-share evidence (verified delta_share_of_full: 0.5428507652…, exact_against_full_handoff: true).
• [ferrite-book Ch 1] — the ancestor’s “one token, costed by hand” device this chapter ports; its 1 GB / 45.95 GB/s / 33.20 tok/s numbers are Ferrite-lineage and do not transfer.
• glossary — terms introduced this chapter: parameter, weights, token, decode, prefill, matvec, FLOP, GGUF, quantization, bandwidth, arithmetic intensity.

Chapter 2 — The Metal compute model

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 1. You know why decode streams ~16.76 GB per token. This chapter assumes you have never written a GPU shader.

2.1 What a GPU actually is

Chapter 1 ended on a prerequisite: everything after that page happens on the GPU, and following it means speaking Metal — devices, command buffers, threads, threadgroups, SIMD groups. This chapter teaches that language from zero. A CPU is good at doing one hard thing fast. A GPU is good at doing ten thousand easy things at the same time. Inference is, almost entirely, ten thousand easy things: multiply this row of weights by this vector, add up the products, do it again for the next row. That is why the GPU does the work, not the CPU.

The “ten thousand easy things” are called threads. A thread is one execution lane: it runs your program once, sees its own small slice of the data, and writes its own small slice of the answer.

But a GPU is not a magical parallel computer where every thread is independent. Threads are organized, and the organization is what makes a kernel fast or slow. On Apple Silicon the organization that matters most is the SIMD group, which we meet in §2.6. First, the API.

2.2 Metal: Apple’s GPU API

Metal is Apple’s API for talking to the GPU. It is to Apple GPUs what CUDA is to NVIDIA GPUs: the vendor’s own first-class compute path. If you want compute work done on an Apple Silicon Mac, Metal is that path.

Metal gives you three things:

1. A shading language (MSL, Metal Shading Language) — a C++ dialect in which you write the code each thread runs. That per-thread program is called a kernel: a single function, written once, that every thread in a launch runs once over its own slice of the data. Muser’s kernels live in crates/muser-engine/src/shaders/, and the census there is small enough to say out loud: 29 .metal files, of which two are Muser-authored and 27 sit in the ferrite/ lineage directory, plus one bench-only candidate shader under crates/muser-bench/. Where an inherited kernel came from is recorded rather than remembered; the extraction manifest is retained [docs/extraction-manifest.md].
2. A host API (Objective-C underneath, wrapped by the Rust crate metal) — the code the CPU runs to compile kernels, allocate memory, and submit work.
3. A memory model — on Apple Silicon, a single physical DRAM pool shared by CPU and GPU. Ch 3 is devoted to it.

The single most important mental model for the host API is the next section.

2.3 “Record a tape, then press play”

The CPU does not tell the GPU “do this now.” Instead the CPU records a sequence of instructions onto an object called a command buffer, and then hands the whole buffer to the GPU in one shot. Think of recording a cassette tape: you can record many songs onto it, in order, and only when you press play does anything actually happen [Metal-PG, "Command Buffers"].

The object that records the tape is the compute command encoder. You ask it to do four things, over and over:

bind the kernel you want to run             (set_compute_pipeline_state)
bind the memory buffers it will read/write  (set_buffer, repeated)
bind any small constants                    (set_bytes)
launch N copies of the kernel               (dispatch_thread_groups)

That four-line sequence is one dispatch — one kernel running once across many threads. A Muser decode token is a dozen-plus dispatches per layer across the model’s 52 layers, plus a head and tail — hundreds of dispatches in all, recorded onto one command buffer per token, then played. The per-group counts are reconciled and measured in Ch 35; here you only need the shape.

Here is the CPU side, verbatim — forward_token, the function that owns one whole decode token:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5448
let command_buffer = queue.new_command_buffer();
// One concurrent encoder owns the complete token. Graph dependencies
// are explicit barriers; independent projection groups share a barrier
// interval and may overlap, matching the accepted Ferrite/llama route.
let serial = GraphEncoder::concurrent(
    command_buffer
        .compute_command_encoder_with_dispatch_type(metal::MTLDispatchType::Concurrent),
);
self.encode_token(&serial, &token_view, self.n_past)?;
serial.encoder.end_encoding();
command_buffer.commit();
self.context
    .wait_for_completion(command_buffer, Duration::from_secs(300))?;
}

Steps, in order: get a blank tape from the queue; get a recorder with concurrent dispatch semantics (§2.9); record the entire token with encode_token (the 52-layer graph — hundreds of encode_* calls, each of which records one or a few dispatches); stop recording; press play (commit); wait — bounded, in Muser, by a 300-second deadline rather than an unbounded block. Recording is pure CPU bookkeeping measured in microseconds; the wall-clock time is paid between commit and the wait returning.

One deliberate difference from the ancestor Ferrite book’s engine is worth a paragraph, because it is the reason the cost model you just learned is the only one you need: Muser does not have a dispatch_* family of test-only wrappers that each pay a full commit-and-wait round trip. In an engine that has them, a kernel can be submitted two ways — the cheap shared-tape way on the hot path and the expensive one-tape-per-kernel way in tests — and the two drift. Here, tests hand-roll the same five steps inline when they need them; the multi-column kernel tests are the example to read [crates/muser-engine/src/metal/encode/multicol.rs:373-390]. Everything on the hot path is an encode_* call onto a shared encoder, and there is no second submission shape to keep in your head.

2.4 The handles: one struct, five GPU objects

Which pieces of GPU state deserve to be created once and kept forever, and what does getting that wrong cost? The question is not academic: opening a device, building a command queue, and compiling a library of shaders are all slow enough that doing any of them per token would swamp the token. Muser’s answer is to hoard exactly the handles that are expensive to make and cheap to share, in one struct, MetalContext:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/context.rs:32
pub struct MetalContext {
    pub device: Device,
    pub queue: CommandQueue,
    pub library: Library,
    /// Strict-f32 copy of the standalone Muse kernels.  The cross-vendor
    /// Q8 projection and integer NVFP4 routes must match CUDA's explicit
    /// scalar boundaries, while the ordinary serving kernels retain fast math.
    pub cross_vendor_library: Library,
    pub ggml_library: Option<Library>,
    pub ggml_library_path: Option<PathBuf>,
}
}

Walk the fields. The Rust types come from the metal crate: Device wraps MTLDevice, CommandQueue wraps MTLCommandQueue, Library wraps MTLLibrary.

• device (MTLDevice) — the handle to the GPU. On Apple Silicon there is exactly one system-default device: Device::system_default() at [crates/muser-engine/src/metal/context.rs:46]. You ask it to allocate memory and to compile shaders.
• queue (MTLCommandQueue) — the queue that accepts command buffers, created once at startup: device.new_command_queue() ([crates/muser-engine/src/metal/context.rs:47]). Every command buffer comes off this queue.
• library (MTLLibrary) — a bundle of compiled kernel functions, addressable by name. This one is built at startup from a concatenation of 24 .metal source files, with fast math on ([crates/muser-engine/src/metal/context.rs:59-110]). Ch 4 covers how, including why there are three libraries where a simple engine would have one.
• cross_vendor_library — the same two source files recompiled with fast math off, so a handful of kernels match CUDA’s arithmetic boundaries bit for bit when the remote-producer lane needs them ([crates/muser-engine/src/metal/context.rs:111-121]).
• ggml_library — an optional third library loaded from a prebuilt llama.cpp .metallib (a kernel library serialized to disk; Ch 4 is its chapter) when MUSER_GGML_METALLIB is set — the pinned upstream kernels Muser dispatches for numerical parity ([crates/muser-engine/src/metal/context.rs:122-131]).

There is a timing rule hiding in that list, and it is the part to carry forward. At construction time, every kernel is looked up by name and compiled once into a cached pipeline state — the MetalKernels constructor and its PIPELINES registry of 66 names [crates/muser-engine/src/metal/encode.rs:21-88], which is Ch 4’s subject again. At dispatch time you never touch a library at all. You reach for the pipeline the registry already holds, and it costs a line: self.bind(encoder, "sigmoid_gate_inplace") [crates/muser-engine/src/metal/encode/gate.rs:17].

Ask the registry for a name nobody put in it and the process dies on the spot: PsoCache::get refuses to return silently for an unregistered name [crates/muser-engine/src/metal/pso_cache.rs:45-49]. That loudness is on purpose — a missing pipeline is a programming error, never a runtime condition — and the tradeoffs section returns to why the crash is the kinder outcome.

These five objects are the only long-lived GPU state. Everything else — buffers, command buffers, encoders — is created per-use or per-token (Ch 3 for the buffers).

2.5 One queue, one owner

When several sequences are decoding at once, who is allowed to talk to the GPU, and in what order? Metal answers half of that: command buffers from one queue run in FIFO order, so the GPU itself will not interleave two tokens’ work. It does not answer the other half. Several resident sequences can still stampede the queue from many threads, and FIFO then means only “whoever got there first, repeatedly” — a hot slot can starve its peers while staying perfectly ordered. Muser’s answer is a single owner:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:1020
/// One owner for the shared Metal queue. Decode work is selected first and
/// resident sequence IDs rotate in ascending cyclic order, preventing a hot
/// slot from repeatedly reacquiring the accelerator ahead of its peers.
struct AcceleratorScheduler {
    state: Mutex<AcceleratorSchedulerState>,
    ready: Condvar,
}
}

The AcceleratorScheduler ([crates/muser-engine/src/decode.rs:1023-1026]) is a Mutex+Condvar gate: acquire(sequence_id, work) blocks until the accelerator is free and this sequence is the chosen next one — decode work is selected before prefill, and decode sequences rotate in ascending cyclic order ([crates/muser-engine/src/decode.rs:1040-1059]). Every graph — every tape — is recorded and committed while holding an AcceleratorPermit. The shared execution resources live next to it, in MetalShared:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:954
/// Immutable Metal execution resources shared by every resident sequence.
/// Metal command submission is scheduler-serialized; retaining one context,
/// pipeline set, mapped weight arena, and GPU vector set avoids loading the
/// 16+ GiB target once per serving slot.
pub struct MetalShared {
}

That comment is Ch 1’s capacity argument made structural: four serving slots share one context, one pipeline set, one mapped 16.76 GB weight arena, and one scheduler. The per-sequence state (KV planes, activations, position) lives elsewhere, in MetalMuseModel ([crates/muser-engine/src/decode.rs:989-998]).

2.6 Threads, threadgroups, and the SIMD group

This is the part that trips people up. There are three nested units of parallelism on an Apple GPU, and you must understand all three to read Muser’s dispatch sizes.

┌─────────────────────────────────────────────────────────────────────────┐
│  GRID  = the entire launch (what dispatch_thread_groups fixes)          │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐         (n threadgroups)  │
│  │ threadgroup│ │ threadgroup│ │ threadgroup│ ...                      │
│  │ ┌────────┐ │ │ ┌────────┐ │ │            │                          │
│  │ │SIMD grp│ │ │ │SIMD grp│ │ │            │   each SIMD group =      │
│  │ │ 32 lanes│ │ │ 32 lanes│ │ │            │   32 threads in lockstep │
│  │ └────────┘ │ │ └────────┘ │ │            │                          │
│  │ ┌────────┐ │ │            │ │            │                          │
│  │ │SIMD grp│ │ │            │ │            │                          │
│  │ └────────┘ │ │            │ │            │                          │
│  └────────────┘ └────────────┘ └────────────┘                          │
└─────────────────────────────────────────────────────────────────────────┘

Figure 2.1: The three nested units. A grid of threadgroups, each threadgroup of SIMD groups, each SIMD group of 32 lockstep threads.

• Thread — the unit that runs the kernel once. Each thread has a unique thread_position_in_grid so it can pick its own slice of the data. Nothing is promised about the order threads run in.
• Threadgroup — a block of threads (32–1024) that share on-chip threadgroup memory and can synchronize with each other using threadgroup_barrier(...). A threadgroup is the unit of co-scheduling: all its threads live on one compute unit together.
• SIMD group (Apple-specific, the important one) — exactly 32 threads that execute in lockstep on one SIMD ALU. All 32 lanes run the same instruction at the same instant. The superpower: lanes within a SIMD group can exchange data in one cycle via intrinsics like simd_sum(x) (a 32-way sum) and simd_shuffle(val, lane). A reduction that would take log₂(32) = 5 barrier-separated passes between independent threads takes one simd_sum inside a SIMD group.

Here is a real Muser kernel that uses all three units — rms_norm_batch, the batched RMSNorm reduction (RMSNorm itself is Ch 12’s subject; here it is just “reduce one row of the residual stream to one scalar, then scale”):

// crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:1
kernel void rms_norm_batch(
    device const float* x      [[ buffer(0) ]],  // [B × n]
    device const float* weight [[ buffer(1) ]],  // [n] (shared)
    device       float* out    [[ buffer(2) ]],  // [B × n]
    constant     uint&  n      [[ buffer(3) ]],
    constant     float& eps    [[ buffer(4) ]],
    uint tgid [[ threadgroup_position_in_grid ]],
    uint tid [[ thread_index_in_threadgroup ]],
    uint sgitg [[ simdgroup_index_in_threadgroup ]],
    uint lid [[ thread_index_in_simdgroup ]],
    threadgroup float* shared [[ threadgroup(0) ]])
{
    const uint batch = tgid;
    device const float* xb  = x   + batch * n;
    device       float* ob  = out + batch * n;
    device const float4* xb4 = (device const float4*)xb;
    device const float4* wb4 = (device const float4*)weight;
    device float4* ob4 = (device float4*)ob;
    const uint n4 = n >> 2u;

    float sum_sq = 0.0f;
    for (uint i = tid; i < n4; i += 128u)
        sum_sq += dot(xb4[i], xb4[i]);
    sum_sq = simd_sum(sum_sq);
    if (lid == 0u) shared[sgitg] = sum_sq;
    threadgroup_barrier(mem_flags::mem_threadgroup);
    if (tid == 0u)
        shared[4] = rsqrt((shared[0] + shared[1] + shared[2] + shared[3]) / float(n) + eps);
    threadgroup_barrier(mem_flags::mem_threadgroup);
    const float inv_rms = shared[4];
    for (uint i = tid; i < n4; i += 128u)
        ob4[i] = xb4[i] * inv_rms * wb4[i];
}

Read it with the vocabulary. Each threadgroup handles one row of the batch (tgid), and the launch is 128 threads = 4 SIMD groups. The [[ ]] attributes are MSL’s way of receiving precomputed coordinates: tgid = which threadgroup (which row), tid = thread 0..127 within it, sgitg = which of the 4 SIMD groups, lid = lane 0..31 within the SIMD group. The reduction inside is the classic two-stage pattern:

1. Each thread accumulates partial sums of squares over float4 (4-wide) loads, striding by 128 — the strided loop that shares work across the threadgroup.
2. simd_sum(sum_sq) collapses each SIMD group’s 32 partials to one value in a single instruction.
3. Lane 0 of each SIMD group writes its group’s value into threadgroup memory (shared[sgitg]).
4. threadgroup_barrier — every thread in the threadgroup waits until all writes to threadgroup memory are visible. This is the thread-level barrier; it is the only way to synchronize across SIMD groups.
5. Thread 0 combines the 4 group values, computes the inverse root mean square, parks it at shared[4].
6. A second barrier, then every thread re-reads inv_rms and scales its slice of the row.

Say the same thing the other way round, because this asymmetry is the one idea that shapes every kernel in the rest of the book: inside a SIMD group, threads talk to each other for free, in a single instruction. Between SIMD groups, they cannot talk at all — they can only leave messages in threadgroup memory and agree, via a barrier, on when it is safe to read them. Two levels of communication, two very different prices. A kernel that does its reducing inside SIMD groups and its combining across them pays the expensive price as few times as possible; that is what the two-stage pattern above is for, and it is why the geometry of a launch is never an arbitrary choice.

The Rust side that launches it:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/norm.rs:270
self.bind(encoder, "rms_norm_batch");
encoder.set_buffer(0, Some(input.metal()), 0);
encoder.set_buffer(1, Some(weight.metal()), 0);
encoder.set_buffer(2, Some(output.metal()), 0);
set_value(encoder, 3, &(dim as u32));
set_value(encoder, 4, &eps);
encoder.set_threadgroup_memory_length(0, 32);
encoder.dispatch_thread_groups(MTLSize::new(rows as u64, 1, 1), MTLSize::new(128, 1, 1));
}

In prose: grid = rows threadgroups (one per row of the batch), threadgroup = 128 threads (4 SIMD groups of 32), and 32 bytes of threadgroup memory (shared[0..4]). Note set_threadgroup_memory_length on the host side matching threadgroup(0) in the kernel — the buffer-slot contract of §2.8, one slot namespace over.

The 32sg tell

Muser’s kernel names carry their geometry. The live decode tail kernel is muser_fused_norm_residual_rms_norm_32sg — “32sg” = 32 SIMD groups = 1,024 threads, because it fuses two norms plus a residual add over Muse Glimmer’s 6,656-wide rows and wants the whole row resident in the threadgroup. The dispatch comment says exactly that:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/norm.rs:236
// 32 SIMD groups keep the 6,656-wide Muse tail resident and match the
// accepted Ferrite geometry. 33 floats are padded to Metal's 16-byte
// dynamic-threadgroup-memory alignment.
encoder.set_threadgroup_memory_length(0, 144);
encoder.dispatch_thread_groups(MTLSize::new(rows as u64, 1, 1), MTLSize::new(1024, 1, 1));
}

(The kernel body at [crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:147-201] is the same two-barrier pattern as rms_norm_batch, with 1024u strides and a 32-entry combine loop — read it now, it will hold no surprises.) Whenever you meet a Muser kernel in this book, decode the suffix first: _4sg, _32sg, _4r2s (4 rows, 2 SIMD groups), _n32 (32 output rows per threadgroup). The suffix is the geometry.

2.7 The grid is the output shape

How many threads should a kernel launch? There is a rule, and it is short enough to memorize: the grid is the output shape. You launch one thread per thing you intend to write, and the shape of the launch follows the shape of the answer, never the shape of the input. Getting it wrong is not a performance bug, it is a correctness bug — too small a grid leaves part of the output holding whatever was there before, and too large a grid runs off the end of the buffer unless the kernel guards. Two contrasting real geometries from Muser’s decode path show the rule doing its work:

An elementwise kernel — one thread per element. Muse Glimmer’s sigmoid attention-output gate multiplies one 4,096-wide vector by an elementwise-gated copy of itself (the architecture’s oddity, covered in Ch 17). The whole kernel:

// crates/muser-engine/src/shaders/ferrite/sigmoid_gate.metal:7
kernel void sigmoid_gate_inplace(
    device       float* attn_out [[ buffer(0) ]],
    device const float* gate     [[ buffer(1) ]],
    constant     uint&  n        [[ buffer(2) ]],
    uint gid [[ thread_position_in_grid ]])
{
    if (gid < n) {
        attn_out[gid] *= 1.0f / (1.0f + exp(-gate[gid]));
    }
}

And the whole dispatch:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/gate.rs:7
pub fn encode_sigmoid_gate(
    &self,
    encoder: &ComputeCommandEncoderRef,
    values: &GpuBuffer,
    gate: &GpuBuffer,
) {
    debug_assert_eq!(values.len(), gate.len());
    if std::env::var_os("MUSER_CROSS_VENDOR_QK").is_some() {
        encoder.set_compute_pipeline_state(&self.cross_vendor_sigmoid_gate);
    } else {
        self.bind(encoder, "sigmoid_gate_inplace");
    }
    encoder.set_buffer(0, Some(values.metal()), 0);
    encoder.set_buffer(1, Some(gate.metal()), 0);
    set_value(encoder, 2, &(values.len() as u32));
    dispatch_1d(encoder, values.len());
}
}

dispatch_1d is Muser’s one-line elementwise helper:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:1337
pub(super) fn dispatch_1d(encoder: &ComputeCommandEncoderRef, count: usize) {
    if count == 0 {
        return;
    }
    let width = count.min(256) as u64;
    encoder.dispatch_threads(MTLSize::new(count as u64, 1, 1), MTLSize::new(width, 1, 1));
}
}

Note it uses dispatch_threads — the non-tiled form where you state the exact total thread count and a threadgroup width, and Metal works out the grid (256-wide groups here, and Metal pads the tail so the gid < n guard matters). One thread per output element; idle guard-exited threads cost a branch, not a stall.

A batched pointwise kernel — 1,024-wide groups. The prefill-side residual_add_batch (dst[i] += src[i] over a batch) instead fixes the geometry explicitly, one thread per element in groups of 1,024:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:515
self.bind(encoder, "residual_add_batch");
encoder.set_buffer(0, Some(destination.metal()), 0);
encoder.set_buffer(1, Some(source.metal()), 0);
set_value(encoder, 2, &(total as u32));
encoder.dispatch_thread_groups(
    MTLSize::new(total.div_ceil(1024) as u64, 1, 1),
    MTLSize::new(1024, 1, 1),
);
}

Same rule — grid = ⌈total/1024⌉ groups of 1,024 threads covers exactly total outputs. And a matvec inverts the aspect ratio entirely: it emits one output element per weight-matrix row, so it launches many small threadgroups (one or a few per row). The exact matvec geometries are Ch 13’s subject; remember only the rule.

2.8 Binding memory: slots are the contract

Before launching, you tell each thread where its inputs live. You bind MTLBuffer objects to numbered buffer slots (0, 1, 2, …) that the kernel reads as [[ buffer(N) ]]. Look back at sigmoid_gate_inplace: its parameters carry [[ buffer(0) ]], [[ buffer(1) ]], [[ buffer(2) ]], and those numbers line up one-for-one with the CPU-side calls set_buffer(0, …), set_buffer(1, …), set_value(encoder, 2, …). The slot index is the whole contract between a kernel and its dispatch code.

Small constants — a uint count, an eps float, a small argument struct — do not get their own MTLBuffer; they are inlined into the command buffer with set_bytes, which is what the set_value helper wraps ([crates/muser-engine/src/metal/encode.rs:1329-1335]).

The subtle half of set_buffer is its third argument, a byte offset into the buffer. Muser’s weight dispatch binds views — one giant mapped buffer plus per-tensor offsets — through exactly this argument:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/multicol.rs:192
encoder.set_buffer(0, Some(weights.metal()), weights.offset() as u64);
}

What weights is, why one buffer serves the whole model, and what the offset arithmetic must respect are Ch 3’s story — the next chapter.

2.9 Serial vs concurrent dispatch

So far the encoder looked like a strict sequence: dispatch N, then dispatch N+1. Metal offers two flavors of compute encoder, and Muser uses both deliberately:

• A serial encoder (new_compute_command_encoder()) runs its dispatches in order, one at a time. Simple, and leaves the GPU under-occupied when consecutive dispatches are independent.
• A concurrent encoder (compute_command_encoder_with_dispatch_type(MTLDispatchType::Concurrent)) may overlap dispatches that have no dependency between them — exactly what a decode graph wants, because its four input projections (Q, K, V, gate) read the same input and write disjoint outputs.

You saw forward_token create the concurrent encoder in §2.3. Freedom is not free: with overlap, you must say where dependencies live. Muser wraps that rule in one type, GraphEncoder, whose entire job is to insert an explicit memory barrier between dispatch groups:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:6256
impl EncodeTarget for GraphEncoder<'_> {
    fn before_dispatch(&self) {
        if self.concurrent && self.has_dispatch.replace(true) {
            // Broad buffer scope exactly matches llama.cpp's dependency reset.
            // Independent kernels are deliberately grouped into one dispatch
            // closure, so every closure boundary is a real graph dependency.
            unsafe {
                let _: () = objc::msg_send![self.encoder, memoryBarrierWithScope: 1u64];
            }
        }
    }
    // …
}
}

Read it as a protocol: the 52-layer graph is encoded as a sequence of closures (dispatch(command, |encoder| { … }) at [crates/muser-engine/src/decode.rs:6273-6279]); independent kernels are deliberately packed into one closure (they may overlap inside it); every closure boundary is a real dependency, and before_dispatch plants a memoryBarrierWithScope there so the GPU drains prior writes before the next group starts. The comment names the provenance: this is llama.cpp’s own dependency-reset discipline, adopted wholesale.

So why does the serial encoder still exist at all? Because which flavor wins is not obvious from first principles, and the losing side was kept so the comparison can be re-run. The decode path had already been converted: its projections sit together in one closure and overlap. Prefill was the open question, and the argument for leaving it serial was respectable — prefill kernels are large, a large kernel should saturate the GPU on its own, and concurrency would then be buying overlap nobody needed while still paying for barriers. The comment records how that argument fared:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:976 (fields of MetalShared)
// Prefill-only concurrent Q/K/V/gate and FFN gate+up. Decode already
// groups those projections; serial prefill paid a launch tax on PP128.
// `MUSER_SERIAL_PREFILL_DISPATCH` restores the previous encoder for A/B.
concurrent_prefill_dispatch: bool,
}

Serial prefill lost. The launch tax it paid was visible in the prompt-processing cell, so concurrent became the default — and the old encoder was not deleted. MUSER_SERIAL_PREFILL_DISPATCH restores it [crates/muser-engine/src/decode.rs:1332], which means the two policies can still be raced against each other on one binary, on whatever machine doubts the result. That is the flag-for-measurement culture you will meet throughout this book: a decision that was made by a number keeps the switch that produced the number.

The same instinct, pushed to its diagnostic extreme, gives every dispatch group its own command buffer. That is deliberately the expensive shape this section argued against — and it is exactly why it is useful, because Metal then reports an exact GPU interval per group instead of one interval per token. It lives in the PhaseProfiler [crates/muser-engine/src/decode.rs:6224-6236], gated by MUSER_METAL_PHASE_PROFILE, diagnostic-only and never on the serving path.

2.10 The full cycle, and a bounded wait

Putting the whole chapter together, the lifetime of one piece of GPU work in Muser:

1. Acquire the accelerator (AcceleratorScheduler::acquire, [crates/muser-engine/src/decode.rs:1040]).
2. queue.new_command_buffer() — get a blank tape.
3. Create the encoder (concurrent on the token path; serial where the A/B flag says so).
4. (Hundreds of times) bind pipeline + bind buffers + set constants + dispatch — via encode_* functions, grouped into barrier-delimited closures.
5. encoder.end_encoding() — stop recording.
6. command_buffer.commit() — press play; the GPU starts, asynchronously.
7. Wait for completion — bounded.

Step 7 deserves its own paragraph, because it is where Muser’s fail-closed culture reaches even the wait. CommandBufferRef::wait_until_completed() blocks a thread unboundedly — a wedged GPU would freeze the serving thread forever. Muser’s wait_for_completion parks that blocking call on a detached watcher thread and bounds the caller’s wait with a condvar, so a hang becomes a logged Deadline error carrying the command buffer’s label and status, not a frozen box ([crates/muser-engine/src/metal/context.rs:149-211]). The 300-second deadline you saw in forward_token is that mechanism.

2.11 Tradeoffs

Three decisions in this chapter had a plausible alternative. Here is what happened when we took each alternative seriously.

One command buffer per token vs. per dispatch. Start with the easy half. The obvious way to run a decode graph is one command buffer per kernel — record, commit, wait, repeat — and it is what a first engine writes, because every step is independently debuggable. It also pays a CPU↔GPU round trip per dispatch, hundreds of them per token, which is why Muser records the whole token (embedding through softcap) onto one buffer and waits once.

The hard half is that the cheap shape is not free either: batching a token onto one tape makes encode-side work and dispatch count grow, and both are measurable. So the campaign measured them. The production one-token decode graph reconciles to 760 profiling closures vs the legacy route’s 564 — a +196 difference — reconciled exactly into 104 separated norm-boundary groups + 39 SWA wrapped-ring staging groups + 52 KV-publication/attention splits + 1 last-row copy [docs/decode-dispatch-gap-20260815.md §Corrected closure-count diff]. (Closures there are Rust profiling closures, not raw Metal dispatches; the two counts are not interchangeable.)

Read that reconciliation and one term looks like free money. The 104 separated norm-boundary groups are the largest single line in the diff, and they exist only because a boundary was drawn between two norms. So we built the fix: a hybrid that fused across the boundary and deleted the separated groups. The expectation was that the graph would shed the biggest slice of its extra structure while computing exactly the same numbers, since fusing adjacent arithmetic is supposed to be a rearrangement, not a change. It was not. The hybrid changed bits — normalized-logprob max error 3.197e-4 against a 1e-4 contract, with the first divergence tracked down to one f16 ULP in layer-1 V — and we kept the postmortem instead of the patch [docs/decode-dispatch-gap-20260815.md §Rejected hybrid postmortem]. (In that sentence a logprob is the logarithm of the probability the model assigns a token, and a ULP — unit in the last place — is the smallest step a float format can take; the parity contract is measured in exactly those units. Ch 38 owns it.)

The lesson is worth stating plainly, because the rest of the book keeps running into it: dispatch structure is a lever, but exactness is the gate. A restructuring that changes the output is not a faster engine, it is a different engine, and it does not get to compete. What survived from the same investigation was the one removal that was provably exact — a single 6,656-element copy — which bought −0.136 ms GPU (−0.34 %) [docs/decode-dispatch-gap-20260815.md §Landed and rejected reductions]. That is the honest size of the prize when you refuse to trade bits for it.

Concurrent vs serial encoding. Concurrency buys overlap between independent projections and sells you a new obligation in return: every real dependency now needs its barrier, spelled out by hand, or the engine computes garbage fast (§2.9). That is a bad trade whenever the overlap is worth nothing — which is what the “prefill kernels are large enough already” argument predicted, and what the measurement contradicted: “serial prefill paid a launch tax on PP128” [crates/muser-engine/src/decode.rs:976-979], PP128 being the 128-token prompt-processing benchmark cell. Concurrent is therefore the default, and MUSER_SERIAL_PREFILL_DISPATCH survives for one purpose only: re-running the comparison that settled it.

Panicking on an unregistered pipeline. The gentle-looking alternative is to hand back nothing and carry on — return an Option, skip the dispatch when the name is unknown, keep the server up. Follow that path to its end, though, and a typo’d kernel name produces no error at all: the output buffer keeps whatever was in it, the graph continues, and the model goes on emitting plausible tokens computed from a step that never ran. Muser takes the crash instead. PsoCache::get panics on a miss [crates/muser-engine/src/metal/pso_cache.rs:45-49], so the mistake surfaces at the first dispatch, in development, as a programming error with a name attached. Fail-closed, even here.

2.12 What’s next

You now know every Metal concept needed to read any kernel in this book: device, queue, command buffer, encoder, thread, threadgroup, SIMD group, grid, buffer slots, the encode/commit/wait cycle, and Muser’s barrier-between-groups protocol. One big thing was deferred: §2.8’s set_buffer(slot, buffer, offset) hid the entire memory story — what a MTLBuffer is on this machine, why one buffer can hold the whole 16.76 GB model, and how the weights get from disk into it without a single copy. That is unified memory, and it is the next chapter.

References

• [crates/muser-engine/src/metal/context.rs:32-42] — MetalContext (device, queue, and the three libraries); :46-47 device/queue creation; :149-211 the deadline-bounded wait_for_completion.
• [crates/muser-engine/src/decode.rs:5432-5463] — forward_token: the one-command-buffer-per-token cycle with the concurrent encoder.
• [crates/muser-engine/src/decode.rs:954-984] — MetalShared: one context/pipeline-set/arena per engine, and the concurrent_prefill_dispatch A/B comment.
• [crates/muser-engine/src/decode.rs:1020-1030] — AcceleratorScheduler, the one owner of the shared queue.
• [crates/muser-engine/src/decode.rs:6124-6279] — EncodeTarget, GraphEncoder (concurrent + memoryBarrierWithScope), PhaseProfiler, and the dispatch closure helper.
• [crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:1-33] — rms_norm_batch (the three-units reduction kernel); :147-201 the 32sg dual-eps fused tail.
• [crates/muser-engine/src/metal/encode/norm.rs:236-240, 270-277] — the 1×128 and rows×1024 dispatches and the 6,656-wide geometry comment.
• [crates/muser-engine/src/shaders/ferrite/sigmoid_gate.metal:7-16] and [crates/muser-engine/src/metal/encode/gate.rs:7-23] — the elementwise kernel and its dispatch.
• [crates/muser-engine/src/metal/encode.rs:500-523] — encode_residual_add_batch (explicit 1,024-wide geometry); :1329-1343 — set_value and dispatch_1d; :21-88 — the 66-name PIPELINES registry.
• [crates/muser-engine/src/metal/pso_cache.rs:45-49] — the panic-on-miss pipeline cache accessor.
• [crates/muser-engine/src/metal/encode/multicol.rs:192] — a real offset-view bind (forward pointer to Ch 3).
• [docs/decode-dispatch-gap-20260815.md] — the +196-closure reconciliation, the rejected hybrid (3.197e-4 vs the 1e-4 contract), and the one exact removal (−0.136 ms GPU).
• [Metal-PG] — Apple, Metal Programming Guide: “Command Buffers,” “Compute Processing,” “Dispatching Threads.”
• [Metal-SS] — Apple, Metal Shading Language Specification: “SIMD-group Functions,” “Threadgroup Functions,” address-space attributes.
• glossary — terms introduced this chapter: Metal, MSL, kernel, thread, threadgroup, threadgroup memory, threadgroup barrier, SIMD group, grid, MTLDevice, MTLCommandQueue, command buffer, compute command encoder, dispatch.
• [ferrite-book Ch 2] — the ancestor’s tape-recorder analogy and three-units pedagogy this chapter ports; its 215-shader, ~171-dispatch and encode-percentage figures are Ferrite-lineage.

Chapter 3 — Unified memory and the buffer substrate

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2. You know what an MTLDevice, a command buffer, an encoder, and set_buffer(slot, buffer, offset) are. This chapter assumes you know nothing about memory architecture.

3.1 The big picture: one DRAM, two brains

Chapter 2 closed on set_buffer(slot, buffer, offset) and deliberately deferred the memory story — what an MTLBuffer is on this machine, and why one buffer can hold the whole 16.76 GB model. That deferral leaves a question standing, and it is the question this chapter answers: where do the weights actually live while the GPU is reading them, and what does it cost to put them there? On this machine the answer is short enough to be surprising, and it starts from the single most important fact about Apple Silicon for inference.

On a discrete GPU — an NVIDIA card in a desktop — the CPU and the GPU do not share memory. The CPU sits next to one pool of memory chips (CPU DRAM, what people call RAM), the GPU sits next to a separate pool on the card (VRAM), and the two pools are wired together by a bus (PCIe). Every time the CPU wants the GPU to see a weight tensor, it must memcpy the bytes across that bus. A 16.76 GB model means shipping 16.76 GB over PCIe before the first token can even be thought about.

Apple Silicon is built differently. The CPU cores and the GPU are on the same piece of silicon (a system-on-chip, SoC), and they both point at the same physical pool of DRAM. There is no second pool and no bus to copy across. This single-pool design is called unified memory; Figure 3.1 draws the contrast with the discrete world.

   APPLE SILICON (unified memory)              DISCRETE GPU (e.g. NVIDIA)

   ┌─────────────────────────────┐             ┌───────────┐   PCIe   ┌───────────┐
   │            SoC              │             │    CPU    │◄════════►│    GPU    │
   │  ┌──────┐        ┌──────┐   │             │  ┌─────┐  │  bus +   │  ┌─────┐  │
   │  │ CPU  │        │ GPU  │   │             │  │ DRAM │  │ memcpy()│  │ VRAM │  │
   │  │cores │        │cores │   │             │  └─────┘  │          │  └─────┘  │
   │  └──┬───┘        └───┬──┘   │             └───────────┘          └───────────┘
   │     │                │      │               two physical DRAM pools;
   │     └───────┬────────┘      │               you memcpy weights
   │        ┌────┴─────┐         │               across the bus
   │        │ one DRAM │         │
   │        │  pool    │         │
   │        │ (96 GB)  │         │
   │        └──────────┘         │
   └─────────────────────────────┘
     one physical DRAM; CPU and GPU
     read/write the SAME bytes

Figure 3.1: Unified memory (Apple Silicon) vs. the discrete split. On Apple Silicon “uploading weights to the GPU” is a no-op — the bytes are already there. (This diagram is also the seed of Part VI: the remote GB10 node is a discrete-memory machine across a 10GbE wire, which is why the disaggregated lane moves KV tiles, not work-in-progress.)

For inference the consequence is transformative: a model file on disk can be mapped into the process’s address space once, and — because the GPU shares that address space — the GPU can read the weights directly from those pages. There is no second copy. §3.6 shows the exact calls; first, what Metal calls its buffers.

3.2 The three Metal storage modes, and the one Muser uses

Unified memory settles where bytes can live. It does not tell Metal what you intend to do with them, and Metal insists on being told: every allocation carries a declaration. So the question for this section is which declaration Muser makes — for every buffer in the engine, without exception — and why a faster-looking alternative was tried and then handed back.

When you ask the MTLDevice for a buffer, you must tell it a storage mode — where the bytes physically live and who can see them. Metal defines three [Metal-PG, "Resource Objects: Storage Modes"]:

• StorageModeShared — one copy of the bytes, in unified memory. CPU and GPU read and write the same physical pages; a CPU write is instantly visible to the GPU. This is the Apple Silicon mode.
• StorageModePrivate — bytes only the GPU can touch; the CPU must stage data through a Shared buffer and a GPU blit copy (a block copy run on the GPU through a dedicated blit encoder, as opposed to the compute encoder that runs kernels). This is how discrete VRAM works.
• StorageModeManaged — two caches, one per side, explicitly synchronized with synchronize/didModifyRange calls. It exists for older discrete-Mac setups.

On Apple Silicon, Private and Managed are simply irrelevant: there is no separate VRAM to be private to and no two caches to manage. Muser uses StorageModeShared for every buffer it creates, through one function:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/buffer.rs:7
fn shared_tracked() -> MTLResourceOptions {
    // Several accepted Muse paths still cross compute encoders (notably
    // target-hidden prefill/capture). Untracked resources are only valid when
    // every such dependency has an explicit fence/barrier. b9678d4 enabled
    // untracked mode globally before that contract existed and empirically
    // changed DFlash conditioning while leaving final greedy IDs unchanged.
    MTLResourceOptions::StorageModeShared
}
}

A function that returns one enum constant should not need a paragraph of commentary above it. This one does, because the interesting part is what the return value omits: it is just StorageModeShared, with no HazardTrackingModeUntracked flag beside it. The comment is a scar, and the story behind it is worth telling in full, because the failure it records is one this book will keep meeting: a change that produces correct-looking output and is wrong anyway.

Lineage — the untracked-hazards gamble, and why we declined it. Metal can optionally mark buffers untracked (HazardTrackingModeUntracked), which tells the driver to skip its automatic dependency tracking between dispatches. Bookkeeping you skip is work you do not pay for, and we had reason to expect a win: the ancestor Ferrite engine ran untracked and made it safe with its own conflict-driven barrier planner, measured there at a real but modest win [ferrite-book Ch 3] (Ferrite-lineage numbers, ~2–4 % on that lab’s A18-class hardware). So we took the fork. Commit b9678d4 flipped the switch engine-wide — and, in the comment’s own words, did it “before that contract existed”: before every cross-encoder dependency in Muser had an explicit fence standing in for the tracker we had just turned off.

What we expected was a small speedup and byte-identical output. What we got was identical output and different internal state. The build empirically changed DFlash [speculative-draft] conditioning while leaving final greedy IDs unchanged ([crates/muser-engine/src/metal/buffer.rs:7-14]). Read that twice. The tokens matched — so every greedy-output test we had would have gone green. What moved was the speculative draft’s conditioning, which is watched on its own precisely so that a change like this has somewhere to show up.

A silent conditioning change with correct-looking output is exactly the failure this engine exists to prevent, so the switch went back. Muser runs Metal’s default tracked storage plus the explicit memoryBarrierWithScope groups you met in Ch 2 §2.9. The general rule survives from the ancestor book, and it is worth stating in the abstract because it recurs for the rest of the engine: turning off a safety net is safe if and only if something else provably enforces the ordering. Ferrite had that something — a barrier planner. Muser, at that commit, did not, and the cheaper provable thing was to keep the net on.

3.3 The buffer substrate: three types, one view

Unified memory decides where bytes live; it says nothing about what they mean. An engine that keeps f32 activations, f16 KV planes and immutable quantized weights in a single address space needs some way of stopping itself from confusing them — and it has to do that without a runtime type tag on the hot path, because the hot path runs per token. That is the job of the buffer module, and it is small enough to read in one sitting. Three concrete buffer types and one view type carry the whole engine:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/buffer.rs:28
#[derive(Clone)]
pub struct GpuBuffer {
    inner: Buffer,
    len: usize,
}

/// Shared Metal storage whose logical elements are IEEE-754 binary16 bits.
///
/// Keeping this distinct from [`GpuBuffer`] prevents an F16 KV plane from
/// being accidentally fingerprinted or indexed as an F32 activation buffer.
#[derive(Clone)]
pub struct GpuHalfBuffer {
    inner: Buffer,
    len: usize,
}

#[derive(Clone)]
pub struct GpuBytes {
    inner: Buffer,
    len: usize,
    _mmap: Option<std::sync::Arc<memmap2::Mmap>>,
}

#[derive(Clone, Copy)]
pub struct GpuByteView<'a> {
    buffer: &'a GpuBytes,
    offset: usize,
    len: usize,
}
}

Walk them:

• GpuBuffer — f32 scratch: the residual stream, activations, logits. len counts floats. CPU access goes through checked as_slice / as_mut_slice views over the shared storage (buffer.rs:216-232).
• GpuHalfBuffer — f16 storage (binary16, the 16-bit “half” float; Ch 5 covers its bit layout). The doc comment says why it is a separate type and not a flag: keeping it distinct “prevents an F16 KV plane from being accidentally fingerprinted or indexed as an F32 activation buffer” (buffer.rs:34-37). A KV plane is one layer’s key/value cache buffer pair; every Metal KV plane is f16 on all lanes, and Ch 15 explains the layout.
• GpuBytes — raw bytes, and the only type that can carry an mmap. _mmap holds the file mapping alive; the leading underscore suppresses a lint for a field that exists for its lifetime, not its value — drop the last GpuBytes, the Arc count hits zero, the mapping is released.
• GpuByteView — a borrowed slice of a GpuBytes: (buffer, offset, len), created by the checked view(offset, len) method (buffer.rs:153-160, which refuses a view that would run past the end). This is the weight-tensor handle the kernels of Part IV receive.

If you know the ancestor Ferrite book: its GpuBuffer was a six-field struct with a dtype, a view length, and an arena offset. Muser’s substrate is deliberately flatter — three typed wrappers instead of one typed field, and views only where views are needed (byte storage). It can afford to be: the engine serves one model, so the dispatch code already knows every buffer’s type statically, and a dtype field would only be describing at runtime what the compiler could have enforced. That simplification was made on purpose when the substrate was pulled across from the ancestor, and the crate documentation records the “pull-and-simplify” provenance [crates/muser-engine/src/lib.rs:163-171].

3.4 Zero-init is policy, not luck

What is in a buffer at the moment you receive it, and who is allowed to read it before anyone has written it? That sounds like a pedant’s question until you notice what the wrong answer looks like in an inference engine: not a crash, but a plausible token. Garbage that happens to be zero is indistinguishable from a real value, and a stale row in a KV cache is a sentence the model half-remembers from someone else’s conversation.

Metal’s new_buffer does not zero the memory it hands you; contents are undefined. Inference is full of buffers that are partially written and then fully read (reductions with guard lanes; ring buffers whose tail rows are not yet meaningful), so Muser makes initialization explicit:

• zeros allocates and CPU-memsets — the default for everything (GpuBytes::zeros at buffer.rs:59-73, GpuBuffer::zeros at :182-196, GpuHalfBuffer::zeros at :238-244). The cost is paid once at allocation, never on the hot path.
• uninitialized exists on GpuHalfBuffer only, and its doc comment is a contract, not an invitation:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/buffer.rs:246
    /// Allocate without CPU-touching the backing bytes.
    ///
    /// This is for multi-gigabyte KV planes ONLY. Their ring/logical
    /// metadata (`MetalKvPlane::origin_logical`/`origin_physical`/`len`)
    /// guarantees every row is written by a `store_kv_*` dispatch before any
    /// row within `[origin, origin + len)` is ever read back, so the CPU
    /// zero-fill `zeros()` performs is pure startup-time cost on an
    /// allocation that can be many gigabytes. Every other caller must keep
    /// using `zeros()` -- this path leaves stale bytes behind in release
    /// builds.
    ///
    /// In debug/test builds the bytes are poisoned (never plain zero)
    /// instead of left stale, so a bug that lets a read reach a
    /// not-yet-written row is conspicuous rather than silently reading
    /// zeros; see `kv_uninitialized_write_then_read_round_trips` below.
    pub fn uninitialized(context: &MetalContext, len: usize) -> Result<Self, MetalError> {
}

That last paragraph is the elegant part, and it is the answer to the question this section opened with. In debug builds the unwritten rows are filled with 0xDEAD (buffer.rs:270-275), so a contract violation reads back conspicuously poisoned values instead of plausible zeros. Put the other way round: the fast path is allowed to leave stale bytes behind precisely because the slow path refuses to leave believable ones. A bug that lets a read reach a not-yet-written row cannot hide as a slightly odd number — it is loud, in exactly the builds where somebody is looking.

Where each path is used, precisely:

• Live session KV planes zero-fill by design. The session constructor allocates through MetalKvPlane::new → GpuHalfBuffer::zeros ([crates/muser-engine/src/decode.rs:229-244]), with an explicit in-source justification at decode.rs:1350-1352: “Zero-filled on purpose: wrapped SWA rows must never expose uninitialized storage during a sequence boundary transition.”
• Detached remote-install generations use uninitialized. When a kvpack tile arrives from the GB10 producer, a fresh plane is built with MetalKvPlane::uninitialized and every retained row is uploaded before the plane goes live ([crates/muser-engine/src/decode.rs:1862-1888]) — a bulk-write-then-publish pattern where the write-before-read guarantee is structural.

One correction while we are here, because we nearly walked into it ourselves. Writing this section we started where you would start — from the engineering doc — and docs/memory-footprint.md says flatly that “Metal KV buffers are allocated without a CPU memset.” Taken at face value that inverts everything above: it makes the fast path the default and the zero-fill the exception, and this section would have been a footnote. The two constructors above say otherwise, and an audit had already caught the same discrepancy. The doc sentence is wrong for live planes — those zero-fill, as above — and right only about the detached remote-install generations. We kept the receipt for the correction [docs/kvpack-merge-handoff.md §3 D2, the 2026-08-20 audit]. The book inherits the fix, not the error; the general lesson is that source outranks prose about source, including prose written by the same people.

3.5 Page alignment: the 16 KB contract

Before the mmap story, one piece of arithmetic — the kind that works for years and then doesn’t. The question is narrow, and it is the one to ask whenever you hand a driver a pointer to memory you did not allocate: what exactly are you promising it about that memory? Apple Silicon uses a 16 KB virtual-memory page (not the 4 KB you may know from x86). The Metal call that wraps external memory — new_buffer_with_bytes_no_copy — requires a page-aligned pointer and a page-aligned length. A file’s byte length is not, in general, a multiple of 16,384.

Muser handles this by rounding the Metal-facing length up to the page boundary while keeping the logical length exact — and the code is unusually careful about why that is safe:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/buffer.rs:91
    pub fn from_mmap(
        context: &MetalContext,
        mmap: std::sync::Arc<memmap2::Mmap>,
    ) -> Result<Self, MetalError> {
        // Metal documents that `newBufferWithBytesNoCopy:length:options:
        // deallocator:` requires a page-aligned length (the pointer is
        // already page-aligned -- POSIX `mmap` always returns one). The raw
        // file length has "worked" here only because `mmap` itself reserves
        // whole pages under the hood and zero-fills the tail of the last
        // one -- that's an undocumented tolerance, not a guarantee, so round
        // the length Metal sees up to the page boundary explicitly. Bytes
        // between the real file length and that boundary are the kernel's
        // own zero-filled mmap tail, so this never reads outside the
        // mapping. `GpuBytes::len()` keeps reporting the exact, unrounded
        // file length; only the Metal-facing allocation grows.
        let mmap_len = mmap.len();
        let rounded_len = if mmap_len == 0 {
            0
        } else {
            let page = page_size();
            mmap_len
                .checked_add(page - 1)
                .map(|padded| padded / page * page)
                .ok_or(MetalError::Allocation(mmap_len))?
        };
        let inner = context.device.new_buffer_with_bytes_no_copy(
            mmap.as_ptr() as *const std::ffi::c_void,
            rounded_len as u64,
            shared_tracked(),
            None,
        );
        // … (allocation check elided: see buffer.rs:122-124) …
        Ok(Self {
            inner,
            len: mmap_len,
            _mmap: Some(mmap),
        })
    }
}

(the allocation-size check at buffer.rs:122-124 is elided; everything else is verbatim.)

That paragraph of comment is doing real epistemic work, and it is worth slowing down for, because the bug it fixes had never once fired. Passing the raw file length straight to Metal worked. It worked because mmap reserves whole pages under the hood and zero-fills the tail of the last one, so the bytes past the file’s end were always mapped and always zero — and no test we could write would tell that apart from being right. The comment separates the two anyway: what Metal guarantees (a page-aligned length is required) from what merely happened to work (“an undocumented tolerance, not a guarantee”). Then the code goes and obeys the documented contract. Nothing was failing when that change was made; the point is that the alternative was a load path resting on an OS behaviour nobody ever promised us, and a future macOS is under no obligation to keep providing it.

Two details finish the section. A unit test pins the rounding down (from_mmap_rounds_metal_length_up_to_the_page_boundary, buffer.rs:347-376): a 10-byte file yields len() == 10 and a Metal-facing length that is a page multiple — the logical length stays exact, only the allocation grows. And the page size itself comes from POSIX getpagesize() via a one-line extern "C" — avoiding a libc dependency for one constant (buffer.rs:16-26).

Why does the pointer never need rounding here, where the ancestor engine had to round tensor offsets down to page boundaries? Because of what gets wrapped — the next section.

3.6 Zero-copy at 16.76 GB scale: mmap → one buffer → offset views

This is the payoff, end to end — the section where “the CPU and the GPU share memory” stops being an architecture diagram and becomes a load path that copies nothing. Hold the question a discrete-GPU engineer would ask first while you read it: when do the weights get uploaded? The answer is that there is no upload, and three code locations are enough to show why.

1. The engine mmaps the whole GGUF once. Loading does not read the weights into RAM; it maps the file and records where each tensor lives:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/weights.rs:171
    pub fn open(path: &Path, gguf: &GgufFile) -> Result<Self, MuseConfigError> {
        let file = File::open(path)
            .map_err(|e| MuseConfigError::Geometry(format!("open {}: {e}", path.display())))?;
        // SAFETY: the checkpoint is a read-only immutable input for the
        // lifetime of this process; we never write through the mapping.
        let mmap = unsafe { Mmap::map(&file) }
            .map_err(|e| MuseConfigError::Geometry(format!("mmap {}: {e}", path.display())))?;

        let mut index = HashMap::with_capacity(gguf.tensors.len());
        for t in &gguf.tensors {
            let start = (gguf.data_offset + t.offset) as usize;
            let n_elem: usize = t.shape.iter().product::<u64>() as usize;
            let be = t.dtype.block_elements();
            let len = n_elem.div_ceil(be) * t.dtype.block_size();
            if start + len > mmap.len() {
                return Err(MuseConfigError::Geometry(format!(
                    "tensor {} runs past end of file ({} + {} > {})",
                    t.name,
                    start,
                    len,
                    mmap.len()
                )));
            }
            index.insert(t.name.clone(), (start, len, t.dtype, t.shape.clone()));
        }
}

The index maps every tensor name to (file_offset, byte_len, dtype, shape) — with a fail-closed bounds check per tensor. The CPU reference path reads weights straight out of this mapping via TensorView (weights.rs:40-48: raw bytes plus geometry, “weights are never materialized as f32” per the module doc).

2. The Metal driver wraps that mapping in one MTLBuffer. The decode path takes the same Arc<Mmap> and hands it to GpuBytes::from_mmap from §3.5 — the entire 16.76 GB file becomes one Metal buffer, zero bytes copied:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:1199
let context = MetalContext::new()?;
let kernels = MetalKernels::new(&context)?;
let mapped_weights = GpuBytes::from_mmap(&context, weights.mapped_file())?;
let residency_set = crate::metal::residency::create_and_attach(
    &context.device,
    &context.queue,
    &[mapped_weights.metal()],
);
}

3. Every weight tensor is a checked offset view of that one buffer. Each projection remembers only its GGUF layout (TensorLayout, parsed at load, weights.rs:350-373); at dispatch time it asks the arena for a slice:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:148
    fn view<'a>(&self, mapped: &'a GpuBytes) -> GpuByteView<'a> {
        mapped
            .view(self.layout.file_offset, self.layout.byte_len)
            .unwrap_or_else(|| panic!("validated GGUF tensor {} left mapped file", self.name))
    }
}

And the bind — the line Ch 2 §2.8 promised to explain — hands the GPU “which buffer” and “where in it” in a single call:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/multicol.rs:192
encoder.set_buffer(0, Some(weights.metal()), weights.offset() as u64);
}

Put the three together and the discrete-GPU question dissolves rather than gets answered. There is no upload step because there is nowhere to upload to: the file’s pages, the Metal buffer, and the bytes a kernel dereferences are one region of physical memory wearing three names. The figure below is the entire load path — a mapping, a wrapper around it, and arithmetic.

   GGUF file on disk (16,756,681,056 B)
   ┌──────────────────────────────────────────────────────────────────────┐
   │ [header] blk.0.attn_q  blk.0.attn_k … blk.51.ffn_down  output (LM)  │
   └───────────────────────────────┬──────────────────────────────────────┘
                                   │  mmap()  (one Arc<Mmap>, read-only)
                                   ▼
   unified memory: ONE MTLBuffer over the whole mapping (no copy, ever)
   ┌──────────────────────────────────────────────────────────────────────┐
   │ file start   +q_off        +k_off      …                 file end    │
   │ ├ q.weight ──┤ ├ k.weight ─┤ ………………………………………… ├ output.weight ──┤│
   └─▲──────────────▲──────────────────────────────────────────▲─────────┘
     │              │                                            │
  GpuByteView{   GpuByteView{                              GpuByteView{
    offset:q_off, offset:k_off,                          offset:out_off,
    len:q_len,    len:k_len,                             len:out_len,
    buffer:arena} buffer:arena}                          buffer:arena}
     └─── all views share the SAME MTLBuffer; set_buffer passes the offset ─┘

Figure 3.2: One mmap → one MTLBuffer → one checked offset view per tensor (offsets symbolic; each is the tensor’s TensorLayout.file_offset from the GGUF index). No weight byte is copied at load or at dispatch; pages are demand-fetched by the OS the first time the GPU reads them.

Why one giant buffer plus views, instead of one MTLBuffer per tensor (hundreds of them)? Each distinct buffer costs Metal bookkeeping, a virtual-memory mapping, and TLB pressure (the TLB — translation-lookaside buffer — is the small cache of virtual→physical page translations; each buffer consumes entries). With one arena there is exactly one mapping, and the per-tensor “allocation” is a 24-byte struct. The ancestor book demonstrated the same trade on its engine’s arena [ferrite-book Ch 3]; Muser’s version is the same idea expressed through GpuByteView over GpuBytes. We are not alone in the choice: llama.cpp’s Metal backend maps the whole file and slices it the same way. That corroboration reaches us second-hand, through the ancestor book’s audit of ggml_metal_buffer_map [ferrite-book Ch 3] — this book has not re-read llama.cpp’s source, so we mark it lineage rather than verification.

Keeping 16 GB resident: the residency set

One more substrate piece, and it exists only because of the scale — at a few hundred megabytes nobody would bother. The mapped arena is bound by every projection in every command buffer — per token. Rather than let Metal redo residency bookkeeping for a 16+ GiB allocation each time, Muser attaches it to an MTLResidencySet once at load:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/residency.rs:1
//! Minimal `MTLResidencySet` owner extracted from Ferrite's Metal substrate.
//!
//! The immutable GGUF arena is bound by every projection in every command
//! buffer. Attaching it once lets Metal skip repeating residency work for the
//! 16+ GiB allocation on every token. The Objective-C surface is public on
//! macOS 15+, and absence fails open to the ordinary Metal residency path.
}

create_and_attach (residency.rs:65-108) builds the set with raw objc::msg_send! calls — the metal crate does not wrap this API — adds the arena’s buffer, commits, requests residency, and attaches the set to the queue. On any macOS release without the API it returns None and the engine proceeds on Metal’s ordinary residency path. That distinction matters more than it looks: this is an optimization that fails open. Lose it and tokens still come out, correct, with Metal doing the residency bookkeeping the long way — which is exactly the opposite of the untracked gamble earlier in this chapter, where dropping the safety net changed behaviour silently. Both are engine paths that may not be taken; only one of them is permitted to be quiet about it.

3.7 What 96 GB buys, revisited from the buffer side

Ch 1 §1.5 budgeted the 96 GB by artifact — how much each thing takes. Now that you know what a buffer actually is, the same budget answers a sharper question: which of these bytes does the engine own, and which is it merely borrowing from the operating system? From the substrate’s point of view the same budget reads:

• Weights are not “used” memory in the ordinary sense — they are a read-only file mapping, page-cache backed [docs/memory-footprint.md §Other material allocations]. The OS can evict clean pages under pressure and re-fault them on next access. What the residency set adds is a request that the pages stay resident while serving.
• Everything the engine allocates itself — activations, logits, workspaces, the f16 KV planes, staging shadows — is a StorageModeShared buffer from §3.3. The KV planes dominate: 1.827 GB per slot at full 131,072 context, ×4 slots = 7.306 GB [docs/memory-footprint.md].
• One arena serves all slots. The MetalShared design ([crates/muser-engine/src/decode.rs:954-957]) exists because unified memory makes it legal: four serving sequences share one context, one pipeline set, one mapped arena. On a discrete-GPU machine the same sharing would be possible but the copy story would not — each GPU residency of the weights would be a distinct 16.76 GB VRAM occupation.

3.8 Tradeoffs

Tracked storage vs the untracked gamble. The story is in §3.2’s lineage box; here is what it costs. Muser pays for Metal’s dependency tracker on every encode, plus the explicit barrier groups, and gets back a safety net that the one experiment we ran showed it needed: the engine-wide untracked build (b9678d4) “empirically changed DFlash conditioning while leaving final greedy IDs unchanged” [crates/muser-engine/src/metal/buffer.rs:7-14]. Now notice what that sentence does not say. It does not say the untracked path was slower, or faster — no Muser document isolates the tracked-vs-untracked encode cost on this machine [unverified]. The ruling was made on correctness alone, and it would stand unchanged if untracked turned out to measure quicker.

One arena + views vs one buffer per tensor. Views are cheap (24 bytes, one bounds check) and make the whole model one mapping — but they collapse identity: Metal sees a single buffer, so its tracker cannot distinguish a write to one tensor from a write to another. That is exactly why the weights are views (immutable after load; a write would be a bug worth a crash) while activations and KV planes get their own typed buffers with distinct identity. The split is the design: share what is immutable, individualize what is written.

uninitialized KV planes: speed with a proof obligation. Skipping the CPU memset on multi-gigabyte planes saves real startup time on allocations that can be many gigabytes (buffer.rs:248-254) — in exchange, the ring metadata must guarantee write-before-read for every row in [origin, origin + len). Muser makes the obligation visible three ways: the doc comment’s “ONLY”, the debug-build 0xDEAD poison, and the test that demonstrates it (buffer.rs:328-344). That is the shape of the whole bargain, and it generalizes past this buffer: when you take a shortcut whose safety lives in some other module’s invariant, spend part of the winnings making that invariant loud.

3.9 What’s next

You now know the entire memory story: one DRAM pool, StorageModeShared everywhere, three typed buffer types plus checked byte views, zero-copy mmap of the 16.76 GB GGUF into one Metal buffer, a residency set to keep it warm, and an initialization policy that decides byte-for-byte what may be read before it is written. One object from Ch 2 is still a black box: MetalContext.library — “a bundle of compiled kernel functions, addressable by name.” How .metal source text becomes that bundle, why Muser deliberately keeps three different kernel sources (fast-math source, strict-f32 source, and a pinned llama.cpp metallib), and how the engine makes a silent kernel fallback impossible — that is Ch 4.

References

• [crates/muser-engine/src/metal/buffer.rs:7-14] — shared_tracked() and the b9678d4 untracked-mode incident comment; :28-56 the four substrate types; :59-73, 182-196, 238-244 the zero-initializing allocators; :246-277 GpuHalfBuffer::uninitialized and the 0xDEAD poison; :91-130 from_mmap with the page-rounding contract; :328-376 the poison and page-rounding tests; :16-26 getpagesize.
• [crates/muser-engine/src/weights.rs:163-218] — MuseWeights::open (whole-file mmap + per-tensor index with bounds checks); :40-48 TensorView; :350-373 TensorLayout; :375-378 mapped_file().
• [crates/muser-engine/src/decode.rs:1195-1206] — load_shared: from_mmap + residency set; :148-161 Projection::view / nvfp4_scale_view; :229-244, 246-261 the two MetalKvPlane constructors; :1350-1352 the live-plane zero-fill rationale; :1862-1888 the detached remote-install generation; :954-984 MetalShared.
• [crates/muser-engine/src/metal/residency.rs:1-107] — the MTLResidencySet owner (raw msg_send!, fails open below macOS 15).
• [crates/muser-engine/src/metal/encode/multicol.rs:192] — the real offset-view set_buffer bind.
• [docs/memory-footprint.md] — artifact sizes, KV formula, the on-disk-vs-resident caveat.
• [docs/kvpack-merge-handoff.md §3 D2] — the 2026-08-20 audit correcting memory-footprint.md’s memset claim for live planes.
• [ferrite-book Ch 3] — the ancestor’s unified-memory chapter: the arena-view and zero-copy devices this chapter re-grounds; its untracked-hazards A/B (−2 to −4 % on A18-class hardware) and GpuHeap / packed-activations history are Ferrite-lineage.
• [Metal-PG] — Apple, Metal Programming Guide: “Resource Objects: Storage Modes,” “Tracking Resource Dependencies.”
• [Metal-SS] — Apple, Metal Shading Language Specification: “Address Spaces.”
• glossary — terms introduced this chapter: unified memory, SoC, DRAM, VRAM, PCIe, storage mode, blit, mmap, page fault, zero-copy, page alignment, TLB, MTLBuffer, GpuBuffer, GpuHalfBuffer, GpuBytes, GpuByteView, residency set.

Chapter 4 — Pipeline state objects and the three kernel sources

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 and Ch 3. You know what MetalContext’s library, cross_vendor_library, and ggml_library fields are for, and what a dispatch_thread_groups call looks like.

4.1 Three stages between text and machine code

In Ch 2 we said the host API gives you “a bundle of compiled kernel functions, addressable by name,” and deferred how a .metal text file becomes that bundle. That deferral is now due, and it turns out to matter for more than curiosity: Muser feeds three different kernel sources into one dispatch path, and a benchmark that cannot say which of the three ran is not a measurement of anything. So this chapter goes in two movements — first the machinery, then the discipline that keeps the machinery honest.

Start with the machinery. There are three stages between “text a human wrote” and “a kernel the GPU can run”:

1. .metal source text — a C++ dialect a human writes. Pure text.
2. MTLLibrary — a bundle of compiled functions: the text has been parsed, type-checked, and lowered to an intermediate representation. No machine code for any specific GPU exists yet. A .metallib file is exactly this stage serialized to disk — a library you can load without ever running the frontend compiler.
3. MTLComputePipelineState — the PSO — one function taken from the library and lowered all the way to machine code for this specific GPU. This PSO is the handle you hand to set_compute_pipeline_state on an encoder.

Figure 4.1 shows how Muser’s three kernel sources feed that last stage.

flowchart LR
    A["24 .metal files<br/>include_str! + concat!"] -->|"new_library_with_source<br/>fast-math ON"| B["library<br/>(serving kernels)"]
    A2["muse_reference + nvfp4<br/>(same source)"] -->|"new_library_with_source<br/>fast-math OFF"| C["cross_vendor_library"]
    D["pinned llama.cpp .metallib<br/>MUSER_GGML_METALLIB"] -->|"new_library_with_file"| E["ggml_library<br/>(optional)"]
    B --> F["get_function(name,<br/>± function constants)"]
    C --> F
    E --> F
    F --> G["new_compute_pipeline_state<br/>PSO per kernel"]
    G --> H["MetalKernels registry<br/>(PsoCache + typed fields)"]

Figure 4.1: Muser’s kernel pipeline. Three sources feed one PSO-building path; every kernel lands in the MetalKernels registry before any dispatch happens.

Why split source-to-runnable into two compiles? The library stage is frontend work, identical for every GPU model, done once for all kernels together. The PSO stage is backend work, specific to this device’s instruction set, done once per kernel [Metal-PG, "Functions"]. Splitting them lets you pay each at a different time — and lets a prebuilt .metallib skip the frontend entirely on the user’s machine.

One design fact up front, because it shapes everything below: Muser has no build-time shader-compilation step. The engine’s module doc is explicit — the substrate keeps “runtime shader compile (include_str! the .metal sources in shaders/, newLibraryWithSource on first use, cache PSOs — no Xcode step, pure-source checkout)” ([crates/muser-engine/src/lib.rs:163-171]). The ancestor Ferrite engine precompiled a .metallib in build.rs and kept an on-disk MTLBinaryArchive PSO cache [ferrite-book Ch 4]; Muser deliberately dropped both machines’ worth of build plumbing — cold-start compile is paid at engine init, once, and the PSO cache is in-process. The tradeoffs section (§4.9) costs this out.

4.2 Source 1: the concatenated fast-math library

Begin with the library the engine actually serves tokens from. The question is a small one with a long shadow: what exactly gets compiled, under which compiler settings, and what could the order of the files possibly have to do with anything?

The main library is one giant source string built from 24 .metal files with include_str! and concat!, compiled once at MetalContext::new():

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/context.rs:48
let options = CompileOptions::new();
// Ferrite's accepted production kernels and llama.cpp both compile
// with fast math enabled. Exact Muse parity is guarded at the token
// boundary; disabling this here materially slows attention, FFN, and
// the norm/tiny-op stack without changing the imported GGML PSOs.
options.set_fast_math_enabled(true);
options.set_language_version(MTLLanguageVersion::V3_1);
// The fixed Muse driver is local, while the operation kernels below
// are clean extractions from Ferrite at a85048a90. Keeping the exact
// source files separate makes their provenance and future diffing
// auditable without bringing over Ferrite's runtime or route VM.
let source = concat!(
    include_str!("../shaders/muse_reference.metal"),
    "\n",
    include_str!("../shaders/nvfp4.metal"),
    "\n",
    include_str!("../shaders/ferrite/sigmoid_gate.metal"),
    "\n",
    // … (twenty more ferrite/ files elided, in dependency order —
    //     see context.rs:66-106) …
    include_str!("../shaders/ferrite/flash_attn_decode_reduce_v2.metal"),
);
let library = device
    .new_library_with_source(source, &options)
    .map_err(MetalError::ShaderCompile)?;
}

(the elided lines are consecutive include_str! entries; nothing else is removed.)

Three choices are buried in those few lines, and each one is a place where the obvious alternative would have cost us something.

Fast math ON for the serving library. set_fast_math_enabled(true) lets the compiler assume NaN/Inf never happen and reorder or fuse floating-point operations. The comment records both the justification (llama.cpp compiles the imported kernels with fast math; turning it off “materially slows attention, FFN, and the norm/tiny-op stack”) and the safety argument — “exact Muse parity is guarded at the token boundary,” i.e., the exactness contract is enforced by comparing generated tokens against the pinned comparator, not by hoping the arithmetic is bit-stable.

Fast math is not safe everywhere, though, and the ancestor found the edge the hard way. Ferrite let the fast-math compiler near RoPE’s trigonometry, where the rotation angle grows with position; at large angles the fast approximations drift, and the repair was to force precise::cos and precise::sin on exactly those calls [ferrite-book Ch 4 §10]. Muser does not repeat the repair — it removes the hazard. The RoPE frequency table is precomputed on the CPU at load ([crates/muser-engine/src/decode.rs:1256-1263]), so no powf and no trig call survives into the kernel to be approximated in the first place. The distinction is worth holding on to: fast math is cheap insurance on a long chain of multiply-adds whose result you check at the token boundary, and a liability wherever the absolute accuracy of a single transcendental is the thing you depend on.

MSL V3.1, pinned. set_language_version(V3_1) fixes the dialect the compiler accepts. A shader using a newer feature fails loudly at library-build time instead of silently miscompiling.

Concatenation order is load-bearing. The .metal fragments share types and helpers across files (the Q4_K block structs in matmul.metal, the shared MAC helpers in _q4k_helpers.metal), and MSL here has no #include resolution — the files are one translation unit precisely because they are concatenated in dependency order. The provenance comment is part of the design: which files are Muser-authored (muse_reference.metal, nvfp4.metal) versus clean extractions from Ferrite at a85048a90 is readable straight from the concat ([crates/muser-engine/src/metal/context.rs:55-58]), backed by the per-file SHA-256 manifest in [docs/extraction-manifest.md].

4.3 Source 2: the strict-f32 cross-vendor library

Most engines have one shader library. Muser has a second copy of some of the same kernels, and the reason has nothing to do with this Mac — it has to do with a machine on the other end of a network cable. Watch for the inversion: here a compiler flag stops being a build setting and becomes part of an API.

The second library is the same two source files, recompiled with fast math off:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/context.rs:111
let cross_vendor_options = CompileOptions::new();
cross_vendor_options.set_fast_math_enabled(false);
cross_vendor_options.set_language_version(MTLLanguageVersion::V3_1);
let cross_vendor_source = concat!(
    include_str!("../shaders/muse_reference.metal"),
    "\n",
    include_str!("../shaders/nvfp4.metal"),
);
let cross_vendor_library = device
    .new_library_with_source(cross_vendor_source, &cross_vendor_options)
    .map_err(MetalError::ShaderCompile)?;
}

Why would an engine compile its own kernels twice with different compiler flags? Because of Part VI. When a remote NVIDIA GB10 producer computes KV (or a draft) and the Mac must reproduce or verify it, the arithmetic must match CUDA’s explicit scalar boundaries — and a fast-math compiler is free to fuse and reassociate exactly where CUDA’s kernel did not. The struct field’s doc comment states the contract:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/context.rs:36
/// Strict-f32 copy of the standalone Muse kernels.  The cross-vendor
/// Q8 projection and integer NVFP4 routes must match CUDA's explicit
/// scalar boundaries, while the ordinary serving kernels retain fast math.
}

The strict copies are selected per route, not globally: dispatch wrappers check MUSER_CROSS_VENDOR_QK and swap in the strict pipeline for exactly the ops that must match the producer (e.g. [crates/muser-engine/src/metal/encode/gate.rs:14-18], [crates/muser-engine/src/metal/encode/norm.rs:257-260]). One source, two compilation contracts, selected at dispatch — the compiler flag becomes part of the numerical API. Put it the other way round, because this is the part that trips people up: a fast-math setting is normally something you choose once and never think about again, buried in a build file. Here it is a value the dispatch code reads at encode time, as consequential to the result as a dtype. Ch 32 is where this discipline earns its keep.

4.4 Source 3: the pinned llama.cpp metallib

The third source answers a question the first two cannot. What do you do about a kernel whose output you must reproduce exactly, when the thing you have to match is somebody else’s compiled binary? Rewriting it in your own dialect gets you close, and close is the one answer this engine cannot use. So Muser does not rewrite it at all.

Which is why the third source is not Muser source at all. When MUSER_GGML_METALLIB points at a prebuilt llama.cpp .metallib, the device loads it as a library:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/context.rs:122
let ggml_library_path = std::env::var_os("MUSER_GGML_METALLIB").map(PathBuf::from);
let ggml_library = match ggml_library_path.as_ref() {
    Some(path) => Some(device.new_library_with_file(path).map_err(|message| {
        MetalError::GgmlLibrary {
            path: path.clone(),
            message,
        }
    })?),
    None => None,
};
}

This library supplies llama.cpp’s own kernels — kernel_mul_mv_q{4,5,6}_K_f32 matvecs, kernel_mul_mm_* batch matmuls, kernel_rms_norm_mul_f32_4, kernel_rope_norm_f32, the whole flash_attn_ext family — which Muser dispatches instead of re-expressing them. The reasoning is recorded in the registry:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:278
ggml_q4k: ggml_matvec_pipeline(context, "kernel_mul_mv_q4_K_f32")?,
ggml_q5k: ggml_matvec_pipeline(context, "kernel_mul_mv_q5_K_f32")?,
ggml_q6k: ggml_matvec_pipeline(context, "kernel_mul_mv_q6_K_f32")?,
// … (matmul / norm / rope / flash-attn families elided — encode.rs:281-293) …

// crates/muser-engine/src/metal/encode.rs:370
pub(crate) fn supports_projection(&self, dtype: crate::gguf::GgmlType) -> bool {
    match dtype {
        // Standalone fallbacks cover Q4_K and Q5_K for both decode and
        // batch prefill. Q6_K intentionally uses the pinned upstream
        // llama kernels so its math and dispatch remain comparator-exact.
        crate::gguf::GgmlType::Q4_K | crate::gguf::GgmlType::Q5_K => true,
        crate::gguf::GgmlType::Q6_K => {
            self.ggml_q6k.is_some()
                && self.ggml_q6k_mm_aligned.is_some()
                && self.ggml_q6k_mm_bounds.is_some()
        }
        // …
    }
}
}

Two different policies live in that one match. Q4_K and Q5_K have Muser-authored standalone kernels — the metallib versions are preferred (for parity) but the engine can run without them. Q6_K has no fallback: it runs on llama’s kernels or it does not run. If a GGUF carries a Q6_K projection and the metallib is missing, model load aborts with a specific, actionable error:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:113
#[error(
    "tensor {name} uses {dtype:?}, which requires the pinned llama.cpp Metal library; set MUSER_GGML_METALLIB"
)]
MissingProjectionKernel { name: String, dtype: GgmlType },
}

raised per-tensor at load ([crates/muser-engine/src/decode.rs:1294-1310]). This is a fail-closed no-fallback policy: the engine refuses to silently substitute different arithmetic for a dtype whose exactness is contractual.

None of that strictness means anything if the metallib is itself a mystery binary — pinning your arithmetic to a file whose origin nobody can state is not pinning at all. So the build of that file is the strictest step in the whole chain. The metallib itself is built by [scripts/compile_llama_metallib.sh], and the script is almost a manifesto of provenance discipline: it refuses to run unless the llama.cpp checkout’s HEAD equals the requested revision and the three Metal source files are clean in index and working tree (:57-69); it refuses to replace an existing output or receipt, making artifacts append-only (:78-85); and it writes a muser.llama_metallib.source_receipt.v1 JSON binding the binary’s SHA-256 and size to the source commit, the source tree hash, per-file SHA-256s, the merged-source hash, the SDK version, the Metal compiler, and the Xcode version (:129-180). Two refusals and a receipt: the artifact cannot come from a dirty tree, cannot be quietly replaced by a newer one wearing the same name, and cannot be used without a record of which commit and which toolchain produced it.

4.5 From functions to PSOs: the registry and the cache

A library is not runnable; the last stage turns named functions into machine code for this GPU. The interesting question about any cache is what it does when it misses, and this one answers it in a way most caches would not dare to.

Once the libraries exist, every kernel is compiled to a PSO exactly once, at MetalKernels::new. The fixed serving set is a compile-time-checked list of 66 names:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:21
const PIPELINES: [&str; 66] = [
    "rms_norm_batch",
    "fused_rms_norm_residual_add_batch",
    "muser_fused_norm_residual_rms_norm_batch_dual_eps",
    "muser_fused_norm_residual_rms_norm_32sg",
    "sigmoid_gate_inplace",
    // … (61 more names elided — encode.rs:27-88) …
    "muser_embedding_f16",
];
}

and the cache that builds them is 40 lines:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/pso_cache.rs:9
pub struct PsoCache {
    states: HashMap<&'static str, ComputePipelineState>,
}

impl PsoCache {
    pub fn new(
        context: &MetalContext,
        names: impl IntoIterator<Item = &'static str>,
    ) -> Result<Self, MetalError> {
        let mut states = HashMap::new();
        for name in names {
            // Metal requires functions that declare function constants to be
            // obtained through the constant-values API even when every value
            // intentionally remains undefined and the shader uses its default
            // branch. This mirrors Ferrite's `make_fc_default` constructor.
            let constants =
                matches!(name, "ffn_q4k_gate_up_silu_4r2s").then(FunctionConstantValues::new);
            let function = context
                .library
                .get_function(name, constants)
                // … (error mapping elided) …
            let state = context
                .device
                .new_compute_pipeline_state_with_function(&function)
                // … (error mapping elided) …
            states.insert(name, state);
        }
        Ok(Self { states })
    }

    pub fn get(&self, name: &'static str) -> &ComputePipelineStateRef {
        self.states
            .get(name)
            .unwrap_or_else(|| panic!("unregistered Muse Metal pipeline {name}"))
    }
}
}

Three decisions are visible in those forty lines. The first one costs time: the cache is in-process only — there is no on-disk MTLBinaryArchive, so every process start recompiles all 66 PSOs plus the specialized families below. The second buys safety at the price of politeness. The panic on a miss is the policy: a typo’d kernel name takes the process down at first use, loudly, rather than resolving to something else and dispatching it. In an engine whose correctness gate is “same logits as the comparator,” running the wrong kernel successfully is the failure you least want to survive. The third decision is the comment about function constants, and it is the whole of the next section.

Beyond the 66, MetalKernels holds typed fields for everything optional or specialized: the cross-vendor PSOs (one field each, [crates/muser-engine/src/metal/encode.rs:92-116]), the Option<...> ggml/llama families (:117-132), and the Ferrite f16 attention specializations (:135-138). Optionality is visible in the type — an absent metallib is None, not a missing string in a map.

4.6 Function constants: one source, many specialized kernels

A function constant is a value a shader declares with [[ function_constant(N) ]] and the host supplies at PSO-build time. The compiler then specializes: with the value known, it unrolls loops, eliminates dead branches, and folds the constant into machine code. Muser uses this everywhere the imported llama.cpp kernels demand it, because llama.cpp’s own kernels are written as one source with dozens of specialization points.

The pattern, on the Ferrite f16 attention family:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:798
fn ferrite_f16_pipeline(
    context: &MetalContext,
    name: &str,
    nsg: u32,
) -> Result<ComputePipelineState, MetalError> {
    let constants = FunctionConstantValues::new();
    let head_dim = 128u32;
    let decode_params = false;
    constants.set_constant_value_at_index(
        &head_dim as *const u32 as *const std::ffi::c_void,
        MTLDataType::UInt,
        40,
    );
    constants.set_constant_value_at_index(
        &decode_params as *const bool as *const std::ffi::c_void,
        MTLDataType::Bool,
        92,
    );
    constants.set_constant_value_at_index(
        &nsg as *const u32 as *const std::ffi::c_void,
        MTLDataType::UInt,
        98,
    );
    let function = context
        .library
        .get_function(name, Some(constants))
        // …
}

Call it three times with nsg = 1, 2, 4 and you get three PSOs of the same source — which is exactly what MetalKernels::new does ([crates/muser-engine/src/metal/encode.rs:298-313]), mirroring llama.cpp’s own “launch 32 workgroups and grow simdgroups” dispatch table noted at [crates/muser-engine/src/decode.rs:41-46].

The llama kernels go further. The matvec constructor pins four slots:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:844
let constants = FunctionConstantValues::new();
for (value, index) in [(2i16, 600u64), (1, 602), (1, 603), (1, 604)] {
    constants.set_constant_value_at_index(
        &value as *const i16 as *const std::ffi::c_void,
        MTLDataType::Short,
        index,
    );
}
}

and every specialized constructor carries a label that states the specialization, which becomes the error string if the function is absent from the metallib:

kernel_flash_attn_ext_vec_f16_dk128_dv128[ns=128,nsg=1,kvpad=false,mask=true]
kernel_flash_attn_ext_f16_dk128_dv128[mask=true,kvpad=false,ns=128,nsg=4]
kernel_flash_attn_ext_blk[nqptg=8,ncpsg=32]
kernel_mul_mv_ext_q4_K_f32_r1_2[nsg=2,nxpsg=8,ne12=1,r2=1,r3=1]

(the four labels as constructed at [crates/muser-engine/src/metal/encode.rs:985, 1116, 1123, 1192])

That label convention is doing double duty: it is a debug message, and it is a fingerprint — §4.7 makes that load-bearing. The slot numbers are llama.cpp’s own function-constant indices (600+ for mul_mv, 700+ for mul_mm, 800+ for RoPE, 1000+ for the flash-attn family), so the Rust side reads as a table of the upstream kernel ABI. The mul_mv_ext group even documents why it exists: “llama.cpp’s source-pinned small-batch K-quant projection pipelines… Upstream changes from repeated mul_mv to these mul_mv_ext kernels at batch size four. Keeping that dispatch boundary is required for numerical API parity as well as performance parity” ([crates/muser-engine/src/metal/encode.rs:169-178]).

One family sits outside the concat, and we got it wrong on the first pass. The detour is worth taking, because the mistake is the kind this book exists to prevent. The multi-column matvec family (matvec_multicol.metal) is compiled by its own new_library_with_source call rather than living in the main source string. Its module doc at multicol.rs:12-14 states plainly that “nothing here is compiled unless MUSER_MULTI_COL_VERIFY is set”, and we took that at face value — an experimental family behind an env var is exactly what you would expect, and it meant this whole group could be left out of the startup accounting.

Then we read the constructor, and it does not agree with the doc. The multicol builder is called unconditionally, because the exact multi-sequence decode route is always on; only the experimental DFlash verify route is still gated by that variable ([crates/muser-engine/src/metal/encode.rs:294-297], [crates/muser-engine/src/metal/encode/multicol.rs:90-132]). The comment is simply stale. The lesson is the cheap one to learn here rather than downstream: a module doc describes an intent at the moment it was written, and the constructor describes what your process does tonight. The code wins, as ever in this book — and the multicol pipelines are counted in the cold-start tally of the next section, which they would not have been if we had trusted the prose.

4.7 Fingerprinted selection: making a silent fallback impossible

Here is the chapter’s real subject, and the question the machinery above exists to answer: when you run the engine, how do you know which kernels ran? An engine with three kernel sources has four ways to not use the kernel you think it is using: the env var is unset, the metallib failed to load, the flag is inert, the fallback route took over. None of those four announces itself. Each silently changes the arithmetic — and therefore the numbers — of every benchmark that follows, which means a performance result and a correctness result can both be true of a route nobody intended to measure. Muser’s answer is layered.

Layer 1 — absence is loud where absence matters. We have already met the sharpest version of this in §4.4: a Q6_K projection with no metallib does not fall back, it aborts the load with MissingProjectionKernel. Where a fallback is permitted, the same honesty moves into the test suite. A test that needs the pinned kernels does not quietly pass on the fallback path and let you read its green tick as evidence about llama’s kernels; it says "skipping: MUSER_GGML_METALLIB is unset" and declines to run ([crates/muser-engine/src/metal/encode/multicol.rs:462]).

Layer 2 — the record states what ran, by hashing it. The benchmark harness resolves its actual route and records a SHA-256 of the metallib that actually loaded — a resolved signal, not an echo of the environment:

#![allow(unused)]
fn main() {
// crates/muser-bench/src/main.rs:317
let Some(path) = std::env::var_os("MUSER_GGML_METALLIB").map(PathBuf::from) else {
    return Ok(RouteIdentity {
        matvec_route: "muser-local-q4k-q5k",
        // … (other route fields elided) …
        ggml_metallib_sha256: None,
    });
};
let bytes = std::fs::read(&path).map_err(|error| {
    format!(
        "cannot fingerprint GGML metallib {}: {error}",
        path.display()
    )
})?;
let digest = Sha256::digest(bytes);
Ok(RouteIdentity {
    matvec_route: "llama-ggml-metallib",
    // …
}

matvec_route is derived from whether the library loaded, and ggml_metallib_sha256 binds the run to the exact artifact bytes. A qualifier binary goes further and refuses ambiguity: if MUSER_GGML_METALLIB is already set but differs from the --ggml-metallib argument, it errors out rather than running with either ([crates/muser-bench/src/composite_dflash.rs:246-251]).

Layer 3 — artifacts are append-only and receipted. A hash of the loaded file only helps if that file has a history, which is the job of the build script from §4.4: it will not overwrite an existing output, and it emits the source receipt that binds the binary to its commit, its tree, its per-file hashes, and its toolchain. That receipt is not just for whoever built the file. Its path travels — the node-onboarding orchestrator accepts it via MUSER_GGML_METALLIB_RECEIPT ([crates/muser-server/src/node/mod.rs:80-83]), so even the remote-lane qualification knows which provenance it measured.

Lineage — the hole that made this discipline necessary. The ancestor Ferrite engine had an optional llama.cpp metallib bridge, and when the env var was unset it “falls back to JIT with a warning” [ferrite-book Ch 4 §4.4] — a silent fallback in practice, because nothing in the benchmark record said which path had run. Its correction register records the confirmed defect: “silent metallib fallback can time the ~12 %-slower native path unknowingly” (D06-3, this book’s _research/ferrite-port-map-2.md digest of the ancestor’s CORRECTIONS register; the ~12 % figure is a Ferrite-source-comment number, Ferrite-lineage). The fix there was a fingerprint line printing resolved signals (lib=metallib+ggml-bridge), and the same fix caught a second hole — an inert flag that changed nothing while appearing to [ferrite-book Ch 4 §4.4.4]. Muser’s Layer-2 hashing and Layer-1 refusals are that lesson, built in from day one: a benchmark that cannot state which kernels ran is not evidence.

4.8 What it costs to start the engine

Every choice in this chapter — compile at runtime, keep two contracts of the same kernels, load a third library from disk — is paid in the same currency, at the same moment: process start, while somebody waits. So what does the reader’s engine actually do before the first token appears?

At MetalMuseModel load, the engine compiles: the 24-file fast-math library, the 2-file strict-f32 library, optionally loads the metallib, then builds 159 PSOs — 66 registry PSOs, 25 cross-vendor PSOs, 55 ggml/llama family PSOs (matvecs, matmuls, mul_mv_ext, and the flash-attention specializations), 9 multicol PSOs, and 4 Ferrite f16 PSOs, counted from the constructor at [crates/muser-engine/src/metal/encode.rs:202-315]. All of that happens before any weight page is touched (Ch 3’s mmap is demand-paged behind this) — the model file is barely being read while the compiler works.

We went looking for the wall-clock figure that belongs at the end of that paragraph, and it does not exist: no Muser document measures the total cold-start compile time [unverified]. The absence is worth naming rather than papering over, because it is where this chapter’s evidence runs out. What is recorded is the bet — that a one-time compile at init is simpler than shipping and versioning a build artifact — in the module doc’s own words, “no Xcode step, pure-source checkout” ([crates/muser-engine/src/lib.rs:163-171]). The book can tell you the bet was taken deliberately; it cannot yet tell you the premium.

4.9 Tradeoffs

Runtime JIT vs prebuilt metallib (vs the ancestor’s binary archive). Muser compiles its own kernels at every start; the ancestor precompiled at build time and cached PSOs on disk [ferrite-book Ch 4]. Muser’s win is provenance simplicity: the kernel source is the shipped artifact, and there is no .metalbin to go stale or drift from the binary. The cost is the per-start compile of the 159 PSOs counted in §4.8 — unmeasured in any Muser doc [unverified], paid once per process. Note the asymmetry with the llama kernels: for those, Muser does ship a prebuilt metallib — because there the goal is not convenience but bit-parity with a pinned upstream build, which a local recompile could not guarantee (the source-receipt toolchain of §4.4 exists to make that guarantee auditable).

Fast-math and strict-f32 side by side. Compiling muse_reference + nvfp4 twice doubles the frontend work for those two files and splits the kernel namespace across two libraries. The payoff is that numerical parity with the CUDA producer becomes a compilation flag selected per dispatch, not a rewrite. The alternative — one strict library everywhere — was measured by implication in the source comment: disabling fast math “materially slows attention, FFN, and the norm/tiny-op stack” ([crates/muser-engine/src/metal/context.rs:49-52]); the exact percentage is not recorded [unverified]. The deeper alternative is tempting enough to name plainly: trust fast-math everywhere and rely on token-level tolerance for cross-vendor checks, and the second library disappears entirely. Ch 32 is the chapter that closes that door. During the wizard’s arithmetic-ABI chase, one f16 ULP in a layer-1 V tile was enough to cascade into 51.7 M differing logits [ledger §2b, attempts 10–31]. A tolerance wide enough to absorb a divergence like that is a tolerance wide enough to absorb a bug, which is the same as having no gate at all.

Pinning llama’s kernels vs re-expressing them. Muser could have rewritten the Q6_K matvec or the flash-attention family in its own source and dropped the metallib dependency. The registry comment gives the reason it did not: Q6_K “intentionally uses the pinned upstream llama kernels so its math and dispatch remain comparator-exact” ([crates/muser-engine/src/metal/encode.rs:370-375]). Re-expressed kernels agree only to ULP — the multi-column family documents its own Q6_K case, where the separately compiled body “differs by a few ULP” and is therefore excluded from the bitwise-exact route ([crates/muser-engine/src/metal/encode/multicol.rs:208-211]). When your correctness gate is “same logits as the comparator,” ULP is not a detail; it is the whole game — so the engine pins the comparator’s own machine code.

4.10 What comes next

You now know the complete Metal substrate: how work is submitted (Ch 2), where memory lives (Ch 3), and how source text becomes the three kernel libraries whose selection is fingerprinted and fail-closed (this chapter). Part I is done — you can now read every shader and every dispatch in this book.

Kernels are language. The thing they read is the problem: 27.85 billion parameters that must fit in 16,756,681,056 bytes and still produce exact arithmetic. How four-ish bits per weight can carry a 30B model, what a block scale buys, and why NVFP4 and kquant land at parity — that is Part II, and it starts with Ch 5.

References

• [crates/muser-engine/src/metal/context.rs:32-42] — MetalContext with the three libraries; :46-110 device/queue + the 24-file fast-math concat; :111-121 the strict-f32 recompile; :122-131 metallib loading; :149-211 the deadline-bounded wait (Ch 2).
• [crates/muser-engine/src/lib.rs:163-171] — the “no Xcode step, pure-source checkout” substrate design note.
• [crates/muser-engine/src/metal/encode.rs:21-88] — the 66-name PIPELINES registry; :90-139 MetalKernels’ typed fields; :202-315 the constructor wiring all three sources; :278-293 ggml pipeline construction; :370-385 supports_projection (the Q6_K no-fallback policy); :798-835 ferrite_f16_pipeline (function-constant slots 40/92/98); :837-866 ggml_matvec_pipeline (slots 600+); :957-1003 the mul_mv_ext group and FC_MUL_MV; :1079-1210 the flash-attn families with labeled specializations.
• [crates/muser-engine/src/metal/pso_cache.rs:9-50] — the in-process PSO cache and its panic-on-miss accessor.
• [crates/muser-engine/src/metal/encode/multicol.rs:90-132, 208-211, 458-464] — the lazy multi-column library, the Q6_K ULP note, and the honest skip.
• [crates/muser-engine/src/decode.rs:41-46] — llama’s nwg=32/nsg-growth dispatch note; :113-116, 1294-1310 — MissingProjectionKernel; :1256-1263 — CPU-built RoPE frequency table.
• [scripts/compile_llama_metallib.sh] — revision pinning, clean-tree checks, append-only outputs, the source_receipt.v1 schema.
• [crates/muser-bench/src/main.rs:304-341] — route_identity: resolved route + metallib SHA-256 fingerprint.
• [crates/muser-bench/src/composite_dflash.rs:246-251] — the env-vs-arg ambiguity refusal.
• [crates/muser-server/src/node/mod.rs:80-83] — MUSER_GGML_METALLIB / MUSER_GGML_METALLIB_RECEIPT plumbing in node onboarding.
• [docs/extraction-manifest.md] — per-file provenance for the ferrite/ shader extractions.
• [ledger §2b] — the wizard attempts 10–31 arithmetic-ABI chase (one f16 ULP → 51.7 M differing logits; attempts 9/31 verdicts).
• [ferrite-book Ch 4] — the ancestor’s compilation chapter: the build-time metallib + MTLBinaryArchive cache Muser dropped, the precise:: fast-math lesson, and the silent-metallib-fallback hole (D06-3) whose fix is this chapter’s §4.7.
• [Metal-PG] — Apple, Metal Programming Guide: “Functions,” “Pipeline State Objects,” “Binary Archives.”
• [Metal-SS] — Apple, Metal Shading Language Specification: “Function Constants.”
• glossary — terms introduced this chapter: PSO, MTLLibrary, metallib, JIT compilation, function constant, fast-math, cross-vendor library, PsoCache, fingerprint, fail-closed.

Chapter 5 — Quantization from scratch

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Chapters 1–4. You know why one decode token costs roughly “the whole model in bytes” (Chapter 1’s bytes-per-token spine), and you know that Metal kernels are compiled from one of three fingerprinted sources (Chapter 4). This chapter never launches a kernel — it is pure arithmetic on how a number becomes fewer bytes. The concrete formats arrive in Ch 6 and Ch 7.

Chapter 4 left you with a working compute pipeline: .metal source becomes a runnable kernel through the concatenated fast-math library, the strict-f32 cross-vendor library, or the pinned llama.cpp metallib — and a fingerprint line tells you which one actually ran. Ready to feed it work, we now face the problem that dwarfs every kernel decision: the weights have to fit, and they have to be read — every one of them, every token.

This chapter builds quantization from zero, with no allegiance to any particular format. We will invent a tiny 4-bit scheme, pack a block by hand, dequantize it with every multiply written out, and measure the error we just introduced. The real formats — kquant in Chapter 6, NVFP4 in Chapter 7 — are industrial versions of exactly this construction.

5.1 Why fp16 alone cannot carry a 30B model

Before we can argue about formats we have to know what we are carrying. Two questions decide everything that follows: how many weights are there, and how many bytes may each one cost? The first has an exact answer, and it is worth counting ourselves rather than trusting the nameplate on the box — every byte estimate in this chapter is built on that count, so an error here would quietly poison the whole argument.

A weight (also called a parameter) is one learned coefficient of the model. Muse Glimmer is nominally a 30B-class model; counting the tensors the loader actually validates gives a precise number. The engine’s config asserts every tensor’s shape at load [crates/muser-engine/src/config.rs:286], and the shapes are (Figure 5.1):

per layer (52 layers):
  attn_q          [hidden 6656 → 4096]     attn_gate  [6656 → 4096]
  attn_k, attn_v  [6656 → 256]             attn_output[4096 → 6656]
  ffn_gate, up    [6656 → 19968]           ffn_down   [19968 → 6656]
embedding table  [6656 → vocab 202,048]    lm_head    [6656 → 202,048]

Figure 5.1: The tensor inventory implied by assert_tensor_shapes (config.rs:294-318). Dimensions are GGUF ne order, [in_dim, out_dim].

Multiply it out (Figure 5.2 — we show the arithmetic so you can re-derive it):

attention per layer : 3 × (6656×4096) + 2 × (6656×256)      =  85,196,800
ffn per layer       : 3 × (6656×19968)                      = 398,721,024
per layer total     :                                         483,917,824
× 52 layers         :                                        25,163,726,848
embedding + lm_head : 2 × (6656×202,048)                     =  2,689,662,976
TOTAL               :                                        27,853,389,824  ≈ 27.85 B

Figure 5.2: Parameter count by hand. The “30B” nameplate is nominal; the counted total is ≈ 27.85 B. Norm vectors (~1.7 M values) are lost in the rounding here.

That count is the input to everything else in the chapter. Two walls stand between it and a working decode loop, and they fail in different ways: one asks whether the model fits, the other whether it can be read fast enough. Only the second one settles the format question, but the first is where most people’s intuition starts, so we take it first.

The capacity wall. f16 — a 16-bit IEEE float, the “half precision” format — is the smallest widely-used floating-point representation, at 2 bytes per weight:

27,853,389,824 × 2 B = 55,706,779,648 B ≈ 55.7 GB

The decode host is one Mac with an M3 Ultra and 96 GB of unified memory ([docs/memory-footprint.md] intro). Fifty-odd gigabytes against ninety-six looks like room to spare, and that is exactly the trap: the weights never occupy memory alone. Everything needed to serve one request has to live beside them — the KV cache, the per-token Key/Value attention memory of Chapter 1’s cost model (Ch 22 owns it in depth), which for the release configuration (four full-context slots at 131,072 tokens) is 7.306 GB; the DFlash draft artifact 1,631,205,312 B; the vision projector 1,400,328,928 B; and ~0.99 GB of f32 batch-activation widths for prefill [docs/memory-footprint.md].

That sum is already ≈ 67 GB, and it is a floor, not a total. memory-footprint.md is explicit about its own incompleteness — “summing artifact sizes with the KV formula is … only a lower bound” — because the operating system, Metal’s pipelines and workspaces, and per-slot sampler state all draw on the same 96 GB and none of them appear in the addition. An fp16 model is not a plan; it is a hope.

The bandwidth wall — the one that settles it. Chapter 1 established that decode is ~99% reading weights: every token’s forward pass streams the whole model through the GPU. There is no cache trick and no clever ordering that lets a decode step skip a weight: if it is in the model, it goes across the bus, once per token. Quantization is how Muser attacks that stream.

The pinned kquant artifact is 16,756,681,056 bytes. We hold that figure twice over — it is the size recorded in the artifact manifest [docs/memory-footprint.md], and the identical constant is pinned in the engine crate itself [crates/muser-engine/src/lib.rs:14]. Per weight, that is:

16,756,681,056 B × 8 bits/B ÷ 27,853,389,824 weights ≈ 4.81 bits/weight

Versus fp16’s 16 bits: the artifact is 3.33× lighter per token. If decode stayed bandwidth-bound at the same effective rate — the regime Chapter 1 proved — an fp16 model would run at roughly 35.4 ÷ 3.33 ≈ 10.6 tok/s at best, versus the measured kquant 35.440 tok/s [claims #11]. (Derived ceiling from the bytes ratio, not a measurement.) Capacity might survive an fp16 model on a lucky day. Bandwidth does not.

Quantization is the answer: store each weight in fewer than 16 (or 32) bits, accept a small, controlled error per weight, and buy a 3.3× reduction in the per-token byte stream. Everything else in Part II is the engineering of “small and controlled.”

5.2 Numbers as bits: f32 and f16

Fewer bits per weight, then. But which bits, and taken from where? To shrink a number we must first say what a number is in memory. An IEEE 754 float is a sign, an exponent, and a mantissa (fraction):

 f32 (32 bits):  [ sign:1 ][ exponent:8 ][ mantissa:23 ]   ~7 decimal digits
 f16 (16 bits):  [ sign:1 ][ exponent:5 ][ mantissa:10 ]   ~3 decimal digits

Figure 5.3: IEEE float layouts. The mantissa fixes the relative precision; the exponent fixes the dynamic range. f16 spans 2^-14 to 65504.

The mantissa is a binary fraction between 1 and 2 (for normal numbers), so an f16 weight with 10 mantissa bits is known to about one part in 1,024 — roughly three decimal digits. Real transformer weights live near zero, commonly within ±0.05, and f16 represents that range comfortably. What f16 cannot do is fit two of them in a byte. For that we leave floating point behind.

5.3 The codebook idea: a number as an index

Here is the single idea behind every format in this book. Instead of storing the weight’s value, store an integer index into a small table of allowed values — a codebook. Four bits select among 2⁴ = 16 possible values. Each index is a nibble — half a byte, values 0–15 — and two nibbles pack into one byte, which is where the storage win comes from: 0.5 bytes per weight.

Said the other way round: we stop recording what the weight is and start recording which of a short agreed list of values it sits nearest to. The list itself never goes over the bus, because sender and reader already know it. That substitution — value for index, memory for agreement — is the whole of quantization; everything after it is bookkeeping about how the list is chosen.

Two families of codebook exist, and Part II contains one of each:

• Integer (uniform) codebooks. The 16 values are evenly spaced: a base value plus index × step. Every kquant format (Chapter 6) is integer codebooks all the way down.
• Float codebooks. The 16 values are themselves tiny floats, spread with relative (multiplicative) spacing. NVFP4’s e2m1 (Chapter 7) is exactly a 16-entry float table.

Dequantization is the act of turning an index back into a value: look up the table (or compute base + index × step), and out comes an approximation of the original weight. Quantization is the inverse: pick the index whose value is closest to the original.

The gap between the original value and its reconstruction is the quantization error — the price of the whole enterprise. The rest of this chapter is about driving that error down without spending bytes.

5.4 One scale for everything: too crude

The first scheme everyone writes down is the one that needs no bookkeeping at all: one global step size for the whole tensor, chosen from the largest magnitude weight, shared by every weight in every row. Take it seriously for a moment, because on paper it looks unimprovable — a single stored number of overhead for hundreds of millions of weights, which rounds to no overhead at all.

So follow it through and see where it lands. Say a tensor’s weights span roughly ±0.5, so a symmetric 4-bit grid with 16 levels would use a step of 1.0/15 ≈ 0.067. Any weight is then replaced by the nearest multiple of 0.067. The arithmetic is sound; the assumption buried underneath it is not. Real weight blocks are much narrower than the global span — a few dozen neighboring weights typically cluster inside a band a tenth as wide. A global grid spends most of its 16 levels on values that never occur in that neighborhood, and the local error is needlessly large.

That is the lesson worth carrying out of the dead end: a codebook is worth only as much as its agreement with the values it is actually asked to encode. What you save on the header, you pay back with interest in error. We will watch the same failure play out concretely in §5.7 (the DC-offset problem), and it is why no format in this book uses a single global scale.

The fix is to quantize locally, not globally.

5.5 Blocks and scales

Split the tensor into small, contiguous blocks and give each block its own scale (a local step size). Now the 4 bits express a position inside the narrow band this block actually uses:

value ≈ scale × index            (symmetric: one number per block)
value ≈ scale × index + min      (asymmetric: two numbers per block)

A block of 32 weights might span only ±0.05, so its private scale is ~0.1/15 ≈ 0.0067 — ten times finer than the global grid above, from the same 4-bit index. Any slice of a smooth distribution spans less than the whole distribution; that is the entire trick, and it is the idea behind every “K-family” format in Chapter 6.

Turn it around once more, because this is the sentence the rest of Part II leans on: the narrower the range a scale has to cover, the more of your sixteen levels land where the data actually is. Bits do not become more precise by being more numerous. They become more precise by being asked a narrower question.

The scale itself is stored in floating point (typically f16, 2 bytes), because it must cover a wide range of magnitudes across blocks with only a few values of precision — the same division of labor as Figure 5.3, now between the scale (coarse, wide-range) and the index (fine, local).

5.6 Symmetric vs asymmetric: the min+offset trick

One stored number per block, or two? The choice looks like arithmetic and is really a question about what you are willing to assume about the data before you have looked at it.

Symmetric quantization assumes the block’s values are roughly centered on zero. One number per block — the scale — and the codebook spans −scale·max_index … +scale·max_index with 0 landing exactly on 0.

Asymmetric quantization adds a second number per block — the min (offset) — so the codebook can start wherever the data starts: `value = scale × index

• min`. This is the min+offset trick: it costs one extra stored number per block and buys correct handling of blocks that do not live around zero.

 Symmetric (1 number/block)          Asymmetric (2 numbers/block)
 ────────────────────────────        ────────────────────────────────
 grid:    −A ··· 0 ··· +A            grid:    min ··· min + 15·scale
 assumes: zero-centered              assumes: nothing
 index 0 → −A (or 0)                 index 0 → min        (offset!)
 waste:   shifted blocks lose        waste:   one extra f16 per block
          half their levels
 value = scale × index               value = scale × index + min

Figure 5.4: The two codebook geometries. Q4_0 and Q8_0 (Chapter 6) are symmetric; Q4_K and Q5_K are asymmetric; NVFP4’s float codebook is symmetric by construction (its table is ± pairs).

When does the difference bite? When a block has a DC offset — a mean far from zero. A block whose values all lie in, say, [0.30, 0.42] forces a symmetric grid to cover ±0.42, and every negative level is wasted: only about half the grid is ever addressed. The asymmetric grid puts min = 0.30 at index 0 and uses all 16 levels inside the 0.12-wide band — a resolution roughly 3.5× finer for the same 4 bits.

There is an honest gap in that argument, and it matters for what we are allowed to claim later. Whether weight blocks in a given checkpoint actually carry enough offset to make the trick pay is an empirical property of the checkpoint, and it is not one we measured for this one [unverified for Muse Glimmer]. What we can say is that the format designers thought it worth the bytes: the release artifact pays for asymmetric blocks on most tensors, which is Chapter 6’s byte-level story.

5.7 The worked example: an 8-element block, 4 bits, every step

Everything so far has been argument. Now we do the thing itself, slowly, with nothing hidden — because a reader who has packed one block by hand can read any format in this book, and a reader who has not will be taking the byte layouts ahead on faith. This is the heart of the chapter — the template every later quant chapter reuses. We quantize one block by hand. The numbers below are schematic, chosen so the arithmetic is clean; they are not taken from any checkpoint. Watch two things as they go past: where the error ends up landing, which is not where most people guess, and how few bytes the block costs when we are done.

The block. Eight weights:

x = [ -0.15,  0.29,  0.12, -0.03,  0.20,  0.06, -0.09,  0.17 ]

Step 1 — scan the block.

min = −0.15        max = 0.29        range = max − min = 0.44

Step 2 — fit the scale. 4 bits give 16 levels, indices 0–15. We want index 15 to land exactly on max:

scale = range / 15 = 0.44 / 15 = 0.029333…

For hand arithmetic, round the scale up to 0.03 — quantizers really do pick a convenient scale (Chapter 6’s scales are 6-bit integers times an f16; Chapter 7’s are powers-of-two floats), and a slightly padded range is safe: indices stay in 0–15 without clamping.

Step 3 — quantize each element. index_i = round((x_i − min) / scale):

x[0] = −0.15 → ( 0.00)/0.03 =  0.0 →   0
x[1] =  0.29 → ( 0.44)/0.03 = 14.67 → 15
x[2] =  0.12 → ( 0.27)/0.03 =  9.0  →  9
x[3] = −0.03 → ( 0.12)/0.03 =  4.0  →  4
x[4] =  0.20 → ( 0.35)/0.03 = 11.67 → 12
x[5] =  0.06 → ( 0.21)/0.03 =  7.0  →  7
x[6] = −0.09 → ( 0.06)/0.03 =  2.0  →  2
x[7] =  0.17 → ( 0.32)/0.03 = 10.67 → 11

Step 4 — pack. Eight indices, 4 bits each, two per byte, low nibble first (the convention Chapter 6 will meet in real code — q & 0x0F is the low element, q >> 4 the high one):

pairs : (0,15) (9,4) (12,7) (2,11)
bytes : 0x0F   0x49   0x7C   0x2B
block : [ min as f16 ][ scale as f16 ][ 0F 49 7C 2B ]
         2 bytes        2 bytes         4 bytes  → 8 bytes total

Step 5 — dequantize. x̂_i = min + scale × index_i. Every multiply shown:

x̂[0] = −0.15 + 0.03 ×  0 = −0.15            (exact)
x̂[1] = −0.15 + 0.03 × 15 = −0.15 + 0.45 = 0.30   (true 0.29)
x̂[2] = −0.15 + 0.03 ×  9 = −0.15 + 0.27 = 0.12   (exact)
x̂[3] = −0.15 + 0.03 ×  4 = −0.15 + 0.12 = −0.03  (exact)
x̂[4] = −0.15 + 0.03 × 12 = −0.15 + 0.36 = 0.21   (true 0.20)
x̂[5] = −0.15 + 0.03 ×  7 = −0.15 + 0.21 = 0.06   (exact)
x̂[6] = −0.15 + 0.03 ×  2 = −0.15 + 0.06 = −0.09  (exact)
x̂[7] = −0.15 + 0.03 × 11 = −0.15 + 0.33 = 0.18   (true 0.17)

Step 6 — measure the error. Per-element error x̂_i − x_i:

errors: 0, +0.01, 0, 0, +0.01, 0, 0, +0.01
max  abs error = 0.01
mean abs error = 0.00375

Three observations that generalize far beyond this toy:

1. The elements that defined the range are exact or nearly so. The block minimum reconstructed perfectly; the maximum was off by exactly one scale-step because we padded the scale. A quantizer’s error is worst for values in the middle of the range, never for the extremes that set it.
2. Error is bounded by scale/2. Every true value lies within half a step of a level, so |error| ≤ scale/2 by construction. Here 0.015; we measured ≤ 0.01.
3. Five of eight elements came out exact because they happened to sit on the grid. Real weights don’t sit on grids; real mean error lands near scale/4. The toy flatters us — keep that in mind when extrapolating.

Step 7 — the symmetric control. A worked example that only ever runs one way proves nothing, so before accepting the asymmetric header we send the same block through the other geometry and compare. Quantize it symmetrically: scale = amax/7, reading the 4 bits as signed indices −7…+7, amax = 0.29, so scale = 0.29/7 ≈ 0.0414 — coarser than 0.03 even before accounting for the wasted negative levels this almost-centered block barely uses. The extra stored number has already earned its keep — and it earned it on a block that was nearly centered to begin with, which is the weakest case we could have handed it. The offset here is mild (min = −0.15, max = 0.29); for a strongly one-signed block the symmetric penalty is the full factor-of-two of Figure 5.4.

5.8 What the error costs downstream

A hundredth here and a hundredth there — does any of it survive contact with the model, or does it wash out? The question matters because the answer decides how nervous to be, and the honest answer is: it does not wash out, but it also does not behave like noise.

A weight’s quantization error is not an isolated blemish — it is a deterministic perturbation of the model. Each dot product in the forward pass — the multiply-and-add pairing of two equal-length vectors, the atom under every weight matrix in this book — mixes in one error term per weight. Through 52 layers the perturbations compound multiplicatively (the drift argument Chapter 12 makes for normalization). Three honest statements about the cost:

• The error is fixed and knowable. Weights are quantized once, offline; the dequantized value is the same every token. It is a permanent, exact bias — not noise. That is what makes cross-engine parity possible at all: Muser’s kquant lane reproduces llama.cpp’s numbers bit-for-bit on the shared format precisely because the bytes and the arithmetic order are pinned [crates/muser-engine/src/quant/k_block.rs:169-177].
• Quality loss is real, but it is bounded by measurements rather than by vibes. Muser does not assert that quantization is harmless; it budgets the harm and then checks the budget. NVFP4-versus-kquant relative perplexity and top-token disagreement are gated per depth and per content class, and where one content class came in above its calibrated gate, the number was published rather than buried: docs text at 65,536 tokens, 15.134% against a 13.339% calibrated gate, carried as part of the claim [claims #10]. Chapter 7 tells that story.
• You cannot compare formats by one number. The same 4-bit-class quantization is invisible in plain decode — the lanes land inside each other’s noise, as §5.9 will show — and decisive the moment the shape of the work changes: in batched speculative verification the same bytes produce a 6.81 tok/s no-go, the story Chapter 7 tells. Precision, in other words, does not have a price. It has a price per batch shape and per content class, which is why the gates are written to localize the cost instead of returning a single verdict.

5.9 Block size: the memory-vs-overhead dial

How big should a block be? It sounds like a tuning knob and is really a fork in the road, because the two ends of the dial fail for opposite reasons — and the formats waiting in the next two chapters are best read as two different answers to this one question.

The block header (min + scale, say 4 bytes as two f16s) is paid once per block. The bitrate — bits per weight — is:

bitrate = payload bits + header bits / block size
        = 4            + 32 / N

Turn the dial (Figure 5.5):

 block size N     header/weight     total bits/weight     local range
 ─────────────    ─────────────     ─────────────────     ───────────────
      8              4.000               8.00            very narrow
     32              1.000               5.00            narrow
    256              0.125               4.125            moderate
   1024              0.031               4.03             wide

Figure 5.5: The block-size dial. Smaller blocks buy finer local scales and lower error; bigger blocks amortize the header. (This table assumes the naive one-scale-one-min-per-block header of §5.7.)

Both directions fail. At N = 1,024 the header is nearly free, but a block that wide spans much of the tensor’s dynamic range and the local-scale advantage evaporates — you drift back toward the global grid of §5.4. At N = 8 the error is superb and you have doubled the storage. Every real format parks somewhere in between and then engineers the header down:

• Q4_K (Chapter 6): N = 256 with an asymmetric header, made cheap by a two-level hierarchy — one f16 super-scale and one f16 super-min for the whole 256, plus six-bit sub-scales per 32-element sub-block. Total header: 16 bytes per 256 weights = 0.5 bits/weight → 4.5 bits/weight.
• NVFP4 (Chapter 7): N = 16 with a symmetric float header of a single one-byte e4m3fn scale, plus one f32 per tensor. Total: 4 + 8/16 = 4.5 bits/weight — the same bitrate as Q4_K by entirely different means.

That coincidence is worth pausing on: two formats, two codebook families, two block sizes — and the same 4.5 bits. Format design is the art of spending a fixed half-bit of overhead (on top of the 4 payload bits) in different places. And 4.5-ish bits is also where the real artifact lands on average: the whole-GGUF figure computed in §5.1 was 4.81 bits/weight (the excess over 4.5 is the deliberately more precise tensors Chapter 6 identifies — Q6_K’s 6.5625 and Q5_K’s 5.5).

One more axis the table hides: who pays to use the format. A tiny block with a cheap codebook dequantizes with one multiply (good for a hot kernel); a 256-block with 6-bit bit-packed sub-scales costs real decode work per block (Chapter 6 shows the kernel-side machinery). Block size is a deal between storage, error, and kernel complexity — not just a storage number.

5.10 Tradeoffs

Asymmetric vs symmetric, measured in bytes. What does honesty about offsets actually cost, in the only currency decode cares about? The min+offset trick costs one extra stored number per block. At N = 32 with f16 headers that is 4 + 64/32 = 6 bits/weight symmetric vs 4 + 96/32 = 7 bits/weight asymmetric — a 17% storage tax for offset robustness. Q4_K’s two-level hierarchy is precisely the invention that recovers the tax: min+offset at 4.5 bits/weight, the same bitrate a plain symmetric 32-block would waste (4 + 16/32 = 4.5). Chapter 6 walks the real bytes.

4 bits vs 16, measured in tokens. Does the shrinking actually buy tokens, or does the decode side hand the savings straight back as unpacking work? The lane throughputs answer it: kquant (≈4.81 bits/weight average) 35.440 tok/s and native NVFP4 (4.5 bits/weight) 35.491 tok/s, measured in the same paired five-rep cell — parity within noise, never claimed faster [claims #11]. The measured existence of two independent ~4.5-bit artifacts running at parity with the f16-KV llama.cpp comparator is the strongest statement this book can make that quantization, done at this rate, does not tax decode throughput.

The scope of that cell matters as much as the figures do, so we carry it rather than round it away: a 66-token prefix, 32 teacher-forced tokens, F16 KV, an adjacent lease window, and a +0.1444% edge for NVFP4 — a margin no one should read as a win, and the reason the claim says parity and stops there. What 4 bits does tax — the batched speculative verify path — is a Chapter 7 measurement.

Why not 2 bits? The dial in the previous section has a lower end too, and after a chapter spent shrinking things the fair question is why we stopped where we stopped. Nothing in this chapter’s arithmetic forbids going further — 2-bit codebooks exist in the wild. But at 4 levels per block the quantization error approaches the size of the local scale itself, and §5.8’s error-compounding has nowhere to hide: the error stops being a correction to the weight and starts being most of the weight. Muser’s own gates localized quality cost at 4-bit-class formats to specific content classes [claims #10], and even that took a calibrated per-content gate to see at all. So the ending here is an admission rather than a verdict: we do not know what the next step down would have cost on this model, because no 2-bit lane was ever qualified in this program [unverified — no measurement exists in the retained evidence].

5.11 What comes next

You now own the complete template: a codebook, a local scale, an optional min, a hand-packed block, a dequant with every multiply visible, and an error budget. Chapter 6 deploys it on the bytes Muser actually ships — the kquant family (Q4_K, Q5_K, Q6_K) that fills the 16,756,681,056-byte reference artifact, byte layouts first, then the pinned kernel_mul_mv_q*_K_f32 Metal kernels that consume them and the per-tensor map of which class of weight gets which format.

References

• [crates/muser-engine/src/config.rs:286] — assert_tensor_shapes: every tensor and shape the loader validates (Figure 5.1’s source).
• [crates/muser-engine/src/lib.rs:14] — the pinned artifact byte size 16,756,681,056 asserted in the crate docs.
• [docs/memory-footprint.md] — 96 GB M3 Ultra host, KV formula and the 7.306 GB four-slot figure, artifact manifest (16,756,681,056 / 1,631,205,312 / 1,400,328,928 B), “lower bound” caution.
• [claims #11] — docs/launch-claims.md row 11: plain Mac NVFP4 35.491 tok/s vs adjacent kquant 35.440 tok/s at original scope, parity within noise; also the five-rep cell description via the ledger (P1.3).
• [claims #10] — native NVFP4 quality gates and the published docs@65,536 content-local sensitivity (15.134% vs 13.339%).
• [crates/muser-engine/src/quant/k_block.rs:169-177] — dot_q4_k_f32_llama doc: the pinned llama.cpp accumulation-order contract for Q4_K.
• Ch 6 — the kquant family: real byte layouts, the 6-bit scale packing, and the dispatch table.
• Ch 7 — NVFP4: the float codebook and the native lane.
• [ferrite-book Ch 5] — the ancestor’s Q4_K chapter, whose hand-built-superblock method this chapter ports (pedagogical lineage only; all numbers here are re-derived from the Muser tree).

Chapter 6 — The kquant family on the reference lane

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Chapter 5. You have hand-packed an 8-element 4-bit block, dequantized it with every multiply shown, and measured the error. This chapter runs the same procedure on the real bytes Muser ships — no more schematic numbers. Two Metal kernels get quoted; the rest is byte layout and arithmetic.

Chapter 5 ended on a promise: the concrete formats. Here is the first one. The kquant family — llama.cpp’s “K-quant” block formats — is what fills the 16,756,681,056-byte reference artifact that Muser’s kquant lane decodes from, the lane measured at 35.440 tok/s [claims #11] and used as the program’s reference lock.

Everything that follows answers one question in different disguises: when a weight has been squeezed down to a handful of bits inside a packed block, what exactly must the machine read, and in what order, to get a usable number back out? So we read the byte layouts straight from Muser’s own dequantizers, dequantize real-format elements by hand, map which tensor class carries which member of the family, and then watch the dispatch code choose a kernel for each batch shape. The receipts are all here; they simply arrive after the idea they support rather than in the middle of it.

6.1 One model, several dtypes — decided by the GGUF

Before any bytes, a question of authority: who decides that this tensor is Q4_K and that one is Q6_K? Not the engine, and not an operator flag at startup. The file decides, and it decides tensor by tensor. The stake is higher than it sounds — guess wrong here and you do not get a slightly worse model, you get an engine that refuses to start, because every admission check below is fail-closed.

A GGUF file carries a dtype per tensor, not per model. Muser’s parser enumerates the types it is willing to meet:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/gguf/types.rs:9
pub enum GgmlType {
    F32 = 0,
    F16 = 1,
    Q4_0 = 2,
    Q4_1 = 3,
    Q5_0 = 6,
    Q5_1 = 7,
    Q8_0 = 8,
    // …
    Q4_K = 12,
    Q5_K = 13,
    Q6_K = 14,
    // …
    /// Muser-native NVFP4 E2M1 payload, two logical values per byte.
    /// Per-16 E4M3FN scales and the per-tensor f32 scale2 live in bound
    /// companion tensors; this keeps the serving representation at exactly
    /// 4.5 bits/weight without relying on llama.cpp's experimental format.
    NVFP4_E2M1 = 1000,
    /// Raw E4M3FN bytes used only by NVFP4 companion scale tensors.
    F8_E4M3FN = 1001,
}
}

Every type knows its geometry — bytes per block and elements per block — and those two numbers define the format’s bitrate [crates/muser-engine/src/gguf/types.rs:75-127]:

 format   block bytes   elements   bits/element   min+offset?   codebook
 ───────  ───────────   ────────   ────────────   ───────────   ────────
 Q4_0        18            32         4.50         no (sym)      4-bit int
 Q8_0        34            32         8.50         no (sym)      8-bit int
 Q4_K       144           256         4.50         yes           4-bit int
 Q5_K       176           256         5.50         yes           5-bit int
 Q6_K       210           256         6.5625       no (signed)   6-bit int
 F16          2             1        16.0          exact         float

Figure 6.1: The kquant family as registered in GgmlType::block_size / block_elements. Bits/element = block_bytes × 8 ÷ elements — derive each one yourself: Q4_K is 144×8/256 = 4.5; Q6_K is 210×8/256 = 6.5625.

Two paths read these types, and they are not equally permissive. The CPU reference path will dequant anything on the list — quant/dispatch.rs simply fans out per dtype. The live Metal decode path is deliberately narrower. A projection tensor must be one of Q4_K | Q5_K | Q6_K | NVFP4_E2M1 | F16, and the embedding table is narrower still: Q4_K or F16, nothing else. Anything outside those sets is not quietly converted on the way in; it is refused. The two fences are drawn at [crates/muser-engine/src/decode.rs:136-139] for projections and [crates/muser-engine/src/decode.rs:1209] for the embedding.

The lane itself is chosen the same fail-closed way at load. The GGUF must declare muser.weight_precision, and a kquant artifact becomes the default only when the file contains no native NVFP4 tensors at all [crates/muser-engine/src/loader.rs:72-91]. The artifact picks the lane — not a flag, not a heuristic, not a fallback that guesses. Chapter 7 covers the nvfp4 pairing; this chapter stays on q4_k_xl.

6.2 Q4_K: the 144-byte super-block

Q4_K is the format most of this artifact is written in, which makes it the one worth knowing to the byte. It answers the question the previous chapter left open: how do you give every small group of weights its own local scale without the scales themselves eating the savings you went to all this trouble for? Watch where each header byte goes, and the answer is arithmetic rather than magic.

The unit of Q4_K storage is a super-block: 256 weights in 144 bytes, which is the 4.5 bits/weight of Figure 6.1. As in Chapter 5, we read the layout off the dequantizer — this is Muser’s complete, real function:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/quant/k_block.rs:12
pub fn dequant_q4_k(block: &[u8], out: &mut [f32]) {
    debug_assert!(block.len() >= 144);
    debug_assert!(out.len() >= 256);

    let d = f16_to_f32(u16::from_le_bytes([block[0], block[1]]));
    let dmin = f16_to_f32(u16::from_le_bytes([block[2], block[3]]));
    let scales = &block[4..16];
    let qs = &block[16..144];

    // get_scale_min_k4: extract 6-bit sc and m for sub-block j (0..7)
    let get_scale_min = |j: usize| -> (f32, f32) {
        let (sc, m) = if j < 4 {
            (scales[j] & 0x3F, scales[j + 4] & 0x3F)
        } else {
            let sc = (scales[j + 4] & 0x0F) | ((scales[j - 4] >> 6) << 4);
            let m = (scales[j + 4] >> 4) | ((scales[j] >> 6) << 4);
            (sc, m)
        };
        (d * sc as f32, dmin * m as f32)
    };

    // 4 outer groups of 64 elements, 2 sub-blocks per group, 32 qs bytes per group.
    let mut q_off = 0usize;
    let mut is = 0usize;
    let mut base = 0usize;
    while base < 256 {
        let (d1, m1) = get_scale_min(is);
        let (d2, m2) = get_scale_min(is + 1);
        for l in 0..32 {
            let q = qs[q_off + l];
            out[base + l] = d1 * (q & 0x0F) as f32 - m1;
            out[base + l + 32] = d2 * (q >> 4) as f32 - m2;
        }
        q_off += 32;
        is += 2;
        base += 64;
    }
}
}

The byte map (Figure 6.2), in the ASCII style Chapter 5 promised:

 Q4_K super-block — 144 bytes for 256 elements

 offset   size   field      meaning
 ─────────────────────────────────────────────────────────────────
  0x00     2     d          f16  super-scale        (shared by 8 sub-scales)
  0x02     2     dmin       f16  super-min-scale    (shared by 8 sub-mins)
  0x04    12     scales     8× 6-bit sc + 8× 6-bit m, packed (Fig 6.3)
  0x10   128     qs         256 nibbles (2 per byte) — the weights
  0x90     —    (end)      2+2+12+128 = 144 bytes
 ─────────────────────────────────────────────────────────────────
  0x10 = 16,  0x90 = 144

Figure 6.2: The Q4_K super-block. Four regions: two f16 headers, a 12-byte packed scale/min strip, 128 bytes of nibbles. Compare the 8-byte toy block of Chapter 5 — same idea, one more level of hierarchy.

The structure is Chapter 5’s min+offset scheme with the two-level header that keeps it at 4.5 bits: the 256 elements split into eight sub-blocks of 32, each with its own 6-bit scale sc (0–63) and 6-bit min m (0–63), all multiplied by the shared f16 d and dmin. The value of element i in sub-block j is:

y = d × sc_j × nibble  −  dmin × m_j

Exactly scale × index + min from Chapter 5 — as d·sc_j (effective scale) and −dmin·m_j (effective min) with a sign flip folded in. The header budget: 4 bytes of f16s + 12 bytes of packed 6-bit fields = 16 bytes per 256 weights = 0.5 bits/weight of overhead. Chapter 5’s dial table charged 0.125 bits for a naive one-f16-pair header per 256; Q4_K spends 0.5 — and buys eight independent local scales/mins instead of one.

The 96 bits of scale/min strip are packed with zero padding. Sub-blocks 0–3 live wholly in bytes 0–7; sub-blocks 4–7 split their values between the low/high halves of bytes 8–11 and the top two bits of bytes 0–7:

 byte          b7 b6 │ b5 b4 b3 b2 b1 b0      contents
 ──────────────────────────────────────────────────────────────────────
 scales[0..4]   sc4+ │ sc0 (6 bits)            top2 of scales[j] = sc_{j+4} hi
 scales[4..8]   m4+  │ m0  (6 bits)            top2 of scales[4+j] = m_{j+4} hi
 scales[8..12]  m_lo │ sc_lo                   lo4|hi4 reassembles sc/m of 4..7

Figure 6.3: The 6-bit packing, compressed. The get_scale_min closure above is the authoritative spec: sub-blocks ≥4 reassemble sc from the low nibble of scales[j+4] plus the top 2 bits of scales[j-4], and m from the high nibble plus the top 2 bits of scales[j].

Note also the interleaving in the dequant loop: within each 64-element group, the low nibble of byte qs[q_off+l] feeds element base+l (even sub-block) and the high nibble feeds element base+l+32 (odd sub-block). Said the other way round, because this is the part that trips people up: a byte of qs does not belong to any single sub-block. Its low half is a weight in one sub-block and its high half is a weight in the neighbouring one, and the two halves are reconstructed with different effective scales and different mins. One byte, two sub-blocks, two different scales — the property Chapter 5’s toy deliberately did not have.

6.3 A worked dequant of real-format bytes

Reading a layout table is not the same as believing it. The only way to be sure you have the packing right is to build a block out of nothing, run the extraction the shipping code runs, and watch the values come back. That is the exercise here, and it is worth doing with a pencil rather than skimming.

Hand-build one super-block (values schematic in magnitude — real weights dequant near ±0.05; the mechanics are identical):

Headers. Pick d = 0.03125 = 2⁻⁵ and dmin = 0.0078125 = 2⁻⁷, both exact in f16. f16 bits: exponent = power+15, mantissa 0 → d = 0x2800, dmin = 0x2000; little-endian bytes 00 28 and 00 20.

Sub-block scales. Choose (all 0–63):

 sc = [20, 12, 44,  8, 33, 25, 17, 9]      m  = [ 4, 16,  8, 24, 12, 20, 28, 36]

Pack per Figure 6.3 — each line is the formula with the arithmetic shown:

 scales[0]  = sc0 | ((sc4 >> 4) << 6) = 20 | 128 = 148 = 0x94
 scales[1]  = sc1 | ((sc5 >> 4) << 6) = 12 |  64 =  76 = 0x4C
 scales[2]  = sc2 | ((sc6 >> 4) << 6) = 44 |  64 = 108 = 0x6C
 scales[3]  = sc3 | ((sc7 >> 4) << 6) =  8 |   0 =   8 = 0x08
 scales[4]  = m0  | ((m4  >> 4) << 6) =  4 |   0 =   4 = 0x04
 scales[5]  = m1  | ((m5  >> 4) << 6) = 16 |  64 =  80 = 0x50
 scales[6]  = m2  | ((m6  >> 4) << 6) =  8 |  64 =  72 = 0x48
 scales[7]  = m3  | ((m7  >> 4) << 6) = 24 | 128 = 152 = 0x98
 scales[8]  = (sc4 & 0x0F) | ((m4 & 0x0F) << 4) = 1 | 192 = 193 = 0xC1
 scales[9]  = (sc5 & 0x0F) | ((m5 & 0x0F) << 4) = 9 |  64 =  73 = 0x49
 scales[10] = (sc6 & 0x0F) | ((m6 & 0x0F) << 4) = 1 | 192 = 193 = 0xC1
 scales[11] = (sc7 & 0x0F) | ((m7 & 0x0F) << 4) = 9 |  64 =  73 = 0x49

Round-trip check for sub-block 5, substituting into get_scale_min exactly as the code does:

 sc5 = (scales[9] & 0x0F) | ((scales[1] >> 6) << 4) = 9 | (1 << 4) = 25  ✓
 m5  = (scales[9] >> 4)    | ((scales[5] >> 6) << 4) = 4 | (1 << 4) = 20  ✓

Nibbles. Set qs[0] = 0x5E (offset 16) and qs[32] = 0x3C (offset 48); all other qs bytes zero.

Dequantize three elements. Precompute the effective scale/min per sub-block (d·sc, dmin·m):

Element 0   (sub-block 0, low nibble of qs[0]):
 q = 0x5E → nibble = q & 0x0F = 14
 y0  = 0.625 × 14 − 0.03125 = 8.75 − 0.03125 = 8.71875

Element 32  (sub-block 1, high nibble of the SAME byte):
 q = 0x5E → nibble = q >> 4 = 5
 y32 = 0.375 × 5 − 0.125     = 1.875 − 0.125  = 1.75

Element 64  (sub-block 2, low nibble of qs[32]):
 q = 0x3C → nibble = q & 0x0F = 12
 y64 = 1.375 × 12 − 0.0625   = 16.5 − 0.0625  = 16.4375

One byte, two sub-blocks, two different effective scales — Chapter 5’s “zoom into the local band,” now twice per byte. You have dequantized Q4_K by hand against the shipping code.

6.4 Q5_K and Q6_K: the siblings

Q4_K has two siblings on this artifact, and each one changes exactly one thing. Both are answering the same question — where do you put the bits that did not fit in a nibble? — and they answer it differently: Q5_K adds a plane, Q6_K changes the codebook. Keep that distinction in hand; it is what makes their layouts feel different when they are really not.

Q5_K is Q4_K plus one extra bit per element, stored in a separate plane. Its 176 bytes (Figure 6.4): the same d/dmin/12-byte scale strip, then 32 bytes qh holding one high bit per element (256 bits), then the same 128 bytes of low nibbles [crates/muser-engine/src/quant/k_block.rs:51-60]:

 Q5_K — 176 bytes / 256 elements      y = d·sc·(nibble | qh_bit<<4) − dmin·m
 ────────────────────────────────────────────────────────────────────────
  0x00   2   d          0x02   2   dmin       0x04  12   scales
  0x10  32   qh         0x30 128   qs         0xB0   —  (end = 176)

Figure 6.4: Q5_K layout (offsets hex: 0x30 = 48, 0xB0 = 176). The 5-bit codebook lifts the index range from 0–15 to 0–31; 176×8/256 = 5.5 bits/element ([crates/muser-engine/src/quant/k_block.rs:61-105] for the dequant).

Q6_K changes shape more: a signed 6-bit codebook split across two planes, with the per-sub-block scales as plain int8 and the single f16 super-scale at the end of the block (Figure 6.5):

 Q6_K — 210 bytes / 256 elements      y = d · sc_j · q ,  q = 6-bit signed
 ────────────────────────────────────────────────────────────────────────
  0x00 128   ql   low 4 bits of each code, packed 2/byte
  0x80  64   qh   high 2 bits of four codes per byte
  0xC0  16   sc   16 × int8 sub-block scales (sub-blocks of 16 elements)
  0xD0   2   d    f16 super-scale (LAST field — bytes 208..210)

Figure 6.5: Q6_K layout (0x80 = 128, 0xC0 = 192, 0xD0 = 208). The Muser kernel reads these offsets verbatim: ql = &block[0..128]; qh = &block[128..192]; sc = &block[192..208]; d = f16(block[208..210]) [crates/muser-engine/src/quant/k_block/q6.rs:21-26].

Each of the 16 sub-blocks (16 elements each) reconstructs its codes as ((ql_nibble) | (qh_2bits << 4)) − 32 — an unsigned 0–63 payload shifted to signed −32…+31 — then scales once: 210×8/256 = 6.5625 bits/element. No min term: a signed symmetric grid, but with 64 levels the zero-waste argument of Chapter 5 matters less. The code-extraction quadruple (q1..q4 pairing ql low/high nibbles with two-bit qh fields) is at [crates/muser-engine/src/quant/k_block/q6.rs:45-52], in the llama-pinned deferred-scaling order (§6.7).

6.5 Which tensor carries which format

So the family has members. Which tensor gets which, and who is keeping score? This is where a quantization recipe stops being theory: a few tensors get promoted to more bits, most do not, and every promotion is paid for out of the per-token byte budget this book opened with.

The release artifact is a mix, like llama.cpp’s “Q4_K_M” recipes. The authoritative in-repo map is the shape table of the M=16 microbenchmark harness, which enumerates the exact verify/draft projection shapes with their dtypes and how often each fires per speculative cycle ([crates/muser-bench/src/m16.rs:137-226]):

 label                    dtype   shape (n_in→n_out)   per-cycle mult
 ───────────────────────  ─────   ──────────────────   ──────────────
 attn_q/gate              Q4_K    6656→4096            104  (= 2 × 52 layers)
 attn_k/v                 Q4_K    6656→256              78
 attn_v                   Q6_K    6656→256              26
 attn_output              Q4_K    4096→6656             52
 ffn_gate/up              Q4_K    6656→19968           104
 ffn_down                 Q4_K    19968→6656            26
 ffn_down                 Q6_K    19968→6656            26
 lm_head                  Q5_K    6656→202048            1
 draft.k                  Q4_K    6656→1024              5  (draft layers)
 draft.v                  Q6_K    6656→1024              5
 draft.fc                 Q4_K    33280→6656             1

Figure 6.6: Dtypes and shapes on the release path, from the M16 bench SHAPES table. The multiplicities are the harness’s own counts of dispatches per verify cycle / draft block.

Read the mix off the counts. Attention k/v projections: 78 + 26 = 104 tensors = two per layer, so 26 of the 104 k/v tensors are Q6_K and the rest Q4_K. FFN down-projections: 26 + 26 = 52, so ffn_down alternates Q4_K/Q6_K by layer. Everything bandwidth-dominant — q, gate, output, ffn_gate/up — is Q4_K; the FFN gate/up tensors are pinned Q4_K by the fused kernel’s own guard ([crates/muser-engine/src/decode.rs:5819-5821]). The lm_head is Q5_K — one tensor, but a 6656×202,048 one, worth 924,571,648 B by the arithmetic: 26 blocks/row × 176 B × 202,048 rows. The embedding table rides Q4_K on this artifact through the dedicated muser_embedding_q4k kernel (§6.7). This is Chapter 5’s bitrate analysis made flesh: bulk at 4.5 bits, promoted tensors at 5.5/6.5625, averaging the whole-artifact 4.81 bits/weight computed in Chapter 5.

The draft rows (draft.*) preview Ch 8: the DFlash assistant is itself kquant — its loader requires Q4_K/Q5_K/Q6_K ([crates/muser-engine/src/dflash/weights.rs:148-158]).

6.6 The bytes-per-token tie-back to Chapter 1

Chapter 1’s whole thesis was “one token ≈ stream the model.” Now you can compute that stream precisely for one projection. A matvec — a matrix-by-vector multiply, one dot product of the weight row against the input vector per output element; the shape every projection takes when exactly one token is in flight (Ch 13 derives it from zero) — reads n_in / 256 super-blocks per row; each row of the q projection (n_in = 6656) is:

26 blocks × 144 B = 3,744 B per row   × 4,096 rows = 15,335,424 B per q matrix

Do the same for every tensor class in Figure 6.6 and sum: you converge on the artifact’s own 16,756,681,056 B (≈ 16.76 GB decimal, ≈ 15.6 GiB — this book follows docs/memory-footprint.md in using decimal GB). That file is Chapter 1’s per-token weight read, and its size is not an accident: it is 27.9 B parameters at the mixed 4.5/5.5/6.5625-bit rates of Figure 6.6. The block layout is row-major — each output row is a contiguous run of super-blocks — which is exactly why a matvec kernel can stream it (§6.7) and why prefill of T tokens costs roughly the same DRAM traffic as one token ([crates/muser-engine/src/weights.rs:4-7]).

6.7 The kernels that eat these bytes

A format is only worth what the code reading it is worth, so the next question is: whose kernel actually touches these bytes? The answer is not “ours,” and that turns out to be a deliberate choice with a numerical argument behind it rather than a shortcut.

Muser deliberately runs kquant matmuls through three sources (recall Chapter 4). For a single-token decode projection (encode_quantized_matmul, tokens = 1), the first choice is the pinned llama.cpp metallib:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:429
if tokens == 1 {
    if let Some(pipeline) = self.ggml_matvec(dtype) {
        let (block_bytes, rows_per_group) = match dtype {
            GgmlType::Q4_K => (144, 2),
            GgmlType::Q5_K => (176, 1),
            GgmlType::Q6_K => (210, 2),
            _ => unreachable!("ggml_matvec returned only for K-quant projections"),
        };
        // …
}

ggml_matvec resolves to kernel_mul_mv_q4_K_f32, kernel_mul_mv_q5_K_f32, or kernel_mul_mv_q6_K_f32 from the metallib loaded via MUSER_GGML_METALLIB ([crates/muser-engine/src/metal/encode.rs:278-280]). The launch geometry is named in prose as the contract requires: grid = n_out ÷ (rows_per_group × 2) threadgroups, threadgroup size (32, 2) — 64 threads, two SIMD groups (Chapter 2’s 32-lane hardware execution unit), Q4_K/Q6_K computing two rows per group and Q5_K one ([crates/muser-engine/src/metal/encode/qkv.rs:444-448]).

Now look at what the unreachable! arm quietly admits. Without the metallib there is no Muser-authored single-token Q6_K matvec. Only Q4_K and Q5_K have hand-written siblings — muser_matvec_q4k_4r2s and muser_matvec_q5k_4sg — so when the pinned metallib is absent the fallback switch does not degrade gracefully on Q6_K, it panics [crates/muser-engine/src/metal/encode/qkv.rs:451-459]. That is the fail-closed reflex again, and it is the right reflex: a wrong logit is worse than no logit.

When the ferrite-lineage fallback does run, here is the Q4_K kernel’s core — read it against §6.3:

// crates/muser-engine/src/shaders/muse_reference.metal:735
kernel void muser_matvec_q4k_4r2s(
    device const uchar *weights [[buffer(0)]],
    device const float *input [[buffer(1)]],
    device float *output [[buffer(2)]],
    constant uint &rows [[buffer(3)]],
    constant uint &cols [[buffer(4)]],
    uint group [[threadgroup_position_in_grid]],
    uint lane [[thread_index_in_simdgroup]],
    uint simd [[simdgroup_index_in_threadgroup]]) {
    uint block_count = cols / 256;
    uint row_bytes = block_count * 144;
    uint base_row = group * 8 + simd * 4;
    // …
    for (uint block_index = 0; block_index < block_count; ++block_index) {
        for (uint row_index = 0; row_index < active_rows; ++row_index) {
            device const uchar *block = row[row_index] + block_index * 144;
            uint delta = *reinterpret_cast<device const uint *>(block);
            float d = float(as_type<half>(ushort(delta & 0xffff)));
            float dmin = float(as_type<half>(ushort(delta >> 16)));
            uint sd0 = *reinterpret_cast<device const uint *>(block + 4);
            uint sd1 = *reinterpret_cast<device const uint *>(block + 8);
            uint sd2 = *reinterpret_cast<device const uint *>(block + 12);
            float d_scale[8];
            float neg_min[8];
            muser_decode_all_q4k_scales(d, dmin, sd0, sd1, sd2, d_scale, neg_min);
            uint input_base = block_index * 256;
            for (uint quant_group = 0; quant_group < 4; ++quant_group) {
                uint packed = uint(block[16 + quant_group * 32 + lane]);
                float low = input[input_base + quant_group * 64 + lane];
                float high = input[input_base + quant_group * 64 + 32 + lane];
                accumulator[row_index] +=
                    fma(d_scale[quant_group * 2], float(packed & 0x0f), neg_min[quant_group * 2]) * low;
                accumulator[row_index] +=
                    fma(d_scale[quant_group * 2 + 1], float(packed >> 4), neg_min[quant_group * 2 + 1]) * high;
            }
        }
    }

(Lines elided: the row-pointer setup and the final simd_sum reduction — see the file.) Every offset you computed by hand is there: bytes 0–3 read as one uint and split into the two f16s, bytes 4–15 as three uints feeding the 8-way scale decode (muser_decode_all_q4k_scales, muse_reference.metal:41), nibbles from byte 16 onward with the low/high sub-block pairing of §6.3. The lane geometry (one lane per byte, 32 lanes covering one 64-element group per iteration) is Part IV material — Ch 13 walks it properly.

What a matvec must read per block, then: 4 bytes of headers, 12 bytes of packed scales, and 128 bytes of nibbles, per 144-byte super-block, per row — plus the activation vector it is dotted against. Nothing else exists to read; the block is entirely self-describing. That is the access pattern in one sentence, and it is why the bytes-per-token accounting of §6.6 is exact.

Batch shapes get their own kernels, all selected in encode_quantized_matmul — the dispatch ladder of Figure 6.7 ([crates/muser-engine/src/metal/encode/qkv.rs:414-641]):

 token count   route (kquant)                                    source
 ───────────   ──────────────────────────────────────────────   ──────────
 1             kernel_mul_mv_q{4,5,6}_K_f32                      llama metallib
 2–3           same kernel, one launch per token                 llama metallib
 4–8           mul_mv_ext family (rows-per-TG 2/3/4/5)           llama metallib
 16            m16_q{4,5,6}k_n32 weight-stationary tile          muser (ferrite)
 Q4_K, ≥16,    matmul_q4k_batch_sgm_aligned                      muser (ferrite)
 aligned
 any (else)    kernel_mul_mm_q{4,5,6}_K_f32 aligned/bounds       llama metallib

Figure 6.7: The kquant dispatch ladder. MUSER_CROSS_VENDOR_QK=1 swaps any rung for the strict-f32 muser_cross_vendor_q* kernels (Chapter 4’s second source), and MUSER_MULTI_COL_VERIFY gates an exact multi-column verify route ([crates/muser-engine/src/metal/encode/multicol.rs:12-14]).

Why keep llama’s batch boundaries at all? Look at the ladder and an obvious simplification suggests itself: the middle rungs could just call the single-token kernel repeatedly, once per activation row, and a whole family of kernels would disappear from the engine. It is the kind of cleanup that looks free. The comment in the dispatch exists to close that door, and it is the chapter’s most important citation:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:476
// Match the source-pinned llama.cpp Metal dispatch boundary exactly:
// K-quant projections with four through eight activation rows use
// `mul_mv_ext`, with a token-count-specific number of rows per
// threadgroup. This changes the floating-point reduction order, so
// substituting repeated decode GEMVs here breaks embedding/logprob
// numerical parity even when every other layer is identical.
}

That is worth restating slowly, because it is the second genuinely hard idea in this chapter. Floating-point addition is not associative: add the same products in a different order and you get a different sum, usually differing only in the last bits. A different kernel means a different reduction order, a different order means different last bits, and different last bits mean a logprob that no longer matches the comparator’s. So these batch boundaries are not a performance-tuning quirk inherited by accident. They are part of the numerical contract. Match the comparator’s kernels and you inherit its arithmetic; improvise, and parity is gone before anyone has asked a speed question.

The embedding kernel completes the single-token picture. muser_embedding_q4k ([crates/muser-engine/src/shaders/muse_reference.metal:961]) is one thread per output element: it resolves row_bytes = (hidden_dim/256) × 144 — the same 3,744-byte arithmetic as §6.6 — and calls the shared muser_q4k_value scalar dequant (muse_reference.metal:946), which is §6.3’s formula indexed by element rather than streamed by block.

6.8 Tradeoffs

The n32 tile: measured occupancy over intuition. Start with the fork. The 16-row batch — the DFlash verify shape of Chapter 8 — had to run on something, and the retained kernels already covered it. Keeping them was the cheap answer, and on paper it looked like the right one: same shapes, same arithmetic, only the batch width had changed. The bandwidth counter disagreed. Those projections came back at ~50–180 GB/s on an ~800 GB/s-class M3 Ultra, and the reason was structural rather than arithmetic: the n_out/64 K-serial threadgroup shape carried 6–20 KiB of threadgroup memory, and a threadgroup that fat could not fill a core [ledger §Stage B L0]. The kernel was not doing too much work. It was leaving most of the machine standing idle.

That reframed the problem from arithmetic to occupancy — and occupancy is a shape question, so the microbenchmark-first L-series went shape-hunting before anything touched the engine. The t128/t64/cross-sg designs were iterated and none of them won. What won was a weight-stationary tile, m16_q*k_n32: 32 output rows per threadgroup, 64-K stages, 6 KiB of threadgroup memory. The verify-cycle matmul estimate fell 148.3 → 82.8–85.0 ms against the retained SGM tile, with the pinned mul_mm measured at ~177 for scale [ledger §Stage B L0]; in the integrated engine the verify forward followed it down, 202.3 → 128.4 ms [ledger §Stage B L1].

Speed alone would not have shipped it. Exactness was gated, not assumed: the tile landed “within the accepted half-staged error envelope (max-abs identical to the pinned kernels on the same data, zero argmax flips on every shape)” [ledger §Stage B L0]. That sequence is the house style — hypothesis (occupancy), apparatus (muser-m16-bench at the exact Figure 6.6 shapes), then a gate — and the lesson outlives this particular tile: when a kernel runs far under the machine’s rated bandwidth, suspect its shape before you suspect its math.

Q5_K at 5.5 bits on the lm_head — and the transcode temptation. The promoted lm_head is the one place this artifact pays a premium you can see from orbit, so it is fair to ask what that premium buys and whether a cheaper trick could have dodged it. The 5-bit lm_head costs 924,571,648 B (§6.5 arithmetic) — 1.22× a Q4_K equivalent (756,467,712 B).

There is a tempting move available here, and the tree already contains a worked example of it. The Q5_0 sibling has an instructive in-tree trade: transcode_q5_to_q8 expands 5-bit blocks to Q8_0 at GPU upload time, “trading 30 % more memory bandwidth for 3× fewer GPU decode instructions” [crates/muser-engine/src/quant/blocks.rs:100-103]. The reason that digression matters for the lm_head is that it is the same class of trade the n32 tile makes, run in the opposite direction: one spends bandwidth to buy decode instructions, the other spends decode work to buy occupancy. On the live path the lm_head takes neither deal. It stays Q5_K and rides m16_q5k_n32 at 16 rows: 5.87 → 4.11 ms per dispatch against mul_mm [ledger §Stage B L0].

Why not Q6_K everywhere? By Figure 6.1, Q6_K is 6.5625/4.5 = 1.458× the bytes of Q4_K. Every weight byte is read per token (§6.6), so Q6_K-everywhere would scale the per-token stream by ~1.46× and, in the bandwidth-bound regime Chapter 1 proved, cut throughput by the same factor. The artifact spends 6.5625 bits only where the recipe says it pays (ffn_down on half the layers, 26 k/v tensors — Figure 6.6); whether that specific allocation is the quality optimum is the recipe designer’s claim, not a Muser measurement [unverified].

6.9 Where the gap lives

Every chapter in this book has to answer the same question about its own subject: is this where the missing time hides? Here the answer is no — and the way that answer was reached matters more than the answer itself, because it was reconciled rather than assumed.

The quant format is not the decode gap. When the engine’s one-token graphs were reconciled closure-by-closure (the +196 dispatch-gap diagnosis), the families were 104 norm-boundary groups, 39 SWA staging groups, 52 KV-publication splits, one copy — not matvec arithmetic [docs/decode-dispatch-gap-20260815.md]. The kquant matvecs themselves run on llama.cpp’s own pinned kernels for numerical parity ([crates/muser-engine/src/metal/encode.rs:278-280]), so on this lane the weight-format question was settled before it could become a gap: same bytes, same kernels, same arithmetic order as the comparator. The lane’s measured decode ratios — 1.0274–1.0504× across the six-depth plain matrix, five exact-token reps per depth [docs/benchmarks.md §1] — carry the paradox this book keeps meeting: parity engines can still differ in dispatch, and the gap lives there, not here.

6.10 What comes next

The kquant family is integer codebooks: uniform grids, local scales, a min where the format wants one. Chapter 7 meets the other family — NVFP4, where the 4-bit payload is itself a tiny float, the block scale is itself a tiny float, and the whole lane was built for the same bytes-per-token argument with a different arithmetic inside. You will pack one of its groups by hand (using the repo’s own fixture bytes), watch the fail-closed loader bind its scale tensors, and read the Metal kernel that keeps the whole contraction in integers — and you will see why this lane’s 35.491 tok/s must never be called “faster” than kquant’s 35.440.

References

• [crates/muser-engine/src/gguf/types.rs:9-127] — GgmlType, block sizes/elements (Figure 6.1’s source), the NVFP4_E2M1/F8_E4M3FN doc.
• [crates/muser-engine/src/quant/k_block.rs:12-49] — dequant_q4_k, this chapter’s primary source (§6.2, §6.3).
• [crates/muser-engine/src/quant/k_block.rs:51-105] — dequant_q5_k and the 176-byte layout doc.
• [crates/muser-engine/src/quant/k_block/q6.rs:15-75] — dot_q6_k_f32_llama: the 210-byte layout (ql/qh/sc/d offsets), signed-code extraction, deferred-scaling order.
• [crates/muser-engine/src/loader.rs:72-91] — weight_precision: fail-closed lane selection (q4_k_xl default).
• [crates/muser-engine/src/decode.rs:136-139], :1209 — Metal-path dtype admissions for projections and the embedding.
• [crates/muser-bench/src/m16.rs:137-226] — the SHAPES table: dtypes, real dimensions, per-cycle multiplicities (Figure 6.6).
• [crates/muser-engine/src/metal/encode/qkv.rs:414-641] — encode_quantized_matmul: the full dispatch ladder (Figure 6.7), including the source-pinned boundary comment at :476.
• [crates/muser-engine/src/metal/encode.rs:278-280] — the pinned kernel_mul_mv_q{4,5,6}_K_f32 metallib PSOs.
• [crates/muser-engine/src/shaders/muse_reference.metal:735-788] — muser_matvec_q4k_4r2s; :41-62 muser_decode_all_q4k_scales; :961-977 muser_embedding_q4k.
• [crates/muser-engine/src/quant/blocks.rs:100-103] — the Q5→Q8 transcode trade comment.
• [crates/muser-engine/src/weights.rs:4-7] — row-contiguity / prefill DRAM-amortization note.
• [docs/decode-dispatch-gap-20260815.md] — the +196-closure reconciliation (§6.9).
• [ledger §Stage B L0], [ledger §Stage B L1] — docs/goal-parity-ledger-2026-08.md, the M16 microbenchmark and integration entries (§6.8’s ms figures).
• [docs/benchmarks.md §1] — the six-depth plain matrix ratios.
• [claims #11] — kquant 35.440 / NVFP4 35.491 tok/s scope (§6.1, §6.9).
• Ch 5 — the format-agnostic template this chapter instantiated.
• [ferrite-book Ch 5] — the ancestor Q4_K chapter; Muser’s dequant_q4_k is the same extraction lineage (NOTICE, docs/extraction-manifest.md).

Chapter 7 — NVFP4: the native lane

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Chapters 5 and 6. You know what a codebook, a block scale, and a bitrate are; you have unpacked kquant super-blocks by hand. This chapter does the same for the other weight lane — NVFP4 — and then tells the lane’s measured story, which is a story about what a format may and may not claim.

Chapter 6 left the kquant family as integer codebooks: uniform grids under local scales, in a 16.76 GB artifact that the reference lane streams every token. NVFP4 — Native Float Precision 4-bit, the format of NVIDIA’s Blackwell FP4 tensor cores — is the same budget spent on a different idea: the 4-bit payload is itself a tiny floating-point number, and so is its block scale. Muser’s native lane decodes the Muse Glimmer weights directly from this format on the Mac, and the same format runs the remote GX10 prefill producer (Ch 28).

The chapter has two halves. The first is bytes and arithmetic: the e2m1 codebook, the e4m3fn scale, the fail-closed loader, and the Metal decode kernels. The second is discipline: what the measurements permit this lane to claim — and the one claim it must never make.

7.1 The format in one line — and its price tag

What does a weight format actually have to promise before anyone can trust it? Three things: how to read the payload, how to scale it, and in what order to multiply the pieces back together. Skip the third and you do not have a format, you have a suggestion — two implementations can agree on every byte on disk and still disagree on the answer.

Muser’s own CPU oracle states all three in its module doc, order included:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/quant/nvfp4.rs:1
//! CPU oracle for NVIDIA NVFP4 weights.
//!
//! The product format keeps E2M1 values packed two per byte, one raw E4M3FN
//! scale per 16 values, and one f32 `scale2` per tensor.  The operation order
//! is pinned to the ModelOpt/MLX reference: `(e2m1 * e4m3fn) * scale2`.
}

Deconstruct each term:

• E2M1 is the 4-bit float codebook: 1 sign, 2 exponent, 1 mantissa bit. Sixteen entries, and Muser lists them all ([crates/muser-engine/src/quant/nvfp4.rs:7-9]):

 code      :  0     1     2     3    4    5    6    7
 value     : 0.0   0.5   1.0   1.5  2.0  3.0  4.0  6.0
 code (|S) :  8     9    10    11   12   13   14   15
 value     :-0.0  -0.5  -1.0  -1.5 -2.0 -3.0 -4.0 -6.0

Figure 7.1: The complete E2M1 codebook — there is nothing else it can mean. Note the spacing: 0.5-steps up to 2, then 3, 4, 6 — coarser as magnitude grows, the signature of a float grid.

• E4M3FN is the 8-bit block scale: 1 sign, 4 exponent, 3 mantissa, bias 7, no infinities, maximum magnitude 448, and — the “FN,” finite — a NaN at only two encodings (0x7f/0xff), which the loader rejects outright ([crates/muser-engine/src/weights.rs:244-248]).
• One scale per 16 values, packed two values per byte, plus one f32 scale2 per tensor.

The bitrate arithmetic, shown as always:

4 bits/weight (payload)  +  8 bits / 16 weights (E4M3FN scale)  =  4.5 bits/weight

Exactly 4.5 — the same figure as Q4_K, reached with a 16-element block and a one-byte float scale instead of a 256-element super-block with 6-bit integer sub-scales. (The per-tensor f32 scale2 adds 4 bytes per tensor: at the q projection’s 27.3 M weights, that is 1.2×10⁻⁷ bits/weight — nothing.) The GGUF type registry says it out loud: the companion-tensor design “keeps the serving representation at exactly 4.5 bits/weight” ([crates/muser-engine/src/gguf/types.rs:33-36]).

 One NVFP4 row (n_in values) — three separate regions:

   packed E2M1            E4M3FN scales          (per-tensor, once)
   ┌────────────────┐     ┌──────────────┐       ┌─────────┐
   │▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓│ ... │▓ ▓ ▓ ▓ ... ▓ │  ...  │ scale2  │
   └───────┬────────┘     └──────┬───────┘       └─────────┘
   n_in/2 bytes           n_in/16 bytes          f32
   2 values per byte      1 scale per 16 values  whole tensor

   16-value group detail:
   [ w0w1 ][ w2w3 ][ w4w5 ][ w6w7 ][ w8w9 ][w10w11][w12w13][w14w15] [ s ]
     1 B     1 B     1 B     1 B     1 B     1 B      1 B      1 B    1 B
   lo nibble = even index, hi nibble = odd index; s decodes by Figure 7.1.

Figure 7.2: The NVFP4 layout. A group is 9 bytes of storage for 16 values — 4.5 bits each — and the dequant order is pinned: (e2m1 × e4m3fn) × scale2, never regrouped.

7.2 A worked dequant, using the repo’s own fixture

A codebook you have only read about is a codebook you do not yet trust. So before the loader, before the kernels, before any measurement, we decode a real group by hand and then check our answer against the repo’s own. Muser’s unit test for the loader carries a canonical group, and we will dequantize its first bytes with every multiply shown. The fixture: eight packed bytes 10 32 54 76 98 BA DC FE, one scale byte 0x38, and scale2 = 0.25 ([crates/muser-engine/src/weights.rs:462-465, 519-536]).

Decode the scale first. 0x38 = 0b0011_1000: sign 0; exponent = (0x38 >> 3) & 0xF = 7; mantissa = 0x38 & 7 = 0. Normal case ([crates/muser-engine/src/quant/nvfp4.rs:118]):

scale = (1 + 0 × 0.125) × 2^(7−7) = 1.0

Then the weights, in pinned order — look up Figure 7.1, multiply by the scale, multiply by scale2:

byte 0x10 → lo nibble 0x0 → 0.0  → 0.0  × 1.0 × 0.25 =  0.000
          → hi nibble 0x1 → 0.5  → 0.5  × 1.0 × 0.25 =  0.125
byte 0x32 → lo nibble 0x2 → 1.0  → 1.0  × 1.0 × 0.25 =  0.250
          → hi nibble 0x3 → 1.5  → 1.5  × 1.0 × 0.25 =  0.375
byte 0x54 → lo 0x4 → 2.0 → 0.500      hi 0x5 → 3.0 → 0.750
byte 0x76 → lo 0x6 → 4.0 → 1.000      hi 0x7 → 6.0 → 1.500
byte 0x98 → lo 0x8 → −0.0 → −0.0      hi 0x9 → −0.5 → −0.125

The test asserts exactly this row — 0, 0.125, 0.25, 0.375, 0.5, 0.75, 1.0, 1.5, −0.0, −0.125, … — bit-for-bit ([crates/muser-engine/src/weights.rs:527-535]). The GPU agrees: a dedicated fixture kernel muser_nvfp4_dequant_fixture computes (muser_e2m1(nibble) * muser_e4m3fn(scales[index/16])) * scale2 per element and is tested bit-exact against this CPU oracle for every finite E4M3FN byte — all 254 of them ([crates/muser-engine/src/shaders/nvfp4.metal:775-788], [crates/muser-engine/src/metal/encode/qkv.rs:674-699]). The sweep is exhaustive on purpose: a decode table this small can be checked completely instead of sampled, and anything that can be checked completely should be.

Two format details worth internalizing before the loader:

• The rounding contract is pinned too. Quantizing into e2m1 uses ModelOpt’s ties-to-even midpoints (e2m1_from_f32, with the boundary table at [crates/muser-engine/src/quant/nvfp4.rs:150-171]; the GPU twin muser_e2m1_round at nvfp4.metal:121). A format is not just its decode table — it is also how you got there.
• The scale2 is a real second factor, not decoration: it is what lets every per-16 scale stay small while tensors span orders of magnitude. The loader requires it to be finite and positive ([crates/muser-engine/src/weights.rs:263-267]).

7.3 The loader path: fail-closed companions

The format promises a payload, a scale, and an order. What breaks if one of those pieces is missing, mis-shaped, or hiding a NaN? Nothing loud, and that is exactly the danger. A wrong scale tensor does not crash the model. It makes it quietly and unevenly worse, in a way no smoke test catches and no user can report precisely. The native lane’s answer is to refuse to start at all.

NVFP4 tensors never stand alone. Each weight matrix carries two (or three) companion tensors, named by suffix ([crates/muser-engine/src/weights.rs:25-27]):

 token_embd.weight                    ← NVFP4_E2M1 payload (n_in/2 × n_out bytes)
 token_embd.weight.nvfp4_scale        ← F8_E4M3FN, shape [n_in/16, n_out]
 token_embd.weight.nvfp4_scale2       ← F32 scalar, one per tensor
 token_embd.weight.nvfp4_input_scale_inv  ← optional F32 scalar (W4A4 only)

The lane itself is selected by metadata, and the pairing is strict in both directions ([crates/muser-engine/src/loader.rs:72-91]): native NVFP4 tensors present ⇒ muser.weight_precision must say nvfp4; the string nvfp4 with no native tensors ⇒ error; absent or q4_k_xl ⇒ the kquant lane of Chapter 6. Then MuseWeights::open validates every companion before any inference: the scale tensor must exist, be F8_E4M3FN, and match the exact [n_in/16, n_out] shape; it must contain no NaN scale byte; scale2 must be scalar F32, finite, positive ([crates/muser-engine/src/weights.rs:220-267]). A missing companion is a load-time MissingTensor error — there is no “degrade and continue.”

The optional fourth tensor decides the arithmetic mode — and the loader enforces that too. input_scale_inv present ⇔ muser.activation_precision = nvfp4; a mismatch fails the load ([crates/muser-engine/src/weights.rs:200-216]). Its meaning:

• absent ⇒ weight-only W4A16: activations stay wide, and the Mac quantizes them per super-block to a Q8-K-style integer grid at compute time (§7.5). This is the product configuration — the selected candidate is “weight-only W4A16 and has no activation-global-scale tensor” [ledger §P1.4].
• present ⇒ W4A4: activations are dynamically quantized to FP4 in groups of 16 before the dot — the compressed-tensors scheme the Blackwell producer’s tensor cores execute natively. The Mac supports it (for batched verify parity with the producer), and §7.6 shows where that permission ends.

Row addressing falls out of Figure 7.2 and is worth one glance at real code — dequant_row for a NVFP4 tensor slices n_in/2 packed bytes and n_in/16 scale bytes per row ([crates/muser-engine/src/weights.rs:82-94]), the same row-major contiguity kquant enjoys.

7.4 The Metal lane, end to end

Bytes on disk are only half the contract; the other half is which kernel reads them. And that choice is not the kernel author’s to make at runtime — it was already settled at load time, by the presence or absence of the optional fourth companion tensor.

On the Metal side, every projection routes through encode_projection, which dispatches on exactly the dtype set Chapter 6 listed ([crates/muser-engine/src/decode.rs:6044-6091]): F16 → encode_f16_matmul, NVFP4 → encode_nvfp4_matmul, else the kquant ladder. The NVFP4 encoder is where the mode split becomes kernel selection (Figure 7.3; [crates/muser-engine/src/metal/encode/qkv.rs:128-227]):

 encode_nvfp4_matmul(packed, scales, scale2, input_scale_inv, …)

 input_scale_inv = None  →  muser_nvfp4_a16_q8_matvec       (weight-only, W4A16)
                              grid (n_out/8 × columns), 256 threads
 input_scale_inv = Some  →  16-column batch, n_in % 64 == 0:
                            muser_nvfp4_w4a4_m16_n32        (weight-stationary tile)
                          else widths 16/8/4/2/1:
                            muser_nvfp4_w4a4_matvec_c{1,2,4,8,16}
                              grid n_out, 32 threads (one SIMD group per row)

Figure 7.3: The native-lane projection dispatch. The Nvfp4Args triple (n_in, n_out, col0) rides buffer 4; scale2 buffer 5; input_scale_inv buffer 6. All three kernels live in the no-fast-math cross-vendor library (Chapter 4’s second source) — the integer contractions must not be reassociated by the compiler ([crates/muser-engine/src/metal/encode/qkv.rs:203-205]).

7.4.1 The plain decode kernel (muser_nvfp4_matvec_c1 family)

The A16 matvec template (the Ch 6 term — one dot product per output row), trimmed to its arithmetic core:

// crates/muser-engine/src/shaders/nvfp4.metal:183
template <ushort NC>
inline void muser_nvfp4_matvec_impl(
    device const uchar *packed,
    device const uchar *scales,
    device const float *input,
    device float *output,
    constant muser_nvfp4_args &args,
    constant float &scale2,
    uint row,
    ushort lane) {
    // …
    for (uint group = uint(lane); group < args.n_in / 16; group += 32) {
        const float block_scale = muser_e4m3fn(scales[scale_row + group]);
        const uint packed_base = packed_row + group * 8;
        const uint element_base = group * 16;
        for (ushort column = 0; column < NC; ++column) {
            device const float *x = input + (args.col0 + uint(column)) * args.n_in;
            float block_sum = 0.0f;
            muser_nvfp4_accumulate_codes8(
                packed + packed_base, x + element_base, block_sum);
            muser_nvfp4_accumulate_codes8(
                packed + packed_base + 4, x + element_base + 8, block_sum);
            const float scaled = block_sum * (block_scale * 16384.0f);
            sums[column] += scaled;
        }
    }
    for (ushort column = 0; column < NC; ++column) {
        const float total = simd_sum(sums[column]) * scale2;
        if (lane == 0) {
            output[(args.col0 + uint(column)) * args.n_out + row] = total;
        }
    }
}

One lane pair per 16-value group, the group’s scale applied once per block, simd_sum across the 32 lanes of a SIMD group (Ch 2), scale2 once per row — the exact pinned order of §7.2, vectorized. The one mystery is the 16384.0f (= 2¹⁴). The accumulate helper decodes eight E2M1 nibbles as f16 bit patterns shifted into place — each value embedded as an exact half multiple of 2⁻¹⁴ — so the helper’s sum is 2⁻¹⁴ × the true block sum, and the caller “folds 2^14 into the block scale after accumulating one complete 16-value block” ([crates/muser-engine/src/shaders/nvfp4.metal:155-158]). The comment names the lineage: “the same half-bit embedding used by MLX’s native NVFP4 kernels.” This is how you turn a 16-entry float LUT into pure bit-slicing on a GPU.

Turn that around, because it is the kind of trick that only clicks the second time: the kernel never looks a value up. It shifts nibbles straight into half-precision bit patterns, which is fast, but leaves every decoded value shrunk by one fixed power of two. The kernel pays that debt back exactly once per block, folded into the block scale, where it costs a single multiply instead of one per element.

7.4.2 The weight-only contraction (muser_nvfp4_a16_q8_matvec)

Ask what can go wrong inside a dot product and the pedantic answer turns out to be the right one: the order in which you add the terms. On a single machine that order is a rounding footnote nobody reads. Across two machines it becomes a disagreement nobody can debug.

So the product lane’s decode kernel is stricter than floats: it makes the whole block contraction integer-exact. Per 256-value activation super-block, the threadgroup quantizes the activation to a Q8-K-style integer grid once — signed first absolute maximum, iscale = −127/first_max, magic-number round-to-nearest-even, clamp at 127 — then shares the integers across eight output rows. Inside a 16-value group the dot runs on raw integer codes:

// crates/muser-engine/src/shaders/nvfp4.metal:682
for (ushort pair = 0; pair < 8; ++pair) {
    const uchar byte = row_packed[packed_base + uint(pair)];
    dot += muser_e2m1_q1(byte & 15) * int(q8[quant_base + uint(pair) * 2u]);
    dot += muser_e2m1_q1(byte >> 4) * int(q8[quant_base + uint(pair) * 2u + 1u]);
}
weighted = long(dot) * long(muser_e4m3_q9(row_scales[scale_group]));

The helpers are the trick from §7.2 turned algebraic: muser_e2m1_q1 decodes each weight as an integer in units of 2⁻¹, muser_e4m3_q9 each scale as an integer in units of 2⁻⁹ (nvfp4.metal:28-42; the CPU oracle’s e2m1_q1/e4m3fn_q9 at [crates/muser-engine/src/quant/nvfp4.rs:17-51] prove the equivalence exhaustively). Their products share one fixed denominator, so the entire block contraction is an order-free i64 sum. Only the epilogue is floating point — and it is pinned to four scalar operations:

// crates/muser-engine/src/shaders/nvfp4.metal:691
float contribution = fma(float(integer_total), 0x1p-10f, 0.0f);   // 2^-10 = Q1 × Q9
const float q8_scale = 1.0f / block_iscale;
contribution = fma(contribution, q8_scale, 0.0f);                 // activation scale
contribution = fma(contribution, weight_scale2, 0.0f);            // tensor scale
total = fma(1.0f, contribution, total);                           // sequential accumulation

— then one final float(half(...)) at the output, because the producer’s linear layers write F16 results and “otherwise Q/K RMS normalization amplifies hidden low bits” ([crates/muser-engine/src/quant/nvfp4.rs:331-337]). The CPU oracle dot_nvfp4_a16_q8_f32 performs the identical sequence ([crates/muser-engine/src/quant/nvfp4.rs:349-379]).

Now the question this section opened with has its answer. The lane must match a CUDA producer whose reduction topology no Metal simd_sum can reproduce — so rather than chase that topology, the design removes the need for it. Make the parallel part exactly associative, and the only rounding decisions left are the four named scalar FMAs above, in an order both machines can agree to in writing. Chapter 32 tells the full trust story; file the technique now.

7.4.3 The W4A4 contraction and the F16 tail

The other mode is the one the producer speaks, and the Mac keeps it so the two machines can be compared on identical arithmetic rather than on faith. When input_scale_inv is present, the kernel quantizes each 16-value activation group to its own E2M1 + E4M3FN pair (the CPU oracle at [crates/muser-engine/src/quant/nvfp4.rs:174-238]), then contracts weight-Q1 × activation-Q1 × weight-scale-Q9 × activation-scale-Q9 as one i64 integer sum per group — denominator 2⁻²⁰ — with the epilogue × 2^-20, × weight_scale2, × (1/input_scale_inv), and the same F16 boundary ([crates/muser-engine/src/shaders/nvfp4.metal:295-308]). The 16-column batch form (muser_nvfp4_w4a4_prequant_m16_n32) splits the activation quantization into its own pass so each group is quantized once and reused across every N=32 output tile ([crates/muser-engine/src/metal/encode/qkv.rs:8-65]).

The lane’s F16 tail is real code, not a footnote: the unquantized F16 LM head runs on muser_f16_matvec_c* — plain half4 dot products ([crates/muser-engine/src/shaders/nvfp4.metal:704-752]) — and the embedding on muser_embedding_f16 (nvfp4.metal:755), both dispatched by wrappers that pick the F16 route by byte length or dtype ([crates/muser-engine/src/metal/encode/qkv.rs:348-360]). “Checkpoint mandates an unquantized F16 language head” is a property of this artifact, recorded in the ledger (§7.5).

7.5 The measured lane: parity within noise — never “faster”

So: is the native lane faster? That is the question the whole format was supposed to answer, and answering it honestly turns out to require the most carefully worded sentence in this chapter.

Here is the lane’s headline measurement, quoted from the ledger’s P1.3 decode gate — five-rep cell, same 66-token prefix, 32 teacher-forced tokens, F16 KV, flash attention, release binary, adjacent lease window ([ledger §P1.3]):

 lane          mean ns / 32 tokens      CV          tok/s
 ───────────   ─────────────────────   ─────────   ─────────────
 native NVFP4  901,644,358.4           0.13 %      35.490711722
 kquant ctrl   902,946,575.0           0.037 %     35.439527527

 native = 1.001444269× the adjacent kquant control (+0.144427 %)

Figure 7.4: The P1.3 paired decode cells ([docs/goal-parity-ledger-2026-08.md]; receipt SHAs in the entry).

Read the discipline in the numbers. A 0.14% difference between cells whose CVs are 0.03–0.13% is parity within noise, and the claims register locks the wording: “plain Mac NVFP4 35.491 tok/s versus adjacent kquant 35.440 tok/s remain valid at their original scopes” — with the standing instruction “Never call decode faster” [claims #11]. The book will not phrase it loosely either: the native lane’s decode is parity-within-noise, full stop.

Why is the gate so narrow — shouldn’t 4.5-bit float weights with an integer-exact kernel fly? The ledger’s retained diagnostics answer: “the NVFP4 layer stack is faster, but this checkpoint mandates an unquantized F16 language head (about 3.46 ms/token versus the kquant head’s 1.75 ms)” [ledger §P1.3]. The F16 head alone eats most of what the 52-layer FP4 stack saves. So much for speed. The lane’s other two anchors are about quality, and they are gates rather than boasts.

The first is determinism. The standard 2,048-token fixture is deterministic and token-identical versus the exact anchor, with bounded nonzero logit drift: max/mean absolute error 7.270581/1.040619 at 32 tokens, and 10.884401/1.233789 over the five-rep 2,048/256 comparator [docs/nvfp4-fast-lane-evidence-20260817.md §Determinism]. Nonzero is the load-bearing word. The tokens match; the logits underneath them do not, and saying “zero drift” — the easier, rounder sentence — is explicitly prohibited wording [claims #10].

The second anchor is where the cost of the format finally becomes visible. At depth, one content class — documentation and digest text at 65,536 tokens — exceeds its calibrated top-token band, 15.134% against a 13.339% gate. It did not replicate across documents, so it is published as exactly what it is: a content-local sensitivity, with the kquant lane selectable as the reference route [claims #10]. The cost of this quantization is real, measured, and localized — Chapter 5’s §5.8 promise, kept.

7.6 Speculative NVFP4: measured, rejected, fail-closed

The lane’s one forbidden fruit is speculation — and how we found that out teaches more than the verdict does.

The fork looked inviting from both sides. Speculative decoding earns its speed by checking many drafted tokens in a single batched forward pass, and batched verification is precisely the shape W4A4 was built for: the activations already arrive quantized in groups, and it is the arithmetic the Blackwell producer runs natively on its tensor cores. Every argument pointed the same direction. So we ran the diagnostic — native NVFP4 W4A4 batched verification, measured against the kquant speculative bar of 107.9 tok/s — expecting, at worst, the same neighborhood.

It came back at 6.805 tok/s. That is not a regression you tune away; it is a different order of magnitude, and the retained run says plainly where the time went. Verification alone consumed 35.915 s of a 37.619 s decode span [ledger §F-series remediation] — the drafting was effectively free while the verify pass ate the whole budget. We kept the run and labeled it for what it is, “one diagnostic, explicitly unqualified” ([docs/nvfp4-fast-lane-evidence-20260817.md §Measured product numbers]). Both halves of that label are load-bearing: one measurement is enough to stop a launch claim, and not enough to characterize a lane.

The lesson was about shape, not about format. A lane can sit at parity-within-noise in plain decode, as this one does, and still be unqualified the moment the batch shape changes underneath it. Nothing in the decode gate above was ever evidence about verification, and reading it as such is how a slow route ships quietly.

So the disposition is recorded as Fallback B: “speculative serving stays on the qualified kquant lane; native NVFP4 plus DFlash is rejected by the receiver configuration rather than silently serving the measured 6.81 tok/s route” [docs/nvfp4-fast-lane-evidence-20260817.md §Product route], and the claims register repeats it: “Native NVFP4 speculative decode has no launch claim and remains fail-closed” [claims #4].

And it is code, not policy prose. A receiver configuration declaring the native producer mode cannot even enroll a DFlash identity ([crates/muser-cluster/src/config.rs:128-131]), and the server refuses the combination at startup:

#![allow(unused)]
fn main() {
// crates/muser-server/src/state.rs:1667
fn validate_remote_dflash_policy(
    producer_mode: Option<Nvfp4ProducerMode>,
    dflash_configured: bool,
) -> Result<(), InferenceLoadError> {
    if producer_mode == Some(Nvfp4ProducerMode::Native) && dflash_configured {
        return Err(InferenceLoadError::Remote(
            "native NVFP4 fast-lane speculative decode is unqualified; omit --dflash and use plain NVFP4 decode, or route speculative serving to the kquant lane"
                .into(),
        ));
    }
    Ok(())
}
}

The operator sees that message and the process refuses to serve. Ch 33 owns the full postmortem — including why a 240/240 all-accept target+DFlash diagnostic still proved nothing about throughput.

7.7 Exact vs Native: the producer-mode preview

That refusal leaves one question dangling. If the Mac decodes weights that someone else’s GPU also understands, who decides what the numbers are supposed to be? The Nvfp4ProducerMode enum you just saw is where that decision is written down, and it is this chapter’s bridge to Part VI, so meet it on its own terms ([crates/muser-cluster/src/config.rs:10-18]):

#![allow(unused)]
fn main() {
/// Numeric contract selected by the Spark NVFP4 producer. Legacy receiver
/// configurations predate the split and therefore deserialize as `None`;
/// newly generated F-series configurations must name the mode explicitly.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Nvfp4ProducerMode {
    Exact,
    Native,
}
}

• Native is the product: the GX10’s vLLM producer runs the deterministic Blackwell W4A4 path, prefilling KV that the Mac receives over Handoff V2 and decodes with the weights of this chapter.
• Exact is the verification anchor: an integer-dot producer mode whose KV exists to be compared against, not served. It is selected on the producer side — MUSER_NVFP4_EXACT is a Python environment flag for the producer container (scripts/gx10/vllm/benchmark_native_prefill.py:99-102 even refuses to benchmark native with it set; the resident daemon pins MUSER_NVFP4_EXACT=0 at scripts/gx10/vllm/muser_native_prefilld.py:446). It does not exist anywhere in Rust — a reviewer checking the Mac tree for it will find nothing, and that is by design.

The two modes use mode-separated target-cache identities, “so exact and native KV entries cannot alias” [docs/nvfp4-fast-lane-evidence-20260817.md §Product route]. What “good enough” must mean when someone else’s GPU computed your prefill — bounded logit envelopes, exact-token policies, the wizard’s gates — is Ch 32, the trust chapter.

7.8 Tradeoffs

Four decisions in this lane could each plausibly have gone the other way. What follows is what choosing cost — measured, not argued.

Float codebook vs integer codebook at the same 4.5 bits. Chapter 5’s two families meet head-on here. E2M1 spends its 16 codes as 15 magnitudes plus a signed zero — one fewer value than Q4_K’s uniform grid — but buys relative spacing (Figure 7.1: each octave halves the resolution instead of losing it absolutely) and a float block scale (E4M3FN spans 2⁻⁹·mantissa subnormals up to 448) where kquant’s 6-bit sub-scale is an integer under an f16 super-scale. The measured consequence of the whole package is Figure 7.4: parity-within-noise in decode, with quality sensitized by content class rather than collapsed [claims #10] [claims #11]. A clean “float beats int at 4 bits” claim is not what this program’s evidence supports — both 4.5-bit formats land at the same throughput, and the int lane is the reference lock.

Group 16 vs super-block 256. NVFP4’s 0.5 header bits buy a scale refresh every 16 weights — 16× finer than kquant’s 32-element sub-blocks — but the scale is a full byte each. Q4_K’s two-level packing spends its 0.5 bits on more integers under two f16s. The two formats agree on the budget and disagree on everything else, which is the best evidence that 4.5 bits is a genuine economic point and not an artifact of one design.

Integer-exact contraction vs fast floats. §7.4.2’s a16-q8 kernel gives up vectorized-float convenience to make the block dot associativity-free. The alternative — matching CUDA’s reduction by luck — was measured out in this program’s own wizard campaign, where a one-ULP F16 divergence in layer-1 V snowballed into 51.7 M differing logits before the arithmetic ABI was pinned [ledger §2b, 2026-08-24]. The cost of the integer path is kernel complexity; the measured benefit is that “deterministic, bounded drift” became a property rather than a hope [docs/nvfp4-fast-lane-evidence-20260817.md §Determinism].

W4A4 on the Mac: supported, qualified for batch parity, forbidden for speculation. The same input_scale_inv that enables producer-parity M16 verification (§7.4.3) is the mode whose serving use measured 6.805 tok/s (§7.6). The lane keeps the kernels, gates the deployment — Fallback B as a design pattern: fail-closed beats silently-slow.

7.9 Where the gap lives

Every kernel chapter owes the same accounting: where does this component’s time go, and is it the deficit? For the native lane, “the gap” in the dispatch-gap sense barely exists — plain decode sits at parity-within-noise with both the kquant lane and the comparator (§7.5), and the format is not the deficit. Where this lane’s own numbers diverge from its hopes is quantization-shaped but not format-shaped: the mandated F16 LM head (3.46 vs 1.75 ms/token) narrowing the decode gate [ledger §P1.3], and the W4A4 batched verify collapsing to 6.81 tok/s where kquant’s verify was engineered to 107.9-class [ledger §F-series remediation]. Both are batch-shape stories — the cost of precision hides in batch shapes and content classes, and the gates exist to localize it.

7.10 What comes next

Two weight lanes now stand complete: kquant for the reference and speculative lanes, NVFP4 for the native product lane. The next chapter follows the one component that runs on exactly one of them — the DFlash draft model, a five-layer kquant assistant that reads the target’s hidden states and proposes tokens for exact verification. It is the smallest model in this book, the cheapest thing in memory-footprint.md’s manifest after nothing, and — thanks to one wrong constant that survived an entire campaign — the best teacher of what a draft must guarantee.

References

• [crates/muser-engine/src/quant/nvfp4.rs:1-6] — the pinned format contract (§7.1); :7-51 E2M1 LUT and the Q1/Q9 integer encodings; :109-171 E4M3FN decode and the ModelOpt rounding boundaries; :241-255 dequant_nvfp4_row; :349-379 dot_nvfp4_a16_q8_f32.
• [crates/muser-engine/src/gguf/types.rs:33-39] — NVFP4_E2M1 / F8_E4M3FN registration and the “exactly 4.5 bits/weight” comment.
• [crates/muser-engine/src/loader.rs:72-91] — fail-closed lane pairing.
• [crates/muser-engine/src/weights.rs:25-27] — companion-tensor suffixes; :220-290 nvfp4_aux validation (shape, dtype, NaN rejection, positive scalar scale2, activation-precision pairing); :519-536 the fixture test this chapter’s worked example mirrors.
• [crates/muser-engine/src/shaders/nvfp4.metal:1-5] — the GPU format doc; :17-42 LUT and integer twins; :155-158 the MLX half-bit embedding; :183-242 the A16 matvec family; :613-702 muser_nvfp4_a16_q8_matvec; :704-770 the F16 matvec/embedding kernels; :775-788 the dequant fixture kernel.
• [crates/muser-engine/src/metal/encode/qkv.rs:128-227] — encode_nvfp4_matmul dispatch and geometries; :8-65 the two-pass M16 prequant route; :348-360 F16-layout detection; :674-699 the bit-exact E4M3FN sweep test.
• [crates/muser-engine/src/decode.rs:6044-6091] — projection routing; :3254-3291 the verify-route banner (mode names as the engine prints them).
• [crates/muser-cluster/src/config.rs:10-18], :128-131 — Nvfp4ProducerMode; native mode cannot enroll DFlash geometry.
• [crates/muser-server/src/state.rs:1667-1678] — validate_remote_dflash_policy, the fail-closed Fallback B refusal.
• [ledger §P1.3], [ledger §P1.4], [ledger §F-series remediation], [ledger §2b 2026-08-24] — docs/goal-parity-ledger-2026-08.md: the paired decode cells, the weight-only artifact correction, the 6.81 no-go, the one-ULP wizard chase.
• [docs/nvfp4-fast-lane-evidence-20260817.md] — Fallback B disposition, measured product table (incl. 6.805), determinism/drift envelopes.
• [claims #4], [claims #10], [claims #11] — docs/launch-claims.md: native spec fail-closed; quality gates and prohibited wording; the 35.491/35.440 parity scope.
• [scripts/gx10/vllm/benchmark_native_prefill.py:99-102], [scripts/gx10/vllm/muser_native_prefilld.py:446] — MUSER_NVFP4_EXACT is producer-side Python only.
• Ch 5, Ch 6 — the codebook/block template and the kquant counterpart.
• Ch 32, Ch 33 — the trust and speculation chapters this one forward-points to.

Chapter 8 — The DFlash draft

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Chapters 5–7. You know the kquant blocks the draft is made of, the lanes it may run on, and — from Chapter 7’s Fallback B — the lane it may not. This chapter is about a model small enough to hold in your head, and about the contract that lets something approximate participate in exact inference.

Chapter 7 ended with a refusal: native NVFP4 plus DFlash is rejected by the receiver configuration, so speculative serving stays on the qualified kquant lane. This chapter opens that last clause. DFlash is Muser’s draft model — the small assistant that guesses tokens cheaply so the 52-layer target only has to confirm them — and it is a kquant artifact from end to end, down to the very blocks you unpacked in Chapter 6.

One boundary statement before anything else. Speculative decoding has two halves: drafting (this chapter) and accept/verify (the algorithm that decides which guesses become output). The algorithm — verify_full_speculative_mt_ordered, the carried-frontier state machine, the rollback choreography — gets its full treatment in Ch 33. Here we build the draft, load it, condition it, and pin down what it must guarantee so that Chapter 33’s verification can stay exact.

8.1 Why a draft model at all

Start with the question the rest of the chapter is an answer to: why run a second model at all, when the whole problem is that we already cannot afford to run the first one fast enough?

Chapter 1’s fourth lever: move the work somewhere cheaper. Decode is bandwidth-bound because each token reads the whole 16.76 GB model for one token’s worth of math. Speculation attacks the denominator: if a cheap model proposes k tokens and the expensive target verifies them in one batched forward pass, then each accepted proposal cost a fraction of a full target read. The target still does all the deciding — that is what keeps the output exact — but it does k+1 rows of decision per read instead of one. The draft is pure overhead that pays for itself only when its guesses are good; the entire engineering of this chapter is making the guesses good and the overhead small.

The measured stakes, stated with the campaign’s own scope language: in retained fixed-window synthetic packets, kquant DFlash decode ratios (llama ÷ muser means) are 1.23692× at 2,048, 1.20323× at 16,384, and 1.19616× at 32,768 tokens, with 5/5 exact-token reps per depth [claims #15]. Hold onto both halves of that sentence — the ratios and the scope. We return to them in §8.6.

8.2 What DFlash is

So what does a model small enough to be worth guessing with actually look like? Small enough to describe on one page — and odd enough that the description is worth reading slowly, because DFlash is not a miniature language model in the ordinary sense. It is a passenger.

The module doc says it in five lines:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/dflash.rs:1
//! Five-layer DFlash assistant extracted from Ferrite's accepted CPU oracle.
//!
//! The target hook is Muse-specific, but the assistant math and artifact
//! format supports both the development SafeTensors export and the official
//! llama.cpp-compatible k-quant GGUF sidecar.
}

Concretely, from the config contract (DFlashConfig, [crates/muser-engine/src/dflash/config.rs:69-98]) and the bench shape table Chapter 6 quoted:

• Five layers, enforced: validate refuses anything else — “release assistant must have exactly 5 layers” (config.rs:263-268).
• It reads the target’s mind. This is DFlash’s defining trick and the reason it can be so small: it does not re-derive the target’s thinking from raw tokens. It is fed the target’s hidden states from five pinned target layers, and the engine captures those rows as the target computes them. The capture is gated rather than opportunistic: DFlashHiddenCache::write_rows accepts a row only when its layer is on the pinned list. The list itself lives in DFlashConfig.dflash_config.target_layer_ids (config.rs:66), and the gate that enforces it is [crates/muser-engine/src/dflash/hidden.rs:44-52].
• Its front door is fc.weight, shape 33,280 → 6,656 ([crates/muser-bench/src/m16.rs:219-225]). Do the arithmetic and the architecture falls out: 33,280 = 5 × 6,656 — five sampled target hidden states (the target’s hidden width, Chapter 6) concatenated, projected into the draft’s 6,656-wide hidden stream.
• Its own attention is small GQA: k/v projections are 6,656 → 1,024 (eight KV heads × head-dim 128 — the same per-token element count the context-geometry test uses, config.rs:380-386), with QK-norms and RoPE like an ordinary transformer block (config.rs:310-335 lists the tensor names: q/k/v/o projections, q_norm/k_norm, SwiGLU MLP).
• It proposes in the target’s vocabulary by borrowing the target’s own machinery: draft_greedy embeds its 16-token block with target.embed_tokens and scores the draft’s block output with target.project_hidden ([crates/muser-engine/src/dflash/spec.rs:744-752]) — the draft owns only its five layers, and the shared embedding/LM-head work is booked under assistant time in the telemetry (spec.rs:62-64). A proposal the target cannot score would be worthless, so the draft never needed a vocabulary of its own.
• It is kquant: draft.k is Q4_K, draft.v Q6_K, draft.fc Q4_K (Figure 6.6), and the GGUF loader requires every dense projection to be Q4_K/Q5_K/Q6_K ([crates/muser-engine/src/dflash/weights.rs:148-158]).
• It is cheap to keep: 1,631,205,312 B on disk ([docs/memory-footprint.md] artifact manifest), “loaded only when configured.”

Read that front-door arithmetic once more, because it is the entire design compressed into a single projection. The draft’s input is not text and not a summary of text; it is the target caught mid-thought, five times over. Every other economy in the list — the few layers, the narrow hidden stream, the borrowed embedding and LM head — is affordable only because that first matrix hands the assistant a running start it never had to earn for itself. A model that had to understand the conversation could not be this small.

The draft runs 16-row blocks: block_size defaults to 16 (config.rs:138-140), and drafting produces up to 15 proposals in one block forward — which is exactly why the M=16 batch kernels of Chapter 6 exist.

8.3 The context ABI: a 64-row sink plus a trained window

Two questions govern this section, and the second is the dangerous one. How much of the conversation does the draft get to see? And who is allowed to answer that question — the artifact, or the machine that happens to be serving it? Hold the second one; the next section is a postmortem of what happened when the wrong party answered.

The draft’s attention does not see the full conversation. Its context is a fixed ABI: the first 64 rows are pinned forever, and a trailing window slides over the rest:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/dflash/config.rs:59
/// The sink span used by the released DFlash cache ABI. It is made explicit
/// in every newly enrolled combined identity; it is not a receiver default.
pub const DFLASH_CONTEXT_SINK_SIZE: usize = 64;

// crates/muser-engine/src/dflash/config.rs:9
/// The context-cache shape bound to one enrolled DFlash identity.
///
/// The trained window comes from the sidecar metadata. The 64-row sink is
/// part of Muser's DFlash cache ABI rather than GGUF metadata, so enrollment
/// stamps it into both peers' identity configs explicitly. Receivers must
/// never infer either value from a local fallback.
pub struct DFlashContextGeometry {
    pub layers: usize,
    pub elements_per_token: usize,
    pub sink_size: usize,
    pub window_size: usize,
}
}

The sink-plus-window shape matters because the draft conditions on target-derived context rows. Figure 8.1 draws the ABI:

 DFlash context — one row per committed token, per layer (5 layers, f32)

 position:  0 …………………… 63 │ 64 …………………………………… (64 + W − 1)
            ┌───────────────┐ ┌────────────────────────────────┐
            │     SINK      │ │          WINDOW (W)            │
            │  pinned rows  │ │   trailing rows, slides as the │
            │  never evicted│ │   conversation grows           │
            └───────────────┘ └────────────────────────────────┘
             sink_size = 64       window_size = trained sliding_window
                                   (2,048 on the release sidecar)

 rows the ABI may ever hold, per layer: sink + window = 64 + 2,048 = 2,112
 buffered bytes (both planes, all layers):
   5 × 2 × 2,112 × 1,024 × 4 B = 86,507,520 B

Figure 8.1: The draft’s context cache. The sink is ABI (64, stamped at enrollment); the window comes from dflash.attention.sliding_window in the sidecar metadata. The byte bound is identity-derived, not a constant (DFlashContextGeometry::buffered_byte_limit, [crates/muser-engine/src/dflash/config.rs:42-56]).

The geometry declares how many buffered rows the ABI may ever hold, with that exact byte bound computed from the identity itself — for the release geometry (5 layers, 1,024 elements per token, sink 64, window 2,048) the arithmetic above yields 86,507,520 B, asserted by the config’s own test (config.rs:380-395).

That doc comment’s last line — “Receivers must never infer either value from a local fallback” — is fail-closed culture in one sentence: a remote handoff that stamped one geometry must never be served by a receiver that quietly assumed another. The same struct is bound into the cluster config alongside the draft’s SHA-256 identity ([crates/muser-cluster/src/config.rs:44-49]).

8.4 The window bug — read this twice

Where does window_size come from? The sidecar metadata (dflash.attention.sliding_window), and the code documents what happens when you don’t read it:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/dflash/config.rs:86
/// Trained sliding-window span for the draft's context attention, from
/// `dflash.attention.sliding_window`. The draft must be conditioned on
/// exactly this many trailing target rows: measured 2026-08-21, feeding
/// it half (the previously hardcoded 1024) or far more (32768) collapses
/// natural-text acceptance from 72.5% to 2.2%.
}

For an entire campaign, we never asked the artifact. Muser hardcoded sink 64 + window 1,024 and never read the key; the sidecar had been trained at 2,048. The draft was running on half its trained window, and nobody knew, because every instrument we owned said the lane was healthy.

That is the part to sit with. The synthetic fixtures kept passing. A period-8 synthetic stream is predictable from token identity alone, so a draft conditioned on the wrong stretch of history can still guess it perfectly; the fixture is structurally incapable of detecting a conditioning defect [ledger §ROOT CAUSE FOUND AND FIXED]. We had built a gate that could not fail for the one reason we most needed it to.

When we finally read the real key, the numbers moved in two directions at once. Natural-text acceptance on the python suffix-8,192 cell went from 1.1% to 72.7%, and natural-text cells became token-exact — the expected half of the result. The unexpected half: every synthetic spec number dropped by ~5%. A regression that is good news takes a moment to accept, and then it is the most convincing evidence in the postmortem. The draft had stopped coasting on a fixture and started doing real work [ledger §Spec re-measurement at the fixed window].

Two durable decisions came out of the postmortem, and both are still in the code. First, a sidecar without the key still loads, but loudly: resolve_sliding_window prints a warning that “draft conditioning may be wrong” rather than silently defaulting (config.rs:109-125). Second, the effective geometry now travels with every result — DFlashSpecStats carries draft_sink_size and draft_sliding_window “so every receipt self-identifies. Conditioning the draft on the wrong window silently invalidated a whole campaign’s spec numbers (2026-08-21)” ([crates/muser-engine/src/dflash/spec.rs:44-50]).

The general lesson is Chapter 5’s error discipline at systems scale: a draft that is conditioned wrongly does not crash and does not fail parity on easy fixtures — it just gets quietly worse at the one job it has.

8.5 Loading: one validated loader, two artifacts

The draft has to arrive from somewhere, and there are two somewheres: the artifact we ship and the artifact we debug against. The real question a loader like this answers is how to serve both honestly — how to keep the development export useful without ever letting it become the thing that ships.

DFlashWeights::load dispatches on what you point it at — a file is the GGUF sidecar, a directory is the SafeTensors development export ([crates/muser-engine/src/dflash/weights.rs:35-39]). Both paths share one config contract (DFlashConfig::from_artifact, config.rs:143-150), and the GGUF path is strict about metadata: the architecture must be dflash, the target-layer list arrives one-based (llama.cpp converter convention) and is converted to zero-based with rejection of invalid entries (config.rs:200-215), and the resolved dtype is recorded as "gguf-kquant" (config.rs:251).

The production Metal path loads a projection shell: norm vectors are expanded to f32, and nothing else is — every dense projection stays mmap’d in its kquant representation for the GPU to consume directly:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/dflash/weights.rs:42
/// Load the official assistant without expanding its projection matrices.
/// Norm vectors remain f32; all large matrices stay mmap'd in their GGUF
/// k-quant representation and are consumed directly by Metal.
}

Why “without expanding” is worth a doc comment: the f32 compatibility path costs real memory — “Expanding the official 1.5 GiB k-quant assistant into ~6 GiB of f32 matrices wastes startup time and resident memory” (weights.rs:55-59). Chapter 6’s block formats are not just a disk story; they are what makes a resident draft affordable. The SafeTensors path remains the development oracle — the CPU forward (dflash/forward.rs) is, like the target’s, the correctness spec the Metal drivers are checked against.

8.6 What the draft must guarantee for exact verification

Speculation is only lossless if the target makes every decision. The draft participates in exactness by guaranteeing four things.

1. It proposes; it never decides. Acceptance runs on the CPU against full target distributions: every proposed token is scored against the target’s complete probability row by verify_full_speculative_mt_ordered ([crates/muser-engine/src/sampling.rs:1033]). The rule is the Leviathan-style accept if rng ≤ min(p/q, 1), with a residual-corrected resample whenever a proposal is rejected (sampling.rs:1051-1088).

Every one of those coin flips is drawn from the source-pinned MT19937 stream — MT19937 being a reproducible random generator that Muser re-implements bit-for-bit to match llama.cpp’s, Ch 21. Keeping it isolated from the engine’s generic RNG is deliberate, so that “a rand algorithm or conversion change” cannot quietly alter which tokens come out (sampling.rs:1001-1007).

Now the payoff, and it is the load-bearing property of the entire chapter: a draft can be arbitrarily bad and the output distribution stays exact — badness only costs speed. Say it the other way round and it is stranger still: nothing the assistant does can make the answer wrong, only slow. That single property is why the window bug (§8.4) was a performance catastrophe and not a correctness one.

2. Its rounds are deterministic and replayable. Proposals are submitted “in deterministic round order. Qualification compares this trace exactly” (DFlashSpecStats.draft_token_trace, [crates/muser-engine/src/dflash/spec.rs:26-28]). The remote qualifier compares 256 greedy tokens plus every full target-logit row, with an acceptance floor of 0.95 (DFLASH_ACCEPTANCE_MINIMUM, [crates/muser-bench/src/remote.rs:3-8, :33]) — exactness as a gate, not a hope.

3. Its block size and verify lengths are pinned. draft_greedy builds a 16-token block (a seed token followed by 15 mask tokens), forwards it once, and takes the argmax of rows 1..=verify_length (the index of each row’s largest score); verify lengths are exactly 3, 7, or 15 — anything else is an error before any GPU work ([crates/muser-engine/src/dflash/spec.rs:730-741]). While tracing is enabled, the engine even checks the block head’s sanity: row 0 saw the real seed, and “a correctly conditioned block head must reconstruct the seed there; if it does not, the conditioning carries no usable signal” (spec.rs:753-757).

4. Its context writes are transactional. A speculation round may fail and roll back, so the target’s KV planes checkpoint before the block: MetalSpeculativeCheckpoint rewinds NoPE-plane metadata only, while SWA rings retain “the ≤16 rows a block may overwrite” ([crates/muser-engine/src/decode.rs:213-226]). Sixteen is not a magic number — it is the block size; a speculative block can touch at most one window’s worth of rows, and the checkpoint is sized to exactly that ABI.

And conditioning — the window story of §8.4 — is the fifth guarantee in disguise: not exactness-breaking, but throughput-breaking, and therefore gated (§8.7).

8.7 The measured scope — and its honest edges

Now the numbers — and the harder question of what each one is allowed to mean. Every figure below is true of a fixture, a depth, and a date. Almost none of them is true of “DFlash” full stop. Carrying the scope along is not pedantry in this chapter; it is the difference between a result and a press release, and the section you just read is the evidence for why we became strict about it.

The synthetic matrix (current, post-window-fix). “In retained fixed-window synthetic packets, exact-token decode ratios (llama/Muser means) are 1.23692× at 2,048, 1.20323× at 16,384, and 1.19616× at 32,768, with 5/5 exact reps per depth” [claims #15]. The runs behind that row are retained rather than summarized — muser-receipt://spec-prefill-fix-20260822/aggregate-a2/… and …/respec2-deep-20260822/aggregate-a1/… — so anyone who doubts the means can go and recount them. The claims row’s own instruction is part of the claim: “Never generalize this to natural text, native NVFP4, or untested depths.” The deeper cells of the same family: exact-token in 5/5 reps at all four depths including 131,008 [claims #3], and the funded-fix 131,008/48 packet crossed end-to-end wall parity for the first time at 1.02536× [claims #16].

The old bar, and why you must not use it as a result. The campaign’s famous 107.9 tok/s (107.9136 median, ratio 1.3273 vs llama’s 81.3047) was measured before the 2026-08-21 draft-window fix; it survives in the record only as the kquant spec bar that later lanes were judged against [ledger §L2 Stage B verdict]. The pre-fix ratios 1.3273/1.3012 are superseded numbers — quoting them as current performance is one of this book’s standing landmines.

Natural text is a different regime. We expected the window fix to make the draft uniformly better. What it actually did was make the draft legible: once conditioning was right, the wins and losses stopped looking like noise and sorted themselves by the kind of text being written. On real corpora, cross-engine outputs diverge (so speed stands without an exactness gate), and the picture splits: spec decode wins python-like content (16,384: 1.186; 8,192 suffix: 1.321) and loses high-acceptance shallow text (rust at 2,048: 0.931, improving only to 0.945 at verify-length 7) — llama’s lighter draft wins there [docs/benchmarks.md §2]. That asymmetry froze the serving verify-length at 7 while the comparison harness pins 15: “the best decode and the most robust acceptance on natural text” [docs/benchmarks.md §2].

The engine distrusts its own draft. One question is left: what should the engine do when the draft stops paying for itself on a particular request? The obvious answer is to track acceptance across the request and switch drafting off once the rate falls too low. We built exactly that, and it carried a defect that only surfaces in production — a cumulative average that has sunk can never climb back, because once drafting stops there are no new proposals to lift it. Disabling speculation was therefore permanent. The comment on the redesign still records that one-way latch, dated like a scar: “a cumulative rate cannot recover once drafting stops… (2026-08-21 root cause)” (spec.rs:184-190).

So the gate that shipped reads only recent evidence: eight rounds, after a two-round warmup, because “the rounds immediately after prefill are the coldest of the request”. It requires ≥32 proposals before it will judge anything at all, and it closes when windowed acceptance drops below 0.25 ([crates/muser-engine/src/dflash/spec.rs:117-133]). The latch is undone in the other direction too: a disabled request re-qualifies after a doubling cooldown (64 → 512 tokens), so a cold start costs a session some throughput instead of costing it speculation for the rest of its life (spec.rs:199-230).

8.8 Tradeoffs

Draft cost vs verify cost, measured. Where should optimization energy go — into making the guess faster, or into making the guess right? The question sounds like a matter of taste until you look at the split, and then it answers itself.

In the L1 in-process qualification (five reps × 256 tokens, verify length 15): median cycle ~157.2 ms, of which draft ≈ 26.9 ms (embed 0.04 + block forward 22.3

• LM-head/argmax 4.5) and verify ≈ 130.2 ms (forward 128.4, decision 1.8) [ledger §Stage B L1]. The draft is the small side of its own loop — which is why Chapter 6’s optimization energy went to the 16-row verify kernels (the n32 tile: verify forward 202.3 → 128.4 ms), and why making the draft correctly conditioned (§8.4) was worth more than making it faster. A draft that proposes 15 tokens per 27 ms is already cheap; a draft whose proposals are accepted is priceless.

Synthetic vs natural — the fixture that lied. The period-8 synthetic stream “certified a broken draft lane for an entire campaign” [ledger §ROOT CAUSE FOUND AND FIXED, consequence 2]. Since 2026-08-21, natural-text cells are a standing part of the matrix even though they cannot carry an exactness gate. The general rule this book keeps re-stating: a measurement is only as good as its fixture’s ability to detect the failure mode you care about.

Why the draft is kquant-only (Fallback B, again). Chapter 7 measured the alternative: native NVFP4 batched verification at 6.805 tok/s against the 107.9 bar [claims #4]. The draft itself could be anything — the barrier is the target’s verify arithmetic in the NVFP4 lane’s W4A4 batch shape. So the smallest model in the system inherits the reference lane’s format: Q4_K/Q6_K blocks, llama-pinned batch kernels, exactness gated by lossless token equality. When your only exactness instrument is bitwise comparison, you build on the lane that can support it.

8.9 What comes next — two hooks, one promise

The accept/verify algorithm itself — the min(p/q, 1) rule applied to full distributions, the carried-frontier state machine, the Mirror-SD overlap that splits the target graph at a capture layer, and the measured rejection of the distributed verifier — is Ch 33’s subject; this chapter deliberately stopped at the draft’s edge of the contract.

But Part II now closes, and Part III opens with a debt we have been accumulating for four chapters: we have quantized, packed, and dispatched the weights of a model we have never actually met. What is the 52-layer graph these formats encode? Which layers slide, which layers have no position at all, why are there two KV heads for thirty-two query heads, and why does a sigmoid gate sit on the attention output? Ch 9 is the Muse Glimmer architecture — the model that all three lanes exist to serve.

References

• [crates/muser-engine/src/dflash.rs:1-5] — the module contract (five layers, SafeTensors + kquant GGUF, Ferrite oracle lineage).
• [crates/muser-engine/src/dflash/config.rs:59-61] — DFLASH_CONTEXT_SINK_SIZE = 64; :15-57 DFlashContextGeometry and the exact byte bound; :86-91 the window-collapse doc (72.5% → 2.2%); :109-136 the loud fallback; :180-255 the GGUF config path (one-based target layers, gguf-kquant); :263-268 the five-layer enforcement.
• [crates/muser-engine/src/dflash/weights.rs:35-66] — the dual loader and the projection shell; :115-161 validate_quantized_gguf_layouts (Q4_K/Q5_K/Q6_K requirement, exact shapes).
• [crates/muser-engine/src/dflash/hidden.rs:44-52] — target-layer-gated hidden-state capture.
• [crates/muser-engine/src/dflash/spec.rs:16-98] — DFlashSpecStats (trace, geometry self-identification); :117-134 the disable-gate constants; :184-230 windowed gate + re-qualification; :730-770 draft_greedy (verify lengths 3|7|15, seed echo).
• [crates/muser-engine/src/sampling.rs:1001-1097] — the MT-pinned stream and verify_full_speculative_mt_ordered.
• [crates/muser-engine/src/decode.rs:213-226] — MetalSpeculativeCheckpoint (≤16 SWA rows, NoPE metadata rewind).
• [crates/muser-cluster/src/config.rs:44-49] — enrollment-stamped dflash_context_geometry bound to the component digest.
• [crates/muser-bench/src/m16.rs:202-225] — draft shapes (fc 33280→6656, k/v →1024) and dtypes.
• [crates/muser-bench/src/remote.rs:3-8, :33] — the 256-token exact compare and DFLASH_ACCEPTANCE_MINIMUM = 0.95.
• [docs/memory-footprint.md] — DFlash GGUF 1,631,205,312 B.
• [claims #15], [claims #3], [claims #16], [claims #4] — docs/launch-claims.md: the fixed-window synthetic ratios and their prohibited generalizations; exact-token 5/5 depths; 131,008 wall parity 1.02536; native spec fail-closed.
• [docs/benchmarks.md §2] — verify-length conventions and the natural-text wins/losses (1.186/1.321 vs 0.931/0.945).
• [ledger §L2 Stage B verdict], [ledger §Stage B L0/L1], [ledger §ROOT CAUSE FOUND AND FIXED], [ledger §Spec re-measurement at the fixed window] — docs/goal-parity-ledger-2026-08.md: the pre-fix 107.9136/1.3273 bar, the M16 microbenchmark lineage, and the window postmortem.
• Ch 6 — the blocks the draft is built from and the M16 kernels its 16-row blocks run on.
• Ch 7 — Fallback B and the 6.805 no-go.
• Ch 33 — the accept/verify algorithm this chapter deferred.

Chapter 9 — The Muse Glimmer architecture

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Part I (Metal) and Part II (quantization). No transformer paper assumed; every term is defined in place, and the kernel-deep treatment starts in Ch 11.

Chapter 8 left us with a small mystery: the DFlash draft was described purely by what it must guarantee — a 64-row sink context, a matching sliding window, exact-enough drafting. What it drafts for stayed off-stage. Now the target takes the stage: Muse Glimmer, the 52-layer, ~30-billion-parameter model every lane in this book exists to serve. We build its architecture from zero — no transformer knowledge assumed — and verify every number against the pinned Muser tree, because this chapter’s tables are the anchors every kernel chapter in Part IV points back at.

9.1 What a transformer is (one page, from zero)

A transformer turns a sequence of tokens (integers, each standing for a word fragment) into a prediction of the next token. Internally it is a stack of identical blocks — layers — wired one after another. Each layer reads one vector per token, the hidden state (or residual stream), does two things to it, and writes the updated vector back out:

• Attention mixes information across tokens: each token looks at the tokens before it and pulls in what it needs. It is the only operation in the model that moves information between positions.
• A feed-forward network (FFN) transforms the vector in place, one token at a time — a per-token “thinking” step.

Everything else is plumbing: normalizations that keep the vector’s magnitude tame, projections that reshape it, positional machinery that tells attention what “before” means. Muse Glimmer is exactly this stack, 52 layers deep, with a handful of unusual choices in the plumbing — the repeating sliding/full attention pattern, a sigmoid gate on attention output, a dual-epsilon norm scheme — each taken one at a time below. The name “transformer” is just the 2017 paper’s label for the attention-plus-FFN stack [arxiv:1706.03762]. Muser implements one specific transformer, and the contract is one sentence in the code — the CPU oracle is “the bit-level spec for every trap in the architecture (inverted-Gemma RoPE/NoPE split, sigmoid gate placement, dual-eps sandwich norms, softcap-after-scale ordering, GPT-J RoPE pairing)” [crates/muser-engine/src/lib.rs:103-108]. Every trap in that sentence gets a section.

9.2 The exact hyperparameters (verified from source)

Before a single kernel can be written, one question has to be settled: how wide is this model, exactly? Every kernel chapter in Part IV sizes its buffers, its threadgroups and its loop bounds from the answer, so one wrong row here does not fail loudly — it propagates quietly into a dozen chapters and into an engine that runs and emits plausible text.

So we did not transcribe these values from a model card. Each one is read twice: once by the live GGUF metadata reader, which fails closed on a missing key, and once by the release-model gate test, which asserts what the pinned artifact actually carries. That is the same double-citation discipline we apply to the pinned SHA-256 identity, and we kept the receipt: [crates/muser-engine/tests/muse_golden.rs:14-15].

Table 9.1 — Muse Glimmer hyperparameters (pinned release GGUF)

Three properties of this table matter more than the numbers. First, nothing is hard-coded: MuseConfig::from_gguf “fails closed” on any missing key, bad shape, or non-constant QK-norm [config.rs:158-162]; the Rust constants that exist (MUSE_LAYER_COUNT, MUSE_SWA_WINDOW, …) are the same values as asserts, not a second source of truth [config.rs:13-21]. Second, head_dim is independent of hidden_dim: with 32 heads over a 6,656-wide hidden state, hidden/n_heads would be 208, but the GGUF declares key_length = 128 and the loader prefers the declared key [config.rs:174-176] — so query space is 32 × 128 = 4,096 while the residual stream is 6,656, and the Q projection is not square. Third, one value is famously not in the file: the 1e-8 post-norm epsilon is a llama.cpp graph constant, not GGUF metadata — §9.9 explains why that is a landmine.

Two rows use names you have not met yet: SWA, sliding-window attention, and NoPE, “no positional embedding.” They earn their own section rather than a parenthesis, because they are the reason this model’s memory cost does not grow the way its depth suggests — that is §9.7, and the rope_base_swa key belongs to the same story. And one last thing before we leave the table: it describes one specific file and no other — the artifact with SHA-256 7e9b74b7c8875e9e265695df9613bf6290f2392e479ce740495a129019c488d8, 16,756,681,056 bytes on disk [crates/muser-server/src/chat_template.rs:237-250]. Every value above was read out of that artifact; held against a different one, the whole geometry is a guess.

Derived quantities used everywhere below: attn_dim = n_heads × head_dim = 4,096; kv_dim = n_kv_heads × head_dim = 256 [config.rs:268-273]. Q, gate, o_proj live in 4,096 space; K, V in 256.

9.3 One Muse Glimmer block — the anatomy

Figure 9.1 is the most important diagram in Part III — every decode kernel in chapters 11–21 implements one box in it. Read it slowly, then again after §9.5.

flowchart TD
    hin["residual in [6,656]"]
    rn1[" RMSNorm (attn_norm), eps 1e-5 — Ch 12 "]
    subgraph ATT["Attention"]
        direction TB
        qkv["Q/K/V/gate projections — one concurrent group of 4 matvecs<br/>Q, gate: [6656→4096]  K, V: [6656→256] — Ch 13"]
        qkn["per-head QK-norm, eps 1e-5<br/>Q weight ≈ 3.87, K weight 1.0 — Ch 14"]
        rope["RoPE — ONLY on sliding layers<br/>theta 500,000, interleaved pairs — Ch 14"]
        attn["attention over the KV cache: scale 1/√128, causal,<br/>GQA 32:2; SWA layers see last 2,048 tokens — Ch 16"]
        sgop["sigmoid gate: attn_out ⊙ σ(gate_proj),<br/>then o_proj [4096→6656] — Ch 17"]
        qkv --> qkn --> rope --> attn --> sgop
    end
    pn1[" RMSNorm (post_attention_norm)<br/>eps 1e-8 — sandwich norm "]
    add1((" + residual "))
    rn2[" RMSNorm (ffn_norm), eps 1e-5 "]
    subgraph FFN["Feed-forward — SwiGLU — Ch 18"]
        ffn["gate [6656→19968], up [6656→19968]<br/>→ SiLU(gate) ⊙ up → down [19968→6656]"]
    end
    pn2[" RMSNorm (post_ffw_norm)<br/>eps 1e-8 — second sandwich norm "]
    add2((" + residual "))
    hout["residual out [6,656] → next layer"]
    hin --> rn1 --> ATT
    hin -. residual .-> add1
    ATT --> pn1 --> add1
    add1 --> rn2 --> FFN
    add1 -. residual .-> add2
    FFN --> pn2 --> add2
    add2 --> hout

Figure 9.1: One Muse Glimmer block. Two residual adds (dashed), and — the signature of the architecture — a norm on the output of each sub-block before it is added rather than on the residual sum. This Gemma-2-style “sandwich” placement [crates/muser-engine/src/config.rs:51-53] gives both post-norms the different 1e-8 epsilon (§9.9), and the sigmoid gate (§9.8) sits between attention and o_proj — nowhere else.

Three things to internalize. The residual adds are the spine: each sub-block only ever adds — nothing overwrites the stream, which is what lets a 52-layer stack train at all (the skip-connection argument of [arxiv:1512.03385]). Attention is the only cross-token mixer; the FFN and all the norms are strictly per-position. And every sub-block reads the stream through a norm — four weighted norms per block plus one per attention head, 209 norm applications per token. Norms are cheap per vector but everywhere at 52 layers, which is why the fused norm tails of Ch 12 exist.

9.4 The full model — 52 blocks in a repeating pattern

Now zoom out one level. If the figure above is the cell, the one below is the organism: everything that happens to a token id between the moment it arrives and the moment a successor is chosen. The question this section answers is where the blocks sit in that path, and what decides which kind of block a given layer is.

flowchart TD
    tok["token_ids (u32)"]
    emb["token_embd.weight [6656 × 202048]<br/>row lookup — Ch 11 (untied from LM head, §9.11)"]
    entry["entry RMSNorm — weightless<br/>(weight = all ones), eps 1e-5"]
    blocks[" 52 blocks ×  Figure 9.1<br/>collar: [SWA, SWA, SWA, FULL] × 13<br/>(full layers at indices 3, 7, …, 51) "]
    fnorm[" final RMSNorm (output_norm), eps 1e-5 —<br/>fused into the last block's tail "]
    lmh[" LM head: output.weight [6656 × 202048] — Ch 20 "]
    tail[" × 0.196116 (1/√26), then tanh cap at 20 — Ch 20 "]
    samp[" sample / argmax — Ch 21 → next token id "]
    tok --> emb --> entry --> blocks --> fnorm --> lmh --> tail --> samp

Figure 9.2: The full Muse Glimmer forward pass. The residual stream is [6,656] between all blocks; the final norm is not a separate dispatch on the decode path — the last block’s fused tail produces the normed output directly [crates/muser-engine/src/decode.rs:5869-5876].

The collar in Figure 9.2 follows one rule, and the type documenting it also records the inversion relative to Gemma 3:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/config.rs:51-59, 84-93  (the enum's variant bodies
//   and the rule's bounds-check arm are elided)
/// Per-layer attention kind. Muse Glimmer alternates
/// `[sliding, sliding, sliding, full]`, and — unlike Gemma 3 — it is the
/// *sliding* layers that carry RoPE while the *full* layers are NoPE.
pub enum MuseLayerKind { /* SlidingRope, FullNoPe */ }

pub const fn layer_kind(layer: usize) -> Result<MuseLayerKind, LayerIndexError> {
    // …
    if layer % 4 == 3 {
        Ok(MuseLayerKind::FullNoPe)
    } else {
        Ok(MuseLayerKind::SlidingRope)
    }
}
}

So 52 layers = 13 groups of [sliding, sliding, sliding, full]: 39 SWA layers and 13 NoPE full layers at indices 3, 7, …, 51 — exactly the set the remote-prefill schedule ships as position-free tiles [crates/muser-cluster/src/schedule.rs:20]. The counts are asserted in code [config.rs:14-15] and in the release test [muse_golden.rs:109-116].

There was a fork at the resolver, and it is worth naming, because the tempting branch is the friendly one. If a GGUF arrives without its sliding-window pattern key, the loader could simply assume every layer is a full-attention layer. The model would load. It would run. It would emit fluent English. We took the unfriendly branch instead: a missing pattern key panics. The comment in the resolver gives the reason in one line — “an all-full model runs and emits plausible text while being wrong” [config.rs:378-380]. That is the fail-closed posture in miniature, applied here not to a tensor or a hash but to the shape of the graph itself, and it rests on a judgement we will keep returning to: an answer that looks right and is wrong costs more than a refusal.

9.5 The components, one short tour each

Before the traps, a quick pass over the parts by name. Nothing in this section is peculiar to Muse Glimmer; it is the shared vocabulary the rest of the book leans on, so skim what you already know and slow down where a term is new.

The embedding is a lookup table — a token id picks one row of token_embd.weight [6656 × 202048]; that row is the initial residual stream. RMSNorm computes x / sqrt(mean(xᵢ²) + ε) ⊙ γ (⊙ is elementwise multiply) with a learned per-channel scale γ (here ε = 1e-5); no mean subtraction, and it rescales the stream to ~unit magnitude so 52 layers of matrix products neither explode nor vanish. In attention, each token computes a Query (what it looks for) while every past token carries a Key (what it offers) and a Value (what it hands over); a query-key score is a dot product over 128 dimensions scaled by 1/√128 ≈ 0.0884 [config.rs:279-281], scores pass through softmax (exp(xᵢ)/Σⱼexp(xⱼ) — positive weights summing to 1), and the output is the weighted sum of Values under a causal mask that forbids seeing future tokens. The KV cache records every past token’s Keys and Values so token N+1 does not recompute tokens 0..N — one cached row in one layer costs 2 KV heads × 128 × 2 B × (K + V) = 1,024 bytes [docs/memory-footprint.md §KV formula]. The FFN’s activation SiLU is x·σ(x) (⊙ is elementwise product), giving the SwiGLU shape down(SiLU(gate·x) ⊙ (up·x)) [crates/muser-engine/src/reference.rs:493-522].

9.6 GQA — 32 query heads, 2 KV heads

The single biggest memory lever in the model’s geometry. Attention runs per head — each an independent 128-dimensional attention with its own slice of the Q projection. In plain multi-head attention every query head would also have its own Key and Value (32 KV heads here); Grouped-Query Attention instead gives the 32 query heads only 2 KV heads to share (Figure 9.3):

query heads:  Q0  Q1 ... Q15   |   Q16 ... Q31        32 heads × 128 = attn_dim 4,096
                 \   ...   /         \   ...   /
KV heads:         K0 , V0               K1 , V1       2 heads  × 128 = kv_dim    256

Figure 9.3: The GQA grouping. Each KV head serves heads_per_kv = n_heads / n_kv_heads = 32 / 2 = 16 query heads [config.rs:274-276].

Said the other way round, because this is the idea the whole memory story rests on: the model keeps thirty-two independent ways of asking a question, but only two distinct sets of answers on file, and every question is served from one of those two sets. Asking is cheap — it is arithmetic on a vector the model already has. Answering is expensive, because the answers have to be fetched from memory. Grouped-Query Attention makes the expensive half small.

Why this is a bandwidth win. During decode, attention must read the K and V of every visible past token out of the KV cache. With full multi-head attention one cached row in one layer would cost 32 × 128 × 2 B × (K+V) = 16,384 B; with GQA it costs 2 × 128 × 2 B × (K+V) = 1,024 B [docs/memory-footprint.md §KV formula] — a 16× reduction in both the per-token KV read and the cache size. On a bandwidth-bound decode path (the argument of Ch 1) that is enormous; the quality cost of 16 query heads sharing each KV pair is the trade GQA was invented to make [arxiv:2305.13245]. Whenever a later chapter says “KV head,” it means one of these 2. GQA also skews the projection widths — K and V read only [6656 → 256], skinny matrices that get a different matvec geometry than Q in Ch 13 and a 16-way fan-in per KV head in attention Ch 16.

9.7 The two attention classes: 39 SWA rings, 13 NoPE planes

This is the section to read twice. Almost everything Part VI does — remote prefill, warm reuse, the delta handoff — is downstream of a single fact established here, and the fact is easy to miss because it hides inside a routine-looking layer schedule. The question: what does a layer of this model see when it looks backwards, and does the answer depend on where the tokens it sees happened to sit?

Muse Glimmer’s most consequential structural choice: its 52 layers do not all attend the same way (Figure 9.4).

layer:   0  1  2  3 | 4  5  6  7 | 8  9 10 11 | ... | 48 49 50 51
kind:    S  S  S  F | S  S  S  F | S  S  S  F | ... |  S  S  S  F

S = SlidingRope : window 2,048 tokens, RoPE applied      (39 layers)
F = FullNoPe    : full causal attention, NO rotation     (13 layers)

Figure 9.4: The repeating collar — 13 copies of [sliding, sliding, sliding, full] (layer % 4 == 3 is full [config.rs:88]).

The 39 sliding layers use sliding-window attention: a query at position p1 sees a key at p0 only if p1 ≥ p0 and p1 − p0 < 2,048 — “Mask follows llama.cpp LLAMA_SWA_TYPE_STANDARD” [crates/muser-engine/src/reference.rs:577-580]. Their KV caches never need more than 2,048 rows, so each SWA layer’s cache is a fixed 2,048-row ring of min(max_context, sliding_window) capacity, token-major [crates/muser-engine/src/decode.rs:1346-1348]. Bounded KV: one SWA layer holds 2,048 × 1,024 B = 2 MiB — per layer, forever, at any depth.

The 13 full layers attend over the entire context — and they carry no positional rotation at all: NoPE means “no positional embedding,” not “a different positional embedding.” The code is categorical:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/config.rs:66-70
/// RoPE runs iff the layer is a sliding layer (`muse-glimmer.cpp:93`:
/// `const bool use_rope = hparams.is_swa(il);`).
pub const fn uses_rope(self) -> bool {
    matches!(self, Self::SlidingRope)
}
}

Because a NoPE layer’s K/V rows do not encode absolute position, its cache rows are relocatable bytes — a KV tile for positions [a, b) can be planted anywhere in the plane without recomputation: “the 13 NoPE layers are position-free (relocate = memcpy) — the whole kvpack free lunch” [crates/muser-engine/src/lib.rs:8-10].

Put it side by side with the other class, because the contrast is the whole point. In a sliding layer, where a token sat has been rotated into its numbers; move its cached row to a new position and the numbers are now lies, so the row must be recomputed. In a NoPE layer nothing about position was ever written down; the row means the same thing wherever it lands, so moving it is a copy and nothing more. One class stores meaning-at-a-place, the other stores meaning. Only the second can be shipped over a wire.

That single property is what pays for Part VI. It is why remote prefill can stream 512-token NoPE tiles while the producer is still computing, why warm reuse restores a 130,815-token prefix in ~1 s instead of ~148 s [ledger, kvpack ladder stage-5], and why that lane exists in the shape it does. (Why the authors trained full layers without rotation is a modeling question this codebase does not answer — [unverified]; the code proves the rotation is absent and the engine exploits that.)

Meanwhile the sliding layers do rotate — with the interleaved (GPT-J) pairing: dimension 2i rotates with 2i+1, adjacent in memory:

// crates/muser-engine/src/shaders/ferrite/rope.metal:651-656
// NORM convention: the pair is adjacent, at 2*pi and 2*pi + 1.
const uint i0 = 2u * pi;
// …(v0/v1 loads elided)
base[i0]      = v0 * cos_a - v1 * sin_a;
base[i0 + 1u] = v0 * sin_a + v1 * cos_a;

The base frequency is theta = 500,000 (read from muse-glimmer.rope.freq_base_swa [config.rs:200-202]; baked into the deterministic trigonometry tables as “Exact f32 bits for 500000^(-2/128)” [crates/muser-engine/src/rope_nco.rs:14]). Larger theta ⇒ angles grow more slowly with position ⇒ positions stay distinguishable over longer ranges (the mechanical trade is Ch 14’s). The two-class split also fixes the cache arithmetic for Ch 22, derived here once:

slot_kv_bytes(C) = (39 × min(C, 2,048) + 13 × C) × 1,024

C = 131,072:  (39 × 2,048 + 13 × 131,072) × 1,024
            = (79,872 + 1,703,936) × 1,024
            = 1,826,619,392 B ≈ 1.827 GB per slot   [docs/memory-footprint.md]

The 13 NoPE layers dominate that sum (1,703,936 of 1,783,808 rows ≈ 95 %): full attention pays for depth, sliding attention does not.

9.8 The sigmoid attention gate

Two questions hang over this component, and only one of them has an answer in this repository. Where does the gate sit, exactly? — answerable to the line, and the answer matters, because getting the placement wrong changes every logit downstream. Why is it there at all? — not answerable from code, and we will say so plainly rather than fill the gap with a plausible story.

Take the placement first. Standard attention ends with o_proj; Muse Glimmer slips a learned gate in ahead of it, and the CPU oracle shows the seam exactly — after attention, before the output projection:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/reference.rs:446-455  (the trailing o_proj
//   `matmul(&self.w("blk.{il}.attn_output.weight"), &attn_out, t, &mut proj)`
//   at :456-461 is elided)
// ── sigmoid gate, then o_proj ─────────────────────────────────
for g in gate.iter_mut() {
    *g = 1.0 / (1.0 + (-*g).exp());
}
// …(capture recording elided)
for (a, g) in attn_out.iter_mut().zip(gate.iter()) {
    *a *= *g;
}
}

The gate vector comes from a fourth attention projection — attn_gate.weight [6656 → 4096], the same width as Q [config.rs:307] — computed concurrently with Q/K/V from the same normed input [crates/muser-engine/src/decode.rs:5569-5598]. After the sigmoid, each of the 4,096 attention-output channels is multiplied by a learned value in (0, 1): a per-channel valve that can pass, dampen, or effectively mute what attention just computed; then o_proj maps the gated result back to 6,656 (reference.rs:456-461). The Metal kernel:

// crates/muser-engine/src/shaders/ferrite/sigmoid_gate.metal:7-16
kernel void sigmoid_gate_inplace(
    device       float* attn_out [[ buffer(0) ]],
    device const float* gate     [[ buffer(1) ]],
    constant     uint&  n        [[ buffer(2) ]],
    uint gid [[ thread_position_in_grid ]])
{
    if (gid < n) {
        attn_out[gid] *= 1.0f / (1.0f + exp(-gate[gid]));
    }
}

What the gate buys the model — dampening unreliable attention channels — is the standard reading of gated-attention variants, but this tree contains no evidence for the authors’ intent: [unverified]. What the code proves is the placement (attention output, pre-o_proj), the width (4,096, matching Q), and the concurrency (the fourth matvec of the QKVG group — the hero chapter Ch 13; kernel in Ch 17).

9.9 The dual-epsilon norm sandwich

Look again at Figure 9.1: the post-attention norm applies to o_proj’s output, then the result joins the residual — ditto the post-FFN norm. This Gemma-2-style sandwich normalizes each sub-block’s contribution on its way into the stream, not the stream after the add. The oracle’s tail:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/reference.rs:466-480
// ── post-attention norm (eps 1e-8) + residual ─────────────────
// …(weight fetch elided)
rms_norm_mul(&mut proj, h_dim, cfg.post_norm_eps, &post_attn_w);
// …(capture recording elided)
for (hv, (p, r)) in hidden.iter_mut().zip(proj.iter().zip(residual.iter())) {
    *hv = *p + *r;
}
}

The second tail, after the FFN, is identical in shape with post_ffw_norm [reference.rs:527-538]. Now the landmine: the two post-norms use a different epsilon than every other norm in the graph — 1e-8 versus the GGUF’s 1e-5 — and that 1e-8 is not in the checkpoint:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/config.rs:23-28
/// Post-attention / post-FFN RMSNorm epsilon.
///
/// llama.cpp uses a *different* epsilon for the two "post" norms of the
/// Gemma-2-style sandwich than for every other RMSNorm in the graph, and it is
/// not carried in the GGUF. See `src/models/muse-glimmer.cpp:67`.
pub const MUSE_POST_NORM_EPS: f32 = 1e-8;
}

This is a llama.cpp graph constant, transcribed into Muser so the two engines compute identical bytes [crates/muser-engine/src/lib.rs:103-108]. Reading one epsilon from the GGUF and using it everywhere — the “obvious” simplification — would produce logits that differ from the comparator’s at every layer boundary.

We found out how little slack there is here the expensive way. A block’s tail is two norm stages with a device-memory boundary between them, which reads like an open invitation to fuse: same data, same thread, one kernel instead of two, and the algebra on paper does not change. So we built it — a hybrid schedule that fused the tail and retained activations across the two stages — and we expected a dispatch saving for free, on the reasoning that arithmetic which is equal on paper is equal in the machine. It was not. Public logprobs moved past the 1e-4 contract: max normalized-logprob error 3.197e-4, with the first divergence a single f16 ULP in layer-1 V, which then had the entire rest of the stack to grow in. The lesson is the one to carry out of this section: an epsilon this small is not a tolerance knob, it is part of the model’s definition — rounding boundaries are the spec, and “the same algebra” is not “the same bits.” The hybrid was rejected rather than waved through under a widened tolerance, and we retained the postmortem: [docs/decode-dispatch-gap-20260815.md §Rejected hybrid postmortem].

The full norm story — including why the fused tail kernel that did ship reproduces “the two pinned ggml f32x4 norm reductions and their intervening f32 device-memory boundary” [crates/muser-engine/src/decode.rs:1328-1330] — is Ch 12.

A related fail-closed check lives at load time: the GGUF’s attn_q_norm/attn_k_norm tensors are converter-synthesized constant broadcasts — every Q-norm weight the same scalar (qk_scale_factor ≈ 3.87), every K-norm weight 1.0 — materialized only so llama.cpp’s weighted-RMSNorm op can carry a scalar [crates/muser-engine/src/lib.rs:74-79]. The loader verifies this tensor-by-tensor and aborts on a genuinely learned norm [crates/muser-engine/src/loader.rs:98-138]. The per-head QK-norm itself (RMS over each 128-wide head vector, eps 1e-5, before RoPE) is Ch 14’s; its measured constant 3.87 × 0.0883883… ≈ 0.3420623 is asserted in [config.rs:426-430].

9.10 Final logits: multiply by 1/√26, then cap at tanh(20)

One question is left before a token can be chosen: what happens to the raw scores the LM head produces? Muse Glimmer does two unusual things to them, and the part that catches people is not either step on its own — it is that their order is contract, not taste:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/reference.rs:559-570
let mut logits = vec![0.0f32; t * cfg.vocab_size];
matmul(&self.w("output.weight"), &hidden, t, &mut logits);
for l in logits.iter_mut() {
    *l *= cfg.logit_scale;
}
if cfg.final_logit_softcap > 0.0 {
    let cap_v = cfg.final_logit_softcap;
    let inv = 1.0 / cap_v;
    for l in logits.iter_mut() {
        *l = cap_v * (*l * inv).tanh();
    }
}
}

First, every logit is scaled by logit_scale = 0.196116 — 1/√26 to five decimals (1/√26 = 0.19611613…). This is GGUF metadata, not a code constant: MuseConfig requires the key muse-glimmer.logit_scale [config.rs:190-192], and the pinned artifact’s value is recorded in the provenance doc (“…muse-glimmer.final_logit_softcapping=20 and logit_scale=0.196116” [docs/release-provenance.md:823]). The literal 1.0 / 26.0f32.sqrt() appears in the tree only as a fixed test input [crates/muser-engine/src/metal.rs:68] — cite the metadata read, not the test. Second, a soft cap l → 20·tanh(l/20) confines every logit to (−20, 20) without a cliff; the GPU kernel is one thread per logit:

// crates/muser-engine/src/shaders/muse_reference.metal:15-27  (the buffer
//   parameter list is elided)
kernel void muser_scale_softcap_inplace( /* … */ ) {
    if (index < count) {
        float value = logits[index] * scale;
        logits[index] = softcap > 0.0f ? softcap * tanh(value / softcap) : value;
    }
}

Scale first, cap second — “softcap-after-scale ordering” is one of the named traps the oracle pins down [crates/muser-engine/src/lib.rs:103-108]. Backwards, the cap barely bites (0.196 × 20 ≈ 3.9 effective ceiling instead of 20). The cap also changes what “comparing logits across engines” means — differences near the ceiling are compressed non-linearly, one reason the parity campaign compares tokens and logprobs, not raw logit deltas, at the tail [docs/muser-architecture.md §Model and engine; deep treatment in Ch 20].

9.11 The per-layer tensor inventory

What does one layer actually consist of, as bytes on disk? The table below is the answer, and it is also a promise the loader keeps: the shape checker asserts that every tensor listed exists with exactly this shape, in every layer, and fails closed if one is missing or misshapen [crates/muser-engine/src/config.rs:286-333]. Read the shapes the way GGUF writes them, [in, out].

Table 9.2 — Per-layer tensors (×52, l = 0..51)

Three global tensors sit outside the loop [config.rs:294-298]: token_embd.weight and output.weight, both [6656, 202048], and output_norm.weight [6656]. The embeddings are untied: output.weight (the LM head) is a required tensor in the shape contract, not an optional one — the model carries two independent [6656 × 202048] matrices, one to enter and one to leave. Whether untying buys quality is a modeling claim this tree cannot support ([unverified]); the cost is countable: both matrices are 2.69 B of the 27.85 B parameters (§9.12).

9.12 Parameter accounting — what “~30B” actually is

The model is sold as a thirty-billion-parameter model. Is it? Everything needed to check now sits in the tables above, so rather than quote the label, we count — and the counting is worth doing slowly, because the shape of the total tells the kernel chapters where to spend their effort.

Deriving the count from Table 9.2: per layer, the five attention projections sum to 3 × (6656×4096) + 2 × (6656×256) = 85,196,800, the FFN adds 3 × (6656×19968) = 398,721,024, the norms 26,880 — 483,944,704 per layer × 52 = 25,165,124,608. The global tables add 2 × (6656×202048) = 2,689,662,976 plus the output norm, for ≈ 27,854,794,240 ≈ 27.85 B parameters. The repo labels the model “Muse Glimmer-30B” [crates/muser-engine/src/lib.rs:1]; the honest arithmetic gives ≈ 27.85 B — same class, not the same number, and this book uses the derived one.

The distribution matters more than the total: 82 % of each layer’s parameters are FFN, the attention projections 18 %, the two untied vocab matrices 9.7 % — decode bandwidth is dominated by the FFN matvecs (Ch 18), and the DFlash verify path of Ch 33 is a batching story. One more division for Ch 1’s roofline: the artifact is 16,756,681,056 bytes [muse_golden.rs:15], so the average weight occupies 16,756,681,056 / 27,854,794,240 ≈ 0.60 B ≈ 4.81 bits — the q4_k_xl mix of Ch 6 slightly above pure Q4_K’s ~4.5 bits.

9.13 Tokenizer and template identity

Geometry is not the whole of a model’s identity. Something with exactly the right shape, that reads text with a slightly different tokenizer or wraps it in a slightly different chat template, is a different model in every way a user would notice — same weights, different answers. So Muser serves one model with one tokenizer and one template, binding the whole surface by hash at load time:

• Tokenizer — a GGUF BPE tokenizer, merge-order aware: “BPE tokenizer that respects merge priority order from GGUF metadata” [crates/muser-engine/src/tokenizer/bpe.rs:8-10], with GPT-2 byte-mapping and byte-fallback paths plus an SPM-style mode selected by tokenizer.ggml.pre [bpe.rs:33-41; loader.rs:39-40]. The loader fails closed on a vocab-length mismatch [loader.rs:41-47], and the whole tokenizer metadata surface — vocab, merges, token types — carries one SHA-256, 61e73226502f8f54455555990c0000852247bbec32b107730ec544bc0b738055 [loader.rs:61], asserted against the pinned artifact [crates/muser-server/src/chat_template.rs:258-261].
• Chat template — a Jinja-style template ships inside the GGUF (tokenizer.chat_template). The pinned artifact’s is exactly 7,167 bytes, SHA-256 114f55ebdc1804c1af371197b9fdf2d6bb925966c9dfe46b73782a71bc07965e, asserted at load [chat_template.rs:252-257; loader.rs:48-55]. Why hash it? Rendering drift changes prompts → tokens → everything downstream; template identity is bound into durable bundles [crates/muser-server/src/session_store.rs:26].
• End-of-generation — two control tokens are declared EOG, EOS 200,001 and EOT 200,008, with the eot/eom metadata keys merged into one set [config.rs:213-226; muse_golden.rs:106-108] — “dual EOS” is an asserted model fact [crates/muser-engine/src/lib.rs:11-13].

The point of this battery of hashes is the same fail-closed posture as §9.2: an engine that quietly accepted a different tokenizer or template would produce plausible text with the wrong identity.

9.14 Tradeoffs

The decisions above were made by the model’s authors, not by Muser — but each dictates what the kernels must be good at, and several have measured consequences here.

GQA 16:1 — the 16× KV lever. Sharing 2 KV heads across 32 query heads shrinks the per-token KV read and the cache by 16× (§9.6 arithmetic). At the 131,072-position limit that is ≈ 1.827 GB versus ≈ 29 GB of KV per slot [docs/memory-footprint.md] — four full-context slots on 96 GB, or none. The measured consequence of the NoPE half of that saving is the kvpack delta handoff: a 32,768-of-65,536 request moved 54.2851 % of full bytes with a bit-exact output [ledger stage-6; benchmarks.md §4] — possible only because NoPE tiles are position-free bytes (§9.7).

The sandwich norms buy training stability and cost an exactness contract. We told this one as a war story a few sections back; here it is as a ledger entry. Every “obvious” fusion across the 1e-8 post-norms has been measured to change public logprobs beyond the 1e-4 contract, and the hybrid retained-activation schedule that reached max normalized-logprob error 3.197e-4 was removed rather than hidden behind a widened tolerance [docs/decode-dispatch-gap-20260815.md]. What ships instead is a fusion built backwards from the requirement: the dual-eps fused tail reproduces the pinned kernels’ rounding boundaries exactly [decode.rs:1328-1330] — a fusion whose design constraint is bits, not speed (Ch 12, Ch 35).

The soft cap changes logit comparison rules. Because 20·tanh(l/20) is non-linear, equal logit differences near the ceiling are not equal evidence differences; cross-engine comparisons at the tail must use tokens and logprobs, not raw logits [docs/muser-architecture.md §Model and engine] — one reason Ch 38’s parity gates are built the way they are.

9.15 What comes next

You now hold the complete map: 52 sandwich-norm blocks in a [sliding, sliding, sliding, full] collar, GQA 32:2 at head_dim 128 over a 6,656-wide residual stream, a sigmoid gate before every o_proj, dual-epsilon norms, and a scale-then-cap logit tail — every number double-cited, every kernel chapter from here pointing back at a row in Tables 9.1–9.2. The open question is movement: how does one token walk this graph on the Metal side — which kernels run, in what order, what is fused, where every buffer lives, and how the DFlash draft and the remote-prefill handoff overlay the loop? Ch 10 answers with one picture.

References

• crates/muser-engine/src/config.rs — :13-28 constants and the 1e-8 post-norm; :51-93 MuseLayerKind/uses_rope/layer % 4 == 3; :158-211 MuseConfig::from_gguf (every key of Table 9.1); :268-281 attn_dim/kv_dim/heads_per_kv/attn_scale; :286-333 tensor-shape contract; :347-381 SWA-pattern resolver; :390-403 and loader.rs:98-138 QkNormProbe.
• crates/muser-engine/src/reference.rs:297-580 — the CPU oracle: layer walk, gate placement, sandwich tails, scale-then-softcap, SWA visibility rule.
• crates/muser-engine/src/lib.rs — :1-15 asserted model facts; :74-79 QK-norm provenance; :99-109 oracle-as-spec; :126-131 tokenizer scope; :175-188 the graph summary.
• crates/muser-engine/src/decode.rs — :5569-5606 QKVG concurrency; :5793-5818 sigmoid gate and fused tail; :5869-5905 final-norm fusion and softcap; :1344-1358 two-class KV allocation.
• crates/muser-engine/src/loader.rs:28-63 — load-time identity hashing. crates/muser-engine/src/metal.rs:46-68, src/rope_nco.rs:12-14 — fixed fixture values (theta 500,000; the test-only 1/√26 literal).
• crates/muser-engine/src/shaders/ — ferrite/rope.metal:624-657 (adjacent-pair comment); ferrite/sigmoid_gate.metal:7-17; muse_reference.metal:15-27.
• crates/muser-engine/tests/muse_golden.rs:14-117 — release-model identity, geometry assertions, dual EOS. crates/muser-server/src/chat_template.rs:235-263, src/session_store.rs:26 — template/tokenizer hashes; bundle binding.
• crates/muser-cluster/src/schedule.rs:1-21 — the 13 NoPE tiles [3,7,…,51]; relocate-as-bytes framing.
• docs/muser-architecture.md §Model and engine; docs/memory-footprint.md; docs/release-provenance.md:823; docs/decode-dispatch-gap-20260815.md.
• [arxiv:1512.03385] He et al., Deep Residual Learning. [arxiv:1706.03762] Vaswani et al., Attention Is All You Need. [arxiv:2305.13245] Ainslie et al., GQA.

Chapter 10 — The forward pass at a glance

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 9 (the architecture — this chapter walks exactly that graph), Part I (command buffers, encoders, dispatch). This is the spine chapter: one diagram of the whole decode loop. The kernel-by-kernel walk begins in Ch 11.

Chapter 9 gave you the map — 52 sandwich-norm blocks, two attention classes, a gated output, a scaled-and-capped logit tail — every number cited to the pinned tree. This chapter makes it move. One token will walk the whole graph on the Metal side, and we will watch the real code open every gate for it: which kernel runs at each step, what is fused into what, which buffers the token’s data flows through, and how the DFlash draft loop and the remote prefill handoff wrap around the plain one-token walk without changing it. Everything quoted here was read from the pinned tree; the op order comes from a real walk of encode_token and its serving twin, not from a design doc.

10.1 Three regimes, one family of graphs

Before we can follow a token anywhere, we have to answer the question the engine itself answers first: what kind of work is this? An LLM generates text in regimes that differ enough that Muser compiles them as different routes over the same weights. Define them precisely, because every performance claim in this book is scoped to one of them — a throughput number quoted under the wrong regime is not a rounding error in the storytelling, it describes a different machine:

• Prefill — the first step of a generation: the entire prompt, all at once. Many query tokens hit each weight matrix, so the projections become GEMMs (matrix × matrix) with heavy arithmetic reuse per weight byte. Prefill is compute-bound.
• Decode — everything after: one token in, one prediction out, repeat. A single vector hits every weight matrix — a matvec with almost no reuse. Decode is bandwidth-bound; it is the regime this book follows, and the regime the whole engine is shaped around Ch 1.
• Speculative verify — decode’s batch-shaped cousin: when the DFlash draft proposes a block, the target must score up to MAX_DFLASH_BLOCK = 16 tokens in one pass [crates/muser-engine/src/decode.rs:47]. For those few dispatches decode temporarily looks like prefill (multiple query rows), which is exactly why speculation is the fourth lever of [Ch 1] — it reintroduces reuse into a serial loop.

Why the regimes differ, in one sentence. Prefill has parallelism across tokens, so it is compute-bound; decode is one token, so it is bandwidth-bound; speculative verify buys back a slice of prefill’s reuse for the decode path. Same matrices, different bottleneck.

10.2 Two routes through the same 52 layers

So which graph does a single token actually run through — and why is the answer not simply “the fastest one we have”? Here is the single most important routing fact in the engine, and it is stated in a code comment at the exact decision point. When serving hands the engine one token, MetalMuseModel::forward_into does this:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:2077-2093
pub fn forward_into(
    &mut self,
    tokens: &[u32],
    logits: &mut Vec<f32>,
) -> Result<(), MetalModelError> {
    if tokens.len() == 1 {
        let scheduler = Arc::clone(&self.shared.scheduler);
        let _permit = scheduler.acquire(self.sequence_id, AcceleratorWork::Decode)?;
        // The legacy one-token graph uses Ferrite fused residual/norm and
        // gate-up kernels whose rounding diverges from the source-pinned
        // llama Metal graph enough to breach public logprob tolerance.
        // The one-row batch graph dispatches the exact pinned kernels and
        // has the same KV transition, so it is the serving correctness
        // path until each fused kernel independently passes full-logit
        // parity.
        *logits = self.forward_batch(tokens)?;
        return Ok(());
    }
}

Read that comment twice, because it records a fork we walked into with the opposite expectation. Muser inherited a single-token graph from Ferrite: fused residual/norm kernels, a fused gate-up kernel, fewer dispatches, less traffic. It was obviously the graph serving should use. We expected a free win and wired it into the hot path.

What came back was arithmetic that no longer matched the comparator. Fusing a norm into an add changes the order the partial sums are accumulated in, and floating-point addition is not associative, so the same weights produced logits that were close — and not close enough. The divergence from the pinned llama.cpp Metal graph breached the public-logprob tolerance. The lesson we took was not “fusion is bad”; it was that on this engine a fused kernel earns the serving path by passing full-logit parity on its own, one kernel at a time, and until it does, its speed is unspendable. Correctness gates speed. So serving refuses to use the fast graph, and the consequences of that refusal ripple through this whole chapter:

1. The serving decode path is the one-row batch graph: forward_into → forward_batch (decode.rs:2857) → forward_batch_hidden (decode.rs:3788) → encode_batch_hidden_range (decode.rs:3858), with token_count = 1. One row, exact pinned kernels.
2. The legacy single-token graph survives as the teacher-forced route: forward_token (decode.rs:5432) → encode_token (decode.rs:5515). It is the benchmark lane that matches the comparator’s no-readback policy (forward_teacher_forced, decode.rs:2118-2137) and the phase-profiling route (MUSER_METAL_PHASE_PROFILE, decode.rs:5440). Its op sequence is what Part IV narrates, because it is the cleanest straight-line telling of the graph. Narrating it costs the reader nothing: the batch route mirrors it row for row, and the packed multi-sequence encoder encode_decode_group (decode.rs:4954) exists precisely to run “batch rows of the same op sequence”.
3. Multi-sequence decode packs up to four sequences into one graph: forward_decode_group (decode.rs:4869) accepts 1..=4 models sharing one MetalShared executor, encodes all rows with one concurrent encoder, commits once, waits once (decode.rs:4920-4937). This is the rendezvous the server’s 250 µs DecodeBatcher coalesce window feeds [crates/muser-server/src/state.rs:231-254].

Whichever route runs, it stops at the same door before it encodes anything: it acquires the scheduler. A single token asks for AcceleratorWork::Decode, a prefill chunk asks for AcceleratorWork::Prefill per chunk (decode.rs:2083-2084, decode.rs:2109). The reason is ownership — one scheduler owns one accelerator — and when both kinds of work want that accelerator the contest is not a tie: decode outranks prefill [decode.rs:1040-1074].

10.3 The master diagram — one decode token, end to end

So far we have argued about which route runs. Now the walk itself: what happens to one token, in the order the GPU is actually told to do it? This is the figure the rest of the book zooms into. Boxes are dispatch groups as the encoder issues them; FUSED marks a fusion that is live by default, and the two opt-in fusions are marked as such. The op order is encode_token’s, which the serving batch route mirrors.

flowchart TD
    START([token_id, position n_past]) --> EMB

    EMB["<b>① Embedding lookup — GPU</b><br/>muser_embedding_q4k<br/>token_embd row → residual [6656] — Ch 11"]
    EMB --> ENTRY["<b>② Entry RMSNorm — GPU</b><br/>weight = all ones (weightless), eps 1e-5<br/>residual → normed"]

    subgraph LOOP["per-layer — repeated ×52 (layers 0..51): [SWA, SWA, SWA, FULL] collar"]
      direction TB
      A0["<b>③ attn-norm</b> (layer 0 only —<br/>later layers receive it fused from ⑪)"]
      QKV["<b>④ QKVG projections — one concurrent group</b><br/>Q [6656→4096], K [6656→256],<br/>V [6656→256], gate [6656→4096] — Ch 13"]
      QKN["<b>⑤ per-head QK-norm</b><br/>32 Q-heads + 2 K-heads, eps 1e-5 — Ch 14"]
      ROPE["<b>⑥ RoPE — SWA layers only</b><br/>rope_norm_batch_cached, theta 500,000<br/>NoPE layers skip this box — Ch 14"]
      ATT["<b>⑦ KV store + attention — route ladder</b><br/>store K,V row into the plane, memory barrier,<br/>then llama-vec / splitk / ferrite-interleaved — Ch 15–16"]
      SG["<b>⑧ sigmoid gate</b><br/>attn ⊙ σ(gate) — Ch 17"]
      OP["<b>⑨ o_proj</b> [4096→6656] — Ch 17"]
      T1["<b>⑩ FUSED dual-eps tail</b><br/>muser_fused_norm_residual_rms_norm_32sg:<br/>residual += post_attn_norm(⑨); then ffn_norm → FFN input — Ch 12, 19"]
      FFN["<b>⑪ FFN gate+up — split by default</b><br/>2 matvecs [6656→19968] + muser_silu_mul_inplace<br/>OPT-IN fusion: ffn_q4k_gate_up_silu_4r2s — Ch 18"]
      DOWN["<b>⑫ ffn_down</b> [19968→6656] — Ch 19"]
      T2["<b>⑬ FUSED dual-eps tail ×2</b><br/>residual += post_ffw_norm(⑫); then the NEXT<br/>layer's attn-norm (or the final norm) — Ch 12, 19"]

      A0 --> QKV --> QKN --> ROPE --> ATT --> SG --> OP --> T1 --> FFN --> DOWN --> T2
    end

    T2 --> LMH["<b>⑭ LM head</b><br/>output.weight [6656→202048] matvec — Ch 20"]
    LMH --> CAP["<b>⑮ FUSED scale + soft cap</b><br/>muser_scale_softcap_inplace:<br/>× 0.196116 (1/√26), then 20·tanh(l/20) — Ch 20"]
    CAP --> RB(["logits read back to CPU — one vocab row<br/>sampling / argmax on CPU — Ch 21"])

    T2 -. "last layer only: ⑬ writes the final<br/>normed hidden — ⑭ reads it" .-> LMH

Figure 10.1: One decode token through Muse Glimmer on the Metal side, as encode_token [crates/muser-engine/src/decode.rs:5515-5907] issues it. Boxes ③–⑬ run 52 times; the SWA layers run ⑥, the 13 NoPE layers skip it. The residual stream lives in two ping-ponging buffers (§10.4). Sampling is CPU work on the read-back row (§10.9).

Read the fusions explicitly. Fusions are where the diagram stops looking like the textbook, and they are where a reader gets lost: you go hunting for a step and it is not there, because it happens inside another one. So it is worth knowing exactly which logical steps the route collapses into single kernels, and what gates each collapse:

• (a) The dual-eps fused tails (boxes ⑩ and ⑬) — live by default. One kernel, muser_fused_norm_residual_rms_norm_32sg [crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:147], does three logical ops: residual += post_norm(sub_block_out) with eps 1e-8, then next_norm(residual) with eps 1e-5 — two norms and an add in one dispatch, emitting the next sub-block’s already-normed input. The same kernel closes every layer: box ⑬’s second output is the next layer’s attention-norm result (next_norm selected at decode.rs:5869-5876) — which is why box ③ runs only for layer 0. The dispatch is one threadgroup of 1,024 threads per row, 32 SIMD-group partials per reduction [crates/muser-engine/src/metal/encode/norm.rs:163-235]. The batch/serving route uses its batch twin muser_fused_norm_residual_rms_norm_batch_dual_eps [rmsnorm_batch_tail.metal:250], bound at decode.rs:4510-4525.
• (b) Scale + soft cap (box ⑮) — live. muser_scale_softcap_inplace multiplies by logit_scale and applies the tanh cap in one pass over the logits [crates/muser-engine/src/shaders/muse_reference.metal:15-27; decode.rs:5898-5905].
• (c) The final norm is fused into the last layer’s tail. For layer 51, box ⑬’s “next norm” is the model’s output_norm and its residual output goes to the buffer the LM head reads (decode.rs:5869-5876): the final RMSNorm is not a separate dispatch on this route.
• (d) FFN gate+up+SiLU (box ⑪) — opt-in, OFF by default. The Ferrite four-row/two-SIMD-group kernel ffn_q4k_gate_up_silu_4r2s [shaders/ferrite/ffn_fused_tail.metal:496] reads both Q4_K weight matrices once and emits the finished SiLU(gate) ⊙ up row — but only under MUSER_FERRITE_FFN_GATE_UP and only when both matrices are Q4_K (decode.rs:5819-5836). The reason it is opt-in is measured, not aesthetic: “The pinned baseline packet regressed with it on this model, so keep the imported route explicitly opt-in for experiments” [decode.rs:980-983]. The default is two matvecs plus muser_silu_mul_inplace [muse_reference.metal:4].
• (e) Q/K/V/gate share one concurrent dispatch group (box ④). Four independent matvecs read one shared normed input and write disjoint activations — issued as one group so the concurrent encoder can overlap them (decode.rs:5566-5598; the comment: “llama.cpp and Ferrite issue the four independent attention projections as one concurrent set”). Note what is deliberately not fused here: there is no fused QKV+RoPE mega-kernel on this path, unlike the ancestor book’s engine [ferrite-book Ch 13] — the pinned-kernels correctness rule (§10.2) forecloses it.
• (f) KV store and attention are separate dispatches with an explicit barrier (box ⑦) on the vec routes: store K/V, memory_barrier_with_resources, attend (decode.rs:5660-5670). The barrier, not a fused kernel, is what orders the store before the read.

Everything from ① to ⑮ is recorded onto one command buffer with one concurrent encoder per token, committed once:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5448-5458
let command_buffer = queue.new_command_buffer();
// One concurrent encoder owns the complete token. Graph dependencies
// are explicit barriers; independent projection groups share a barrier
// interval and may overlap, matching the accepted Ferrite/llama route.
let serial = GraphEncoder::concurrent(
    command_buffer
        .compute_command_encoder_with_dispatch_type(metal::MTLDispatchType::Concurrent),
);
self.encode_token(&serial, &token_view, self.n_past)?;
serial.encoder.end_encoding();
command_buffer.commit();
}

This is command-buffer amortization (Ch 2): the whole 52-layer tape is recorded, then “play” is pressed exactly once per token.

10.4 The residual stream — two buffers in a relay

The master diagram moves a token through boxes. But where do the token’s bytes actually sit while that happens, and who is allowed to overwrite them? Get this wrong and the symptom is not a crash — it is a layer reading a value one dispatch too late, which shows up much later as logits that are subtly, and unfixably, the wrong ones.

In the ancestor book the residual stream was one buffer, mutated in place 56 times. Muser’s decode graph runs a two-buffer relay instead, and the code documents it at the top of the layer loop:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5546-5548
// `normed` is the current hidden buffer at layer entry. Each layer
// writes the attention residual to `hidden`, then the FFN result back
// to `normed`, so no full-width copy dispatch is needed.
}

One naming trap before the figure, because it costs an afternoon if you walk into it: the hidden/normed in that comment are the kernel parameters of box ⑩/⑬, not buffer names — the buffers are activations.normed and activations.post_norm. Per fused tail — Figure 10.2 — the kernel does:

    activations.normed        activations.post_norm       layer.post_attn_norm (w1)
    (residual, 6656 f32)      (o_proj out, 6656 f32)      layer.ffn_norm      (w2)
          │                          │
          ▼                          ▼
     ┌────────────────────────────────────────┐   eps1 = 1e-8 (post norm)
     │ residual += src · rsqrt(mean(src²)+eps1) · w1 │   eps2 = 1e-5 (ffn norm)
     └────────────────────────────────────────┘
          │                          │
          ▼                          ▼
     normed (updated)    →    post_norm = normed · rsqrt(mean(normed²)+eps2) · w2
                               (the FFN's already-normed input)

Figure 10.2: One fused dual-eps tail (boxes ⑩ and ⑬). Two RMSNorms, one residual add, one kernel — semantically exactly the CPU oracle’s rms_norm_mul(proj, post_norm_eps); hidden = proj + residual; rms_norm_mul(hidden, rms_eps) [crates/muser-engine/src/reference.rs:466-489].

So the residual bytes live in activations.normed from the entry norm until layer 51’s tail, which writes the final normed stream into activations.hidden for the LM head. Each layer reads the stream twice (once per tail) and rewrites it twice — 104 full-width reads plus 104 writes per token across the model, all inside fused kernels, zero standalone copies.

Why the relay instead of in-place mutation? Because each tail must read the pre-add residual to norm it for the next sub-block while simultaneously writing the post-add stream — and because the fused kernel’s two reductions must reproduce the pinned ggml kernels’ rounding boundaries exactly, float4 lane for float4 lane [decode.rs:1328-1330; norm.rs:163-235]. The buffer layout is a consequence of the exactness contract, not a style choice.

10.5 The attention route ladder (box ⑦, up close)

Box ⑦ is the only box in the master diagram that hides a decision. Attention is not one kernel but a ladder: per layer, per token, the engine picks a rung by reading the live KV plane’s metadata. Two questions settle it — is the pinned llama vec kernel safe on this plane, and does this layer’s window let it run without padding? Here are the predicates that answer them, verbatim from the route walk:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5646-5657
let llama_vec_rows = (strict_attention || self.kernels.has_llama_flash_attn_vec())
    && plane.len > 0
    // The pinned vec kernel rounds KV reads to a 32-row block.
    // A deliberately tiny raw session can have a smaller backing
    // allocation, so taking the vec path would read past it and
    // poison the full distribution with NaNs.
    && plane.capacity >= 32
    && (plane.origin_physical == 0 || plane.len == plane.capacity);
// Token-major SWA cannot use llama's pad kernel (nb11 is a full
// token row, not one head). Only take that path when the window
// is a multiple of 32 so the vec kernel never pads.
let llama_swa = llama_vec_rows && plane.len.is_multiple_of(32);
}

Read that middle comment as the scar it is. The vec kernel is both the fast rung and the exact rung — it is the comparator’s own arithmetic — so the tempting rule is “take it whenever it exists.” A deliberately tiny raw session breaks that rule. The kernel rounds its KV reads up to a fixed block size; a session whose backing allocation is smaller than one block gets read past its end; and the failure does not announce itself as a crash. It comes back as NaNs spread across the full distribution, which is the worst failure mode there is — silent, and it looks like a model bug rather than a routing bug. So the predicate is fail-closed: the plane must prove the fast rung is safe, or the engine takes a rung it can defend.

The four rungs the predicates select between (decode.rs:5658-5792):

Table 10.1: The decode attention ladder. “llama-pinned” kernels come from the prebuilt llama.cpp metallib (MUSER_GGML_METALLIB) so their arithmetic is the comparator’s own [crates/muser-engine/src/metal/context.rs:122-131]. The capacity ≥ 32 guard is fail-closed: the pinned vec kernel reads in 32-row blocks, and a tiny backing allocation would be read past its end [decode.rs:5648-5652].

Which planes exist in the first place is Ch 9’s two-class story made concrete: SWA layers get a 2,048-row token-major ring, NoPE layers a growing head-major plane, allocated zero-filled so wrapped rows never leak stale storage [decode.rs:1344-1358]. The ring is carried as explicit origin_logical/origin_physical metadata, and append refuses any position that is not exactly the next one — a CacheDiscontinuity error — so no physical placement is ever derived from absolute position [decode.rs:265-284].

10.6 Overlay one: the DFlash draft/verify/accept loop

Everything so far buys exactly one token per trip through the master diagram. Can a trip be made to pay for more than one? That is the question the speculative lane asks, and its answer is a bet: guess ahead cheaply, then check the guesses against the real model instead of trusting them.

Plain decode proposes one token at a time. The speculative lane wraps the same graph in a three-beat loop — draft, verify, accept — and its Metal machinery is a split of Figure 10.1, not a different model (Figure 10.3):

flowchart TD
    subgraph ROUND["one speculative round (verify-length ≤ 16 rows)"]
        direction TB
        D["<b>Draft</b> — DFlash, the 5-layer assistant of Ch 8<br/>draft_greedy: reads target hidden states captured at<br/>pinned target layers (DFlashConfig.target_layer_ids)"]
        B["<b>begin_dflash_verify_suffix</b> decode.rs:3298<br/>prefix command buffer: run the block through layers 0..capture_end<br/>synchronously; copy exact hidden rows out for the draft; then<br/>submit the REMAINING layers + LM head without waiting"]
        W["<b>Wait-free overlap</b><br/>draft consumes the captured rows while the<br/>target suffix is still on the GPU"]
        F["<b>finish_dflash_verify_suffix</b> decode.rs:3635<br/>wait for the suffix command buffer;<br/>read back the block's full-vocab logit rows"]
        V["<b>Verify on the CPU — exact</b><br/>verify_full_speculative_mt_ordered sampling.rs:1033<br/>accepts/rejects each drafted token against the<br/>target's own full distributions"]
        C{"accept prefix?"}
        OK["<b>commit_speculative_prefix</b> decode.rs:1413<br/>accepted K/V rows stay exactly where the<br/>block's execution wrote them"]
        RB["<b>Rollback</b> — restore MetalSpeculativeCheckpoint:<br/>NoPE planes rewind metadata only; SWA planes<br/>restore the ≤16 rows a block may overwrite"]
        D --> B --> W --> F --> V --> C
        C -- "prefix accepted" --> OK
        C -- "first rejection" --> RB
    end
    OK --> D
    RB --> D

Figure 10.3: The DFlash speculative overlay on the decode graph. The target’s KV planes are protected by a transactional checkpoint whose design comment is worth quoting: “Growing NoPE planes only need their logical metadata rewound. SWA planes may overwrite live ring rows, so the small set of destinations touched by the candidate block is retained here instead of copying the complete multi-gigabyte cache on every DFlash round” [crates/muser-engine/src/decode.rs:208-212].

Three properties of this loop carry the engine’s exactness culture:

1. Verification is exact and CPU-side. Acceptance compares every drafted token against the target’s full distributions — the very same read-back vocab rows that plain decode samples from. Nothing cheaper is allowed near the accept decision: no approximate verifier, no shortcut on the logits. One lane did try the shortcut, and the distributed verifier was measured and rejected for it; that war story is Ch 33. The code to read is verify_full_speculative_mt_ordered [crates/muser-engine/src/sampling.rs:1033], reached from the engine entry points Session::verify_batch and begin/finish_dflash_verify_suffix [crates/muser-engine/src/api.rs:913; decode.rs:3298, 3635].
2. The split point is a real command-buffer boundary with a correctness rule. The suffix re-materializes its entry norm from the authoritative residual instead of trusting a cross-command-buffer temporary: “Do not rely on the fused layer-49 tail’s secondary normalized output surviving as an implicit input to layer 50” [decode.rs:3388-3393].
3. KV mutation is transactional. Commit keeps accepted rows in place; rollback restores only what a ≤16-row block could have overwritten [decode.rs:1413-1419].

So what does the bet actually pay? The kquant speculative lane is the engine’s speed lane: the 107.9 tok/s figure survives as the qualification bar, and the current synthetic restatement at the fixed draft window is decode ratio 1.23692 at 2,048 context. The scope language around that ratio matters as much as the ratio does — five of five exact reps, synthetic only, never a natural-text workload claim — and the row that holds that scope is retained: [claims #15]. The loop’s deep treatment, including why native NVFP4 speculation is fail-closed by construction, is Ch 33.

10.7 Overlay two: the handoff that plants KV before decode starts

The second overlay changes what happens before Figure 10.1 runs at all. On the disaggregated lane, the GX10 producer prefills NVFP4 and ships the resulting KV across the wire; the Mac’s job is to plant those bytes and start decoding from a nonzero position (Figure 10.4):

flowchart LR
    P["GX10 producer<br/>vLLM NVFP4 prefill"] -->|"Handoff V2<br/>mTLS + HMAC-sealed tiles"| R["Mac receiver"]
    R --> S["scatter-on-arrival:<br/>each authenticated tile unpacked into a<br/>DETACHED Metal generation as it arrives"]
    S --> C["validate_complete — every expected<br/>K/V row arrived and verified"]
    C --> SW["commit: n_past = tokens.len();<br/>cache = install.planes<br/>(atomic swap, decode.rs:1990-1994)"]
    SW --> D["Figure 10.1 starts at position n_past —<br/>no local prefill ever ran"]

Figure 10.4: The remote-KV install path ([crates/muser-engine/src/decode.rs:1852-1994]). “Detached” is the operative word: the install builds its own MetalKvPlane set alongside the live one, and the swap happens only after the seal validates — live decode never observes a half-planted cache.

The engine-side entry is begin_remote_kv_install, and its ring handling encodes a subtle exactness rule — the planted SWA ring must sit at the same physical rotation a sequentially-built one would have:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:1880-1885
plane.origin_logical = origin;
// Match a sequentially-built ring exactly. Physical scan order is
// numerically observable, so remote restore cannot repack the
// retained logical tail at row zero.
plane.origin_physical = origin % live.capacity;
plane.len = len;
}

“Physical scan order is numerically observable” is a sentence to sit with: because the attention kernels accumulate in physical row order, where the ring rows sit changes floating-point sums — so a remote install that prettily repacked the tail to offset 0 would produce different bits than a local prefill of the same tokens. The delta variant (begin_remote_kv_install_delta, decode.rs:1908) extends this to warm prefixes: the held [0, cut) span is copied out of the live planes with the exact ring mapping and only suffix tiles are accepted.

The wire schedule turns Ch 9’s two-class split into a scheduling advantage. The 13 NoPE layers’ tiles are position-free bytes — nothing in them depends on where in a ring they will eventually sit — so they need not wait for the producer to finish. They stream during CUDA prefill, one HMAC/TLS frame per 512-token NoPE tile, ~6.5 MiB. The SWA tiles have no such freedom, and the tail groups ship in the last ubatches (micro-batches) [crates/muser-cluster/src/schedule.rs:1-12, 20].

What that overlap buys is the Part VI headline: TTFT (time to first token) 4.149× faster than local prefill at 130,815 tokens, remote 137.405 s against local 570.122 s. The arm matters as much as the ratio — this is the EEE-off arm, Energy-Efficient Ethernet disabled on the link, which is Ch 31’s invariant — and it is five counted reps. We kept the claims row that carries the whole scope: [claims #6]. Reusing a planted cache across requests is its own ladder, in Ch 25; the transport underneath — mTLS, the HMAC-sealed manifest, the replay ledger — is Ch 30; and whether NVFP4-produced KV can be trusted at all is Ch 32.

10.8 Where every buffer lives

Figure 10.1 names buffers; this section sizes them — and on a machine with unified memory, sizing is design. How many sequences a Mac can serve at once is settled less by the kernels than by which allocations every sequence must own and which can be paid for once. So two scopes matter here. Per-sequence state: each of up to four slots owns a full set. The shared executor MetalShared: one context, one kernel set, one mmap’d weight arena, retained deliberately because “retaining one context, pipeline set, mapped weight arena, and GPU vector set avoids loading the 16+ GiB target once per serving slot” [decode.rs:954-957]. Sizes below are derived; the arithmetic is shown so you can re-derive it.

Table 10.2 — Per-sequence activation pool (Activations::new, [decode.rs:897-952]; all GPU, all allocated once at session construction)

Table 10.2: The one-token activation pool totals ≈ 15 MB dominated by the speculative block’s logit rows — noise next to the weights (below), and every buffer is reused every token with zero hot-path allocation.

Table 10.3 — Per-layer KV planes (from_shared, [decode.rs:1344-1358]; f16 element type on every Metal lane [decode.rs:236-237])

One slot at the 131,072 limit: (39 × 2 MiB) + (13 × 128 MiB) ≈ 1.827 GB [docs/memory-footprint.md]; four slots ≈ 7.306 GB.

Shared, GPU-mapped, read-only during decode: the mmap’d weight arena — 16,756,681,056 bytes of GGUF, zero-copy views [crates/muser-engine/src/lib.rs:14; docs/muser-architecture.md]; the entry norm’s ones vector (6,656 × f32); the RoPE frequency table (head_dim/2 = 64 f32 values built once at startup, [decode.rs:1240-1264]) and the position table (131,072 u32); per-layer norm weights. Prefill-only workspace (BatchWorkspace, [decode.rs:799-856]): token-scaled activation twins plus two SWA staging shadow planes of 131,072 × 256 × 2 B = 64 MiB each and flash-attention scratch — allocated per chunk width, reused across chunks. Packed decode workspace (DecodeBatchWorkspace, [decode.rs:858-866]): up to 4 rows × vocab logits (4 × 808,192 B ≈ 3.2 MB).

CPU side: the retained distribution Vec<f32> (202,048 × 4 = 808,192 B) refilled in place per token, sampler/RNG/grammar state, the detokenizer — all outside the accelerator owner [docs/muser-architecture.md §Slots and scheduling].

10.9 CPU vs GPU — the division of labor

Where does the seam between the two processors fall, and who is holding the token when something goes wrong? The GPU owns the entire arithmetic graph — embedding through softcap, one command buffer per token (§10.3). The CPU owns everything around it. The serving loop shows both the handoff and the failure policy in the same few lines:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/api.rs:699-708
// The decode path refills the retained distribution in place, so a
// token costs one vocabulary-sized copy for the result instead of two
// fresh allocations.
let mut logits = self.last_logits.take().unwrap_or_default();
if let Err(error) = self.forward_into(&[input.token_id], &mut logits) {
    // A failed forward leaves the buffer untouched, so the
    // distribution installed before the call is still the current one.
    self.last_logits = (!logits.is_empty()).then_some(logits);
    return Err(error);
}
}

After forward_into returns, the CPU validates finiteness (ensure_finite_logits — a NaN row installs no distribution, fail-closed), takes the argmax or samples (api.rs:715-737), and hands the token back. Speculative acceptance (§10.6) is likewise CPU work on read-back rows. A GPU two-phase argmax exists (argmax_f32_phase{1,2}, greedy_argmax_f32_phase{1,2} [shaders/ferrite/argmax_f32.metal:7,41,77,125]) but serves the no-readback benchmark lanes, not the sampling path — the read-back here is one vocab row per token, 808 KB, which the CPU must see anyway to sample.

10.10 Prefill is a different graph

Everything above is the decode route. Prefill earns a section even in a decode book for a reason worth stating plainly: the remote lane above is an argument about this graph, and you cannot judge what shipping a cache across a wire saves until you know what the local prefill it replaces would have cost. When forward_into receives more than one token it becomes prefill, and the route changes shape at every joint (decode.rs:2095-2113):

• Chunking: prompts stream through a 512-row physical batch (PREFILL_BATCH_TOKENS, decode.rs:53); once any decoder is queued the boundary shrinks to 64 rows (MAX_TEACHER_FORCED_TOKENS, decode.rs:54) so decode can take the accelerator “without another long accelerator interval in front of it” [decode.rs:2098-2101].
• Projections become batch GEMMs with weight-row reuse (encode_batch_projection, decode.rs:5946-5980; the M16 NVFP4 route fires at 16-row multiples). This is the roofline flip of §10.1: same weights, arithmetic per byte now dominates.
• Attention routes differ: contiguous cache ranges take llama’s pinned prefill kernels or the local FA2 (decode.rs:4090-4365); an SWA ring that has wrapped stages old rows into a detached F16 shadow first (encode_stage_swa_prefill_f16) and commits ring metadata only after the chunk attends from the shadow.
• The one-row output graph gets an exactness special case: when the last row of a deep prompt produces the output logits, the fused dual-eps tail splits back into the exact pinned three-dispatch sequence (llama_final_row_boundary, decode.rs:4473-4478) — the same correctness-first instinct as §10.2, applied at a batch boundary.

Prefill’s full treatment is Ch 36; the remote lane that replaces Mac-side prefill entirely is Part VI.

10.11 Tradeoffs

Three structural bets shaped this code. None was obvious in advance, and each was settled by a measurement that contradicted what somebody — usually us — expected.

Bet 1 — route serving decode through the one-row batch graph, not the fused single-token graph. The routing section told this fork from the code’s side; here is what it cost and what settled it. The cost is real: the Ferrite-lineage fused kernels sit off the hot path, and the dispatch savings they were written for go unspent. The first piece of evidence is the routing comment itself — the fused kernels’ rounding “breaches public logprob tolerance” against the pinned llama graph [decode.rs:2085-2091]. The second came from trying a variant and watching it fail. We ran a hybrid retained-activation schedule expecting the divergence to stay inside the parity contract, and it did not: max normalized-logprob error 3.197e-4 against a 1e-4 contract, with 201,970 of 202,048 logits differing, and the first divergence traced to a single f16 ULP in layer-1 V. It was removed [docs/decode-dispatch-gap-20260815.md]. Sit with that ratio for a moment — one unit in the last place, in one tensor, in one early layer, and nearly the entire vocabulary row comes out different. A verified-slow path beats an unverified-fast path.

Bet 2 — keep the fused FFN gate-up kernel opt-in. This one we expected to win outright. The imported ffn_q4k_gate_up_silu_4r2s reads both Q4_K weight matrices once rather than twice, which is strictly less traffic in the hungriest part of the layer, and on the ancestor engine it paid. So we enabled it and ran the pinned baseline throughput packet — and the packet regressed on this model [decode.rs:980-983]. Fusing is not automatically faster; the arithmetic on paper does not overturn a measurement. The code keeps the experiment behind a flag instead of deleting it, because what lost here is a result about this model on this machine, not a verdict on the kernel forever.

Bet 3 — one concurrent command buffer per token, explicit barriers, no scheduler surgery to close the dispatch gap. The gap announced itself as waste. The one-token diagnostic at a 2,048-token fixture counted 760 profiling closures against the legacy graph’s 564 — and a delta that size looks like something you can simply delete. So we went hunting for what to remove, and the first surprise was that nothing was left over: the +196 delta reconciles exactly into 104 norm-boundary groups + 39 SWA staging groups + 52 KV-publication splits + 1 bookkeeping copy [docs/decode-dispatch-gap-20260815.md §label table]. Every one of those groups exists because something must be ordered before something else. The second surprise was the price list. Every cheap removal changed bits — those are Bet 1’s numbers — and the one exact removal, the last row copy, bought −0.136 ms GPU (−0.34 %) on a 40.330 ms token [docs/decode-dispatch-gap-20260815.md §Landed and rejected reductions]. That is what a structural gap looks like as opposed to a sloppy one: the engine chose exactness over the ~3 % it could have stolen. Ch 35 tells the whole story.

10.12 What comes next

You have now seen one token’s whole journey: staged, embedded, normed, and sent through 52 layers of concurrent matvecs, gated attention, and fused norm tails to a scaled, capped, read-back distribution — plus the two overlays (speculative draft/verify, remote KV install) that wrap the loop without altering its arithmetic. The map is complete; the descent begins. The first kernel the token actually meets is the humblest one in Figure 10.1 — box ①, the embedding lookup, a single-row gather from a 1.3-billion-value quantized table ([6656 × 202048], Table 9.2) that turns an integer into the residual stream. Ch 11 opens it.

References

• crates/muser-engine/src/decode.rs:41-54 — split-workgroup cap, DFlash block width, chunk constants.
• crates/muser-engine/src/decode.rs:182-226 — MetalKvPlane and the speculative checkpoint contract.
• crates/muser-engine/src/decode.rs:265-314 — fail-closed append/append_batch.
• crates/muser-engine/src/decode.rs:799-866 — batch and packed-decode workspaces.
• crates/muser-engine/src/decode.rs:897-984 — the per-sequence activation pool; MetalShared.
• crates/muser-engine/src/decode.rs:1216, 1240-1268 — entry-norm ones; RoPE tables.
• crates/muser-engine/src/decode.rs:1328-1334 — fused-tail / concurrent-prefill / FFN-fusion env defaults.
• crates/muser-engine/src/decode.rs:1344-1358 — two-class KV allocation.
• crates/muser-engine/src/decode.rs:1386-1419 — speculative checkpoint and prefix commit.
• crates/muser-engine/src/decode.rs:1852-1994 — remote KV install: begin, delta variant, commit/swap.
• crates/muser-engine/src/decode.rs:2077-2137 — forward_into and the serving-route comment; teacher-forced sink.
• crates/muser-engine/src/decode.rs:2857-2927 — forward_batch entry.
• crates/muser-engine/src/decode.rs:3298-3407, 3635-3666 — begin/finish_dflash_verify_suffix; the boundary-norm rule.
• crates/muser-engine/src/decode.rs:3788-3912 — forward_batch_hidden / encode_batch_hidden_range.
• crates/muser-engine/src/decode.rs:4473-4549, 4570-4611 — llama_final_row_boundary; batch dual-eps tails; split FFN.
• crates/muser-engine/src/decode.rs:4869-4937, 4954-4975 — packed decode group and its mirror encoder.
• crates/muser-engine/src/decode.rs:5432-5513 — forward_token; phase labels (the op census).
• crates/muser-engine/src/decode.rs:5515-5907 — encode_token, the full op walk (Figure 10.1’s source).
• crates/muser-engine/src/metal/encode/norm.rs:163-235 — fused dual-eps dispatch wrappers; 1,024-thread geometry.
• crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:147, 250 — the 32sg and batch dual-eps kernels.
• crates/muser-engine/src/shaders/ferrite/ffn_fused_tail.metal:496 — opt-in ffn_q4k_gate_up_silu_4r2s.
• crates/muser-engine/src/shaders/muse_reference.metal:4, 15-27 — muser_silu_mul_inplace; muser_scale_softcap_inplace.
• crates/muser-engine/src/shaders/ferrite/argmax_f32.metal:7,41,77,125 — GPU argmax phases (benchmark lanes).
• crates/muser-engine/src/api.rs:696-737 — serving decode: retained distribution, fail-closed finiteness, CPU argmax.
• crates/muser-engine/src/sampling.rs:1033 — exact CPU speculative verification.
• crates/muser-engine/src/dflash/spec.rs:730 — draft_greedy (verify-length 3/7/15).
• crates/muser-cluster/src/schedule.rs:1-21 — NoPE-tiles-during-prefill schedule.
• crates/muser-server/src/state.rs:231-254 — the 250 µs decode rendezvous.
• docs/decode-dispatch-gap-20260815.md — 760/564 closure reconciliation; the rejected fusion’s logprob error; the one exact removal.
• docs/muser-architecture.md — lane matrix, scheduler ownership, buffer residency framing.
• docs/memory-footprint.md — KV plane arithmetic; artifact sizes.
• [claims #15], [claims #6] — docs/launch-claims.md: speculative restatement scope; TTFT disaggregation scope.
• [ferrite-book Ch 8] — the ancestor spine chapter (pedagogical lineage; its Figure 8.1 fusion list is Ferrite’s, not Muser’s).

Chapter 11 — Token embedding lookup

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 (the Metal compute model), Ch 3 (mmap’d GGUF, zero-copy weight views), Ch 5 and Ch 6 (Q4_K blocks), Ch 9 (the model’s shapes), Ch 10 (the decode loop and the residual stream). This is the first kernel chapter of Part IV: the operation is small, and the chapter is sized to match.

Chapter 10 ended where the descent begins: box ① of Figure 10.1, the humblest kernel in the whole graph — one integer in, one 6,656-float vector out. This chapter opens it.

We put the smallest kernel first on purpose. Every habit the rest of Part IV leans on — read the shapes before the code, count the bytes before believing a cost, quote the dispatch instead of your memory of it — can be practised here, on an operation that has no arithmetic to hide behind. If a chapter about a table lookup feels like more scaffolding than the lookup deserves, that is the point: the scaffolding is what we are teaching.

11.1 What it computes

The kernel answers one question: how does an integer become a vector? The tokenizer hands the engine a number — the id of a piece of text inside a fixed vocabulary — and everything downstream wants floats. Something has to bridge the two, and this is the something.

Given one token id — an integer in 0 .. vocab_size — the embedding lookup produces the initial hidden vector for that token:

hidden = EmbeddingTable[token_id]        // a [hidden_dim] f32 vector

That is the whole formula. There is no dot product, no reduction, no nonlinearity: it is a row gather out of a learned table, followed by a dequantization from Q4_K (the 144-byte super-blocks of Ch 6) into f32. For Muse Glimmer, hidden_dim = 6,656 and vocab_size = 202,048 (crates/muser-engine/tests/muse_golden.rs:96,101), so the output of this step is 6,656 floats — the seed of the residual stream that all 52 layers will then accumulate into.

This chapter is about muser_embedding_q4k, the Metal kernel that does the gather and the dequant in one dispatch, and about the one structural way Muser’s embedding differs from the ancestor book’s: Muser embeds on the GPU, inside the same command buffer as everything else (crates/muser-engine/src/decode.rs:5523-5533).

11.2 Why it exists — the residual stream is born here

Recall the residual stream from Ch 10: one [6,656] f32 buffer that every layer reads from and adds a delta to. The embedding lookup is the only step in the entire forward pass that writes the initial value of that buffer rather than adding into it. Everything downstream — 52 layers of attention and FFN, the final norm, the LM head — starts from the 6,656 floats this kernel produces.

What breaks if you skip it: nothing downstream has an input. The residual stream would hold whatever the buffer was zero-filled with, every token would embed to the same vector, and the model would emit a constant. The lookup is trivial arithmetic carrying non-trivial content — the “intelligence” is in the learned numbers of the table, not in the operation.

11.3 The row gather, explained from zero

Two questions decide everything about this kernel. Where in memory does a token’s row live, and how many bytes long is it? Answer both and the kernel more or less writes itself; get either one wrong and you will read a perfectly plausible row belonging to some other word entirely.

An embedding is a lookup table: a matrix of learned numbers with one row per vocabulary entry. In the GGUF the tensor is token_embd.weight with shape [hidden_dim, vocab_size]. The loader does not take that shape on trust — it asserts it at load (crates/muser-engine/src/config.rs:295) — and for this checkpoint it accepts the table in two dtypes only, Q4_K or F16 (decode.rs:1209-1214). Memory is vocab-major: token t’s 6,656 values are contiguous (Figure 11.1), so the byte offset of a row is just t × row_bytes.

                 token_embd.weight   (vocab-major rows)
               ┌────────────────────────────────────────────┐
        t = 0  │ 6,656 quantized values …  (3,744 B, Q4_K)  │
        t = 1  │ 6,656 quantized values …                   │
          ⋮    │      ⋮                                     │
   t = 202,047│ 6,656 quantized values …                   │
               └────────────────────────────────────────────┘
   To embed token id t:  read row t → dequant → hidden[0..6,656] (f32)

Figure 11.1: The embedding table. One row per token id; one row is read per token. Drawn in memory order this time (each token’s values are contiguous), unlike the transposed pictures some papers prefer.

How big is one Q4_K row? Show the arithmetic, because the same multiply returns in every kernel chapter:

  Q4_K super-block = 256 elements in 144 bytes          (Ch 6)
  row_bytes        = (6,656 / 256) × 144 = 26 × 144 = 3,744 B
  table total      = 202,048 × 3,744    = 756,467,712 B ≈ 756 MB

That multiply is not ours to invent, incidentally — the dispatch computes row_bytes with the same expression (crates/muser-engine/src/metal/encode/qkv.rs:361).

The whole table is ~756 MB of the 16,756,681,056-byte pinned artifact (crates/muser-engine/src/lib.rs:14) — about 4.5 % of the model — but only one 3,744-byte row of it is read per token. Large in memory, trivial in bandwidth; both facts are true simultaneously. Put the other way, because the whole book rests on the distinction: a tensor’s size tells you what it costs to hold, and tells you nothing whatever about what it costs to use. The embedding table is the cleanest illustration in the model — three quarters of a gigabyte resident, under four kilobytes touched per token — and Ch 1 hangs its whole budget on exactly that gap.

11.4 The Metal kernel

Now the code. Read it holding one question: which line is the actual gather? There is exactly one, a pointer expression, and it is outnumbered about a dozen to one by nibble bookkeeping. That ratio is not a defect of this kernel. It is what quantization costs at every kernel in this book, visible here only because there is nothing else going on to hide it.

Here is the kernel, complete, with the dequant helper it calls:

// crates/muser-engine/src/shaders/muse_reference.metal:946
inline float muser_q4k_value(device const uchar *row, uint element) {
    uint block_index = element / 256;
    uint within_block = element % 256;
    uint group = within_block / 64;
    uint within_group = within_block % 64;
    uint scale_index = group * 2 + (within_group >= 32 ? 1 : 0);
    uint lane = within_group % 32;
    device const uchar *block = row + block_index * 144;
    uchar2 scale_min = muser_scale_min(block + 4, scale_index);
    uchar packed = block[16 + group * 32 + lane];
    uint quant = within_group < 32 ? uint(packed & 0x0f) : uint(packed >> 4);
    return muser_f16(block) * float(scale_min.x) * float(quant)
        - muser_f16(block + 2) * float(scale_min.y);
}

// crates/muser-engine/src/shaders/muse_reference.metal:961
kernel void muser_embedding_q4k(
    device const uchar *weights [[buffer(0)]],
    device const uint *token_ids [[buffer(1)]],
    device float *output [[buffer(2)]],
    constant uint &hidden_dim [[buffer(3)]],
    constant uint &vocab_size [[buffer(4)]],
    constant uint &tokens [[buffer(5)]],
    uint index [[thread_position_in_grid]]) {
    uint total = hidden_dim * tokens;
    if (index < total) {
        uint token_slot = index / hidden_dim;
        uint element = index % hidden_dim;
        uint token_id = min(token_ids[token_slot], vocab_size - 1);
        uint row_bytes = (hidden_dim / 256) * 144;
        output[index] = muser_q4k_value(weights + token_id * row_bytes, element);
    }
}

Line by line:

• One thread per output element. Thread index produces output[index]. For a single decode token that is 6,656 threads; for a 512-row prefill chunk it is 512 × 6,656 ≈ 3.4 M threads, all from the same kernel. token_slot = index / hidden_dim picks which token of the batch this thread serves.
• The gather itself is the pointer line weights + token_id * row_bytes: one integer multiply, one add. The min(token_ids[token_slot], vocab_size - 1) clamp is a bounds guard — a hostile or corrupt id reads the last row instead of running off the table.
• muser_q4k_value is the Q4_K dequant from Ch 6, spelled out scalar-style. Each element lives in one of the row’s 26 super-blocks (element / 256); each super-block has 8 sub-blocks of 32 (group = within_block / 64, two sub-blocks per 64 elements, one per nibble-half of each byte); muser_scale_min unpacks the 6-bit scale/min pair for the sub-block (muse_reference.metal:32-39); the nibble is the low or high 4 bits of one byte depending on which half of the sub-block the element falls in. The value is d × sc × nibble − dmin × m, where d/dmin are the two f16 header values of the super-block (muser_f16, muse_reference.metal:27-30).
• Nothing is cached across threads; each thread independently decodes the scale bytes it needs. That is ~26 super-block-header reads per thread in the worst case, all served from L2 (the GPU’s shared on-chip cache) after the first lane touches them — fine for a kernel that runs once per token.

Said the other way round, because this is the part that trips people up: an element index is really a four-level address — which super-block, which sub-block inside it, which byte inside that, and which half of that byte. The kernel stores none of those levels. No lookup table, no precomputed offsets: it re-derives all four from a single integer using divisions and masks, independently, in every thread. Quantized formats trade memory for address arithmetic, and this helper is that trade with the lid off.

A worked example: one element of one row

Take token id t = 42, element e = 300, and a made-up super-block whose header is d = 0.5, dmin = 0.01 and whose sub-block scales decode to sc = 40, m = 12, with the packed byte holding nibble 9 in the half that covers element 300. Then:

  block_index = 300 / 256 = 1            (the second super-block)
  within      = 300 % 256 = 44
  group       = 44 / 64   = 0            (first 64-element group)
  within_grp  = 44 % 64   = 44  ≥ 32  → high nibble, scale_index 1
  value       = d·sc·nibble − dmin·m
              = 0.5 × 40 × 9 − 0.01 × 12 = 180 − 0.12 = 179.88

Multiply that by 6,656 threads and the row is dequantized into hidden[0..6,656]. The numbers here are illustrative — the real bytes live in the pinned GGUF — but every index computation above is the kernel’s own arithmetic, quoted from muse_reference.metal:946-959.

And the thread-to-element picture for one token (Figure 11.2):

  thread index:    0     1     2    …   255 |  256   257  …        6,655
  token_slot:      0     0     0    …     0 | (token 1's threads, in batch mode)
  element:         0     1     2    …   255 |    0     1  …          6,655
  reads:        row 42, bytes 0..3,744 — every thread re-derives its own
                block/group/lane indices into the same contiguous row

Figure 11.2: the per-thread decomposition for tokens = 1. Grid is the output shape (dispatch_1d(hidden_dim × tokens)); the gather is one pointer computation per thread.

11.5 The Rust dispatch

The kernel says what to compute. It does not say how many threads run it, where the weight bytes come from, or what should happen if the buffer you bound turns out to be the wrong size. That is the wrapper’s job. It is the layer where a mistake is expensive and silent, which is why it is also the layer carrying the assertions.

The wrapper is encode_embedding_q4k:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:338
pub fn encode_embedding_q4k(
    &self,
    encoder: &ComputeCommandEncoderRef,
    weights: GpuByteView<'_>,
    token_ids: &GpuByteView<'_>,
    output: &GpuBuffer,
    hidden_dim: usize,
    vocab_size: usize,
    tokens: usize,
) {
    // …(F16-table branch elided: a 2-byte-per-element table dispatches
    //    `muser_embedding_f16` instead — see file)…
    let row_bytes = hidden_dim / 256 * 144;
    debug_assert_eq!(weights.len(), row_bytes * vocab_size);
    debug_assert_eq!(token_ids.len(), tokens * std::mem::size_of::<u32>());
    debug_assert_eq!(output.len(), hidden_dim * tokens);
    self.bind(encoder, "muser_embedding_q4k");
    encoder.set_buffer(0, Some(weights.metal()), weights.offset() as u64);
    encoder.set_buffer(1, Some(token_ids.metal()), token_ids.offset() as u64);
    encoder.set_buffer(2, Some(output.metal()), 0);
    set_value(encoder, 3, &(hidden_dim as u32));
    set_value(encoder, 4, &(vocab_size as u32));
    set_value(encoder, 5, &(tokens as u32));
    dispatch_1d(encoder, hidden_dim * tokens);
}
}

And the call site — the first dispatch of every token graph:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5523
dispatch(command, |encoder| {
    self.kernels.encode_embedding_q4k(
        encoder,
        self.embedding.view(&self.mapped_weights),
        token_view,
        &self.activations.hidden,
        cfg.hidden_dim,
        cfg.vocab_size,
        1,
    );
});
}

Three details matter, and each one is a cost that isn’t there. First, self.embedding.view(&self.mapped_weights): the table is a view into the single mmap’d GGUF buffer already mapped onto the GPU (decode.rs:1201), so the “upload” of 756 MB of embedding is a no-op — the zero-copy promise of Ch 3, collecting for the first time. Second, the token id arrives as token_view — four bytes in a small staging buffer the CPU wrote before the command buffer existed (decode.rs:5433-5438), so the GPU never waits on the host. Third, the grid: dispatch_1d(hidden_dim × tokens) (encode.rs:1337-1343) uses dispatch_threads with a threadgroup width of at most 256 — for one token, 6,656 threads in 26+ threadgroups of 256. The GPU, not the CPU, sizes the launch.

The batch-graph sibling encode_embedding_q4k_from_u32_buffer (qkv.rs:376) is the same kernel with the id buffer already GPU-resident — that is the variant the 512-row prefill chunks use, and the reason one kernel serves both regimes.

11.6 The access pattern

Every kernel chapter reaches this point and asks the same question: where does the time go? For the embedding the honest answer is “nowhere,” and it is worth seeing precisely how nowhere, because the shape of the accounting is the one we will reuse on kernels where the answer is not so comfortable.

Per token:

  read : 1 row × 3,744 B (Q4_K)  +  4 B (token id)   ≈ 3.75 KB
  write: 6,656 × 4 B = 26,624 B ≈ 26 KB (f32 hidden)

The read is a single contiguous 3,744-byte span; the write is a single contiguous 26 KB span. Against the ~16.76 GB of weight bytes the token graph reads in total (lib.rs:14), the embedding row is 3,744 / 16,756,681,056 ≈ 2 × 10⁻⁷ of the traffic. The activation write is larger than the weight read here — the only chapter in Part IV where that is true — and both are rounding error. Note also what is not read: the other 202,047 rows of the table stay untouched until some other token id asks for them.

The F16 sibling and the lane matrix

The dispatch you just read has an elided branch: if the table’s byte length says two bytes per element, the wrapper binds muser_embedding_f16 instead (qkv.rs:348-359). Same one-thread-per-element shape, same gather, no dequant — a 2-byte half is widened to f32 directly. Which branch runs is a property of which lane you loaded, not of any flag: the kquant (q4_k_xl) artifact carries the table quantized, and a lane whose tables ship F16 takes the sibling. The lane matrix of Ch 7 decides; the kernel follows the bytes. That is the book’s first example of a pattern that recurs through Part IV — the dispatch adapts to the artifact, while the graph structure (one gather, one norm, four matvecs, …) stays fixed.

11.7 Tradeoffs

GPU gather vs CPU lookup — Muser inverts the ancestor’s choice. We inherited a decision here, and then we reversed it. The Ferrite book did the embedding on the CPU — one quantized row dequantized into a unified-memory buffer, no dispatch at all — and argued the work was too small to amortize a GPU dispatch over [ferrite-book Ch 9]. That argument is sound on its own terms, and going in we expected to keep the CPU path: it is less code, and none of the arithmetic had changed.

What had changed was everything around the arithmetic. Muser runs the lookup on the GPU anyway. Two structural facts, visible in the code rather than in any A/B ledger, turn the cheap-looking CPU path into the expensive one:

1. The token id is already a GPU buffer. Muser’s decode graph is one command buffer from embedding to softcap (decode.rs:5448-5460), fed by teacher-forced or batched token arrays that arrive as staging buffers. A CPU lookup would need to read the id back, dequantize 3,744 B, and write 26 KB through unified memory — crossing the host/device boundary to save one dispatch that was going to be recorded anyway.
2. One kernel serves every batch width. The same muser_embedding_q4k covers 1-token decode, 64-row teacher-forced blocks, and 512-row prefill chunks (qkv.rs:376’s tokens parameter); a CPU path would need a per-width host loop.

There is no measured A/B of CPU-vs-GPU embedding in the campaign ledger [unverified] — the choice is justified structurally, and at ~30 KB of traffic either side would be invisible against the per-token budget.

The lesson we took from the reversal is worth carrying into the rest of the kernel chapters: a dispatch’s cost is not a property of the dispatch. It is a property of what is already queued beside it. An argument that held for an engine recording a handful of command buffers stops holding for one that records the entire token graph as a single buffer.

Q4_K table vs an f32 table. Pre-dequantizing the table to f32 at load would cost 202,048 × 6,656 × 4 ≈ 5.4 GB of resident memory — about a third of the whole artifact again — to save 3,744 bytes of dequant work per token. The loader instead accepts the table in exactly the dtypes the pinned artifact carries (Q4_K or F16, decode.rs:1209-1214) and dequantizes one row on demand. The same trade the ancestor book described, at 7× the scale [ferrite-book Ch 9].

The clamp instead of a fail. A Metal kernel has no way to raise an error. A thread handed a bad index simply reads whatever those bytes happen to be. So what should a shader do with a token id that is out of range? Muser’s answer is the min on the pointer line: an out-of-range id silently maps to the last row rather than erroring (muse_reference.metal:973). It is not the correctness gate — the CPU-side session validates token ids before the graph is ever built (Session::decode, crates/muser-engine/src/api.rs:696-741), so by the time a thread reaches the clamp a bad id should already be impossible. That makes the clamp a second line of defense. It also makes it a silent one, and a silent defense is a defense you never learn has fired. Which of those two descriptions the clamp was actually written for is [unverified] from the tree alone.

The untied twin at the other end of the model

token_embd.weight is not the only [6,656 × 202,048] tensor in the checkpoint. output.weight — the LM head — has the same shape and is a separate tensor: Muse Glimmer’s embeddings are untied, and the loader demands both (config.rs:295-297). The two tensors live opposite lives. The embedding table is read one row per token on the GPU, ~3.75 KB at a time (this chapter). The LM head is read as a full matvec over all 202,048 rows, every token, in the tail of the graph (decode.rs:5892-5897) — at 2 bytes/element on the native lane that is ~2.7 GB of traffic and, measured, ~3.46 ms/token versus ~1.75 ms for the kquant lane’s quantized head [docs/nvfp4-fast-lane-evidence §Measured product numbers]. Same shape, three orders of magnitude different bandwidth. The LM head gets its own chapter, Ch 20; this chapter’s table is the quiet one.

11.8 Where the gap lives

Every kernel chapter closes by facing the same suspicion: is this the one eating the decode time? Here the answer is a flat no, and the accounting is specific enough to say exactly why.

This kernel is not the gap. The dispatch-gap accounting of [docs/decode-dispatch-gap-20260815.md] reconciles the production-vs-legacy closure delta (760 vs 564, +196 at position 2,048) into exactly four families: 104 norm-boundary groups, 39 SWA staging groups, 52 KV-publication splits, and 1 last-row copy — the embedding appears in the common math row (406 closures, delta 0, “required math / Keep”). Its bytes are nanoscopic and its one-dispatch cost is shared by both graphs. If you are hunting decode time, this is the last place to look.

11.9 What comes next

The residual stream now holds 6,656 f32s and the real work can begin. The first consumer is the entry RMSNorm — and on Muse Glimmer the norm story is unusual: a dual-epsilon sandwich with a llama.cpp hard-coded 1e-8 that the GGUF does not even carry. That is Ch 12.

References

• crates/muser-engine/src/shaders/muse_reference.metal:946-959 — muser_q4k_value, the scalar Q4_K dequant used by the embedding (and the batch matvec fallbacks).
• crates/muser-engine/src/shaders/muse_reference.metal:961-977 — muser_embedding_q4k, the kernel this chapter dissects.
• crates/muser-engine/src/shaders/muse_reference.metal:27-39 — muser_f16 and muser_scale_min (the f16 header read and the 6-bit scale/min unpack).
• crates/muser-engine/src/metal/encode/qkv.rs:338-373 — encode_embedding_q4k, the dispatch wrapper; :361 the row_bytes formula; :376-411 the batch sibling.
• crates/muser-engine/src/metal/encode.rs:1337-1343 — dispatch_1d (the dispatch_threads grid used here).
• crates/muser-engine/src/decode.rs:5523-5533 — the call site, first dispatch of encode_token.
• crates/muser-engine/src/decode.rs:1207-1216 — embedding loaded as a Projection from token_embd.weight; dtype gate (Q4_K or F16).
• crates/muser-engine/src/decode.rs:5433-5438 — the four-byte token staging view the kernel reads.
• crates/muser-engine/src/config.rs:294-297 — token_embd.weight shape [hidden_dim, vocab_size] asserted at load.
• crates/muser-engine/tests/muse_golden.rs:96,101 — hidden_dim = 6,656, vocab_size = 202,048 asserted against the pinned GGUF.
• crates/muser-engine/src/lib.rs:14 — the 16,756,681,056-byte artifact.
• crates/muser-engine/src/api.rs:696-741 — Session::decode, the host-side id validation that runs before any of this.
• [docs/decode-dispatch-gap-20260815.md] — the closure reconciliation this chapter’s gap section cites (embedding in the 406-closure common-math row).
• Ch 3 — mmap + zero-copy views.
• Ch 6 — the Q4_K super-block this kernel dequantizes.
• Ch 10 — where this dispatch sits in the one-command-buffer token graph.
• Ch 12 — the next kernel.
• Ch 20 — the untied LM head, the other boundary tensor.
• [docs/nvfp4-fast-lane-evidence §Measured product numbers] — the ~3.46 ms/token F16 LM head vs ~1.75 ms kquant comparison (§11.7).
• [ferrite-book Ch 9] — the ancestor’s CPU-lookup embedding and the too-small-to-amortize argument (pedagogical lineage only).

Chapter 12 — RMSNorm and the dual-epsilon sandwich

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 (SIMD groups, simd_sum, threadgroup_barrier), Ch 9 (the sandwich-norm architecture), Ch 10 (the per-layer kernel chain), Ch 11 (the residual stream’s birth). This chapter defines normalization from zero; no transformer-paper background is assumed.

Chapter 11 ended with the residual stream freshly born — 6,656 f32s from one quantized table row — and named its first consumer: the entry RMSNorm. This is the norm story, and on Muse Glimmer it is an unusual one: two different epsilons, a llama.cpp constant the checkpoint does not carry, and a fused kernel that exists because fusing carelessly would change the model’s public numbers.

Normalization is the least glamorous operation in a transformer. It invents no information; it only puts the numbers back into a range the next matmul can work with. So it is a good joke on us that this is the chapter where we lost the most work: two of the optimizations we were most confident about survive here as war stories rather than as shipped code.

12.1 What it computes

RMSNorm takes a vector x of length n and produces a vector of the same length whose magnitude is predictable — close to unit scale — regardless of how large or small x was:

mean(x²) = (1/n) Σⱼ xⱼ²                    (mean of squares)
rms      = √(mean(x²) + ε)                 (root-mean-square, ε-guarded)
y_i      = x_i / rms × γ_i                 (normalize, then per-channel gain)

Three of the symbols in those lines are settled by the checkpoint, and one is not. On Muse Glimmer n = hidden_dim = 6,656 for the stream norms, and γ is a learned per-channel weight — attn_norm.weight, ffn_norm.weight, and their post-norm siblings (crates/muser-engine/tests/muse_golden.rs:96, config.rs:300-314).

Then there is ε, and ε is the reason this chapter exists: it is not one value but two. 1e-5 is the value the checkpoint carries, and it governs every norm the GGUF declares. 1e-8 is hard-coded for the two “post” norms of the sandwich — a llama.cpp graph constant the checkpoint does not carry at all (config.rs:23-28).

Hold on to that, because it decides the shape of the code further down. A fused kernel that assumes a single epsilon per layer cannot serve this model, so one kernel in this chapter, muser_fused_norm_residual_rms_norm_32sg, exists for no other reason (shaders/ferrite/rmsnorm_batch_tail.metal:142-146).

12.2 Why it exists — 52 layers of multiplication drift

Why spend four normalizations per layer on an operation that adds no information to the stream? Ask it the other way: what breaks if we leave them out?

A transformer layer is a chain of multiplies. The matvec of Ch 13, an activation, another matvec — and at every step the numbers on the residual stream get multiplied together. Multiplications compound. If each of Muse Glimmer’s 52 layers scaled the stream by 1.2, after the stack the magnitudes have grown by 1.2^52 ≈ 39,000. If each scaled by 0.8, the signal has shrunk to 0.8^52 ≈ 1.4 × 10⁻⁵. Either way the network drifts out of a usable range, and the deeper layers see garbage. Normalization is the valve: it pins the magnitude back to a known band before each sub-block, so the layer stack stays numerically stable. Skip it and the model still runs but its late-layer arithmetic saturates or vanishes — on this model, exact-parity decode would be the first casualty, because every ULP that drifts here is amplified by everything downstream.

Muse Glimmer is a Gemma-2-style sandwich: each layer normalizes four times, not two (config.rs:300-314 names all four tensors):

  attn_norm (1e-5) → attention → post_attention_norm (1e-8) → ffn_norm (1e-5)
                                 → FFN        → post_ffw_norm    (1e-8)
                                 → next layer's attn_norm        (1e-5)

The two “post” norms normalize each sub-block’s output before it is added to the residual; the “pre” norms normalize the stream before each sub-block reads it. That is the sandwich — pre-norm, block, post-norm, residual — and the two epsilons live on opposite sides of it.

Why RMS and not standard deviation?

LayerNorm centers and scales: it subtracts the mean and divides by the standard deviation:

μ = (1/n) Σ xⱼ ;  σ² = (1/n) Σ (xⱼ − μ)² ;  y_i = (x_i − μ)/√(σ²+ε) × γ_i

RMSNorm drops the mean subtraction and divides by the root-mean-square only. Two reasons, one costed and one empirical:

1. Cost. LayerNorm needs two global reductions over x — one for μ, one for σ² (which itself needs μ). RMSNorm needs one: the sum of squares. On a GPU a reduction is the expensive part of normalization — it is the one place every thread must wait for every other thread — so one reduction instead of two is a real saving, repeated 100+ times per token on this model.
2. Empirical equivalence. The RMSNorm paper reports that dropping the mean changes downstream quality negligibly across the models tested [arxiv:1910.07467]. [unverified] for Muse Glimmer specifically — that is the paper’s claim, not a measurement in this campaign; the architecture chose RMSNorm upstream and Muser implements what the GGUF declares.

A useful identity links the two: mean(x²) = σ² + μ², so rms = √(σ² + μ²) ≥ σ, with equality exactly when μ = 0. When x is already mean-centered, dividing by rms and dividing by σ are the same operation — the sense in which the two norms’ scaling steps coincide.

The two epsilons, precisely

Here is the fact the whole chapter turns on. Where does each epsilon come from, and what happens to an engine that assumes there is only one?

• rms_eps = 1e-5 is read from the GGUF key muse-glimmer.attention.layer_norm_rms_epsilon; absence is a hard load error (config.rs:184-188). It feeds every norm this chapter dispatches except the two post norms.
• post_norm_eps = 1e-8 is not in the GGUF. llama.cpp’s graph builder hard-codes it (src/models/muse-glimmer.cpp:67, const float post_norm_eps = 1e-8f;), and Muser mirrors the constant with a doc comment saying exactly that (config.rs:23-28). This is a rare thing: a numerical constant that is part of the comparator’s identity rather than the checkpoint’s. An engine that read 1e-5 everywhere — the “obvious” simplification — would disagree with pinned llama.cpp in the last bits of every post-norm, and the parity gates of Ch 38 exist to catch exactly that class of drift.

The ε inside the root has one job: if x is all zeros, mean(x²) = 0 and √0 = 0, which would divide by zero. + ε keeps the denominator strictly positive.

Why two different tiny values matter numerically is harder to say — the difference between √(m + 1e-5) and √(m + 1e-8) is invisible for any m of real magnitude — but the exactness contract does not grade on “of real magnitude”: it grades on bits, and llama’s graph says 1e-8, so 1e-8 it is. Put that another way, because it is the rule behind every decision in this chapter: we are not implementing normalization here, we are implementing llama’s normalization. Where the two differ, the comparator wins — even when it has no numerical argument on its side. The deeper motivation for the sandwich itself is a model-design question this engine inherits [unverified].

12.3 RMSNorm by hand: x = [3, 4, 0, 0], ε = 1e-5

Take n = 4, x = [3, 4, 0, 0], ε = 1e-5, and first γ = [1,1,1,1]:

 Step 1 — sum of squares
   x² = [9, 16, 0, 0] ;  Σx² = 25 ;  mean(x²) = 25/4 = 6.25

 Step 2 — root-mean-square and its reciprocal
   rms     = √(6.25 + 0.00001) ≈ 2.500002
   inv_rms = 1/rms            ≈ 0.3999997   (call it 0.4)

 Step 3 — normalize, then apply γ
   y_i = x_i × inv_rms × γ_i
   y   = [3×0.4×1, 4×0.4×1, 0, 0] = [1.2, 1.6, 0, 0]

Figure 12.1: RMSNorm worked example. The input whose RMS was 2.5 comes out with RMS 1 (before γ).

Check: mean(y²) = (1.44 + 2.56)/4 = 1, so rms(y) = 1. Now change γ to [2, 0.5, 1, 1]: y = [2.4, 0.8, 0, 0] — the RMS is no longer 1, on purpose. Normalization fixes the scale; γ re-learns the shape. On Muse Glimmer, one special γ is programmer-chosen rather than learned: the entry norm binds an all-ones vector (decode.rs:1216), because the graph needs the RMSNorm operation but the model has no entry-norm tensor.

Everything the kernels do below is a parallel implementation of those three steps, at n = 6,656.

12.4 The Metal kernels

That was arithmetic you could do on paper. What follows is the same three steps run by a thousand threads at once — and what separates the kernels below is not the math, which is identical in all of them, but which other implementation each one has to agree with, bit for bit.

Four kernels from two lineages serve the stream norms. Read them in order of increasing specialization.

12.4.1 rms_norm_batch — the ferrite-lineage base kernel

Start with the plainest of the four: one row in, one row out, one epsilon, and nobody to agree with. This is the unfused workhorse, byte-for-byte from the Ferrite shader pull at a85048a90 (docs/extraction-manifest.md):

// crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:1
kernel void rms_norm_batch(
    device const float* x      [[ buffer(0) ]],  // [B × n]
    device const float* weight [[ buffer(1) ]],  // [n] (shared)
    device       float* out    [[ buffer(2) ]],  // [B × n]
    constant     uint&  n      [[ buffer(3) ]],
    constant     float& eps    [[ buffer(4) ]],
    uint tgid [[ threadgroup_position_in_grid ]],
    uint tid [[ thread_index_in_threadgroup ]],
    uint sgitg [[ simdgroup_index_in_threadgroup ]],
    uint lid [[ thread_index_in_simdgroup ]],
    threadgroup float* shared [[ threadgroup(0) ]])
{
    const uint batch = tgid;
    device const float* xb  = x   + batch * n;
    device       float* ob  = out + batch * n;
    device const float4* xb4 = (device const float4*)xb;
    device const float4* wb4 = (device const float4*)weight;
    device float4* ob4 = (device float4*)ob;
    const uint n4 = n >> 2u;

    float sum_sq = 0.0f;
    for (uint i = tid; i < n4; i += 128u)
        sum_sq += dot(xb4[i], xb4[i]);
    sum_sq = simd_sum(sum_sq);
    if (lid == 0u) shared[sgitg] = sum_sq;
    threadgroup_barrier(mem_flags::mem_threadgroup);
    if (tid == 0u)
        shared[4] = rsqrt((shared[0] + shared[1] + shared[2] + shared[3]) / float(n) + eps);
    threadgroup_barrier(mem_flags::mem_threadgroup);
    const float inv_rms = shared[4];
    for (uint i = tid; i < n4; i += 128u)
        ob4[i] = xb4[i] * inv_rms * wb4[i];
}

This is the reduction pattern of Ch 2 in miniature, and every kernel in this chapter repeats it:

1. Vectorized load — the float4 cast reads four floats per transaction; n4 = n/4 iterations instead of n.
2. Strided partial sum — thread tid walks i = tid, tid+128, …, accumulating dot(xb4[i], xb4[i]) (a 4-way sum of squares per load).
3. simd_sum — one hardware instruction collapses 32 lanes to their sum, no barrier, in-lockstep.
4. Threadgroup handoff — lane 0 of each of the 4 SIMD groups publishes its group’s partial into shared[sgitg]; one barrier makes all four visible.
5. Single-thread combine — thread 0 sums the four partials and computes the reciprocal root; a second barrier broadcasts it.
6. Normalize-and-scale — every thread re-walks its elements and writes x × inv_rms × γ.

Two barriers per row is the price of crossing SIMD-group boundaries; simd_sum inside a group is free. Note rsqrt(...) — that single function name is load-bearing, and §12.7 comes back to it.

12.4.2 The pinned ggml norm — kernel_rms_norm_mul_f32_4

So which of the four actually fires when you serve a token? Not, as a rule, the one you have just read.

On the serving path the standalone norm almost never runs, because encode_rms_norm_mul prefers the pinned llama.cpp metallib kernel whenever the library is loaded and n is a multiple of 4:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/norm.rs:244
pub fn encode_rms_norm_mul(
    &self, encoder: &ComputeCommandEncoderRef,
    input: &GpuBuffer, weight: &GpuBuffer, output: &GpuBuffer,
    dim: usize, eps: f32, rows: usize,
) {
    // …(cross-vendor strict branch elided — §12.7)…
    if dim.is_multiple_of(4) {
        if let Some(pipeline) = self.ggml_rms_norm_mul() {
            self.encode_ggml_rms_norm(
                encoder, pipeline, input, weight, weight, output, dim, eps, rows, false,
            );
            return;
        }
    }

    self.bind(encoder, "rms_norm_batch");
    // …(buffers 0–2, dim, eps; elided)…
    encoder.set_threadgroup_memory_length(0, 32);
    encoder.dispatch_thread_groups(MTLSize::new(rows as u64, 1, 1), MTLSize::new(128, 1, 1));
}
}

The ggml pipeline is kernel_rms_norm_mul_f32_4 from the metallib pinned by MUSER_GGML_METALLIB (metal/encode.rs:287, loaded per Ch 4), and encode_ggml_rms_norm (norm.rs:53-94) packs llama’s own GgmlMetalKargsNorm argument struct and picks llama’s own thread count — (dim/4).next_power_of_two().clamp(32, 1024), which is 1,024 threads at dim = 6,656 (norm.rs:89). The ferrite-lineage rms_norm_batch at 128 threads is the fallback when the metallib is absent. Same math, two reduction shapes — and since floating-point addition is not associative, the two shapes can differ in the last bits.

Read that last clause slowly; the rest of the chapter is built on it. The same values, added in a different order, are not the same float. A reduction shape is therefore not an implementation detail we are free to choose — it is part of the answer. On the serving route the pinned one is the point.

12.4.3 The dual-eps fused tail — muser_fused_norm_residual_rms_norm_32sg

The heart of the chapter. After each sub-block, the graph must do three things to the stream: add the block’s post-norm-normalized output into the residual, then produce the next sub-block’s pre-normed input. The fused kernel does all three in one dispatch, with two different epsilons:

// crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:142
// Decode-only dual-epsilon tail fusion:
//   hidden += rms_norm(src, eps1) * weight1
//   output  = rms_norm(hidden, eps2) * weight2
// Muse uses eps1=1e-8 for sandwich post-norms and eps2=1e-5 for the
// following pre-norm, so the older single-epsilon batch kernel is not valid.
kernel void muser_fused_norm_residual_rms_norm_32sg(
    device float* hidden [[buffer(0)]],
    device const float* src [[buffer(1)]],
    device float* output [[buffer(2)]],
    device const float* weight1 [[buffer(3)]],
    device const float* weight2 [[buffer(4)]],
    constant uint& n [[buffer(5)]],
    constant float& eps1 [[buffer(6)]],
    constant float& eps2 [[buffer(7)]],
    uint row [[threadgroup_position_in_grid]],
    uint tid [[thread_index_in_threadgroup]],
    uint sgitg [[simdgroup_index_in_threadgroup]],
    uint lid [[thread_index_in_simdgroup]],
    threadgroup float* shared [[threadgroup(0)]]) {
    const uint n4 = n >> 2u;
    device float4* hidden4 = (device float4*)(hidden + row * n);
    device const float4* src4 = (device const float4*)(src + row * n);
    device float4* output4 = (device float4*)(output + row * n);
    device const float4* weight14 = (device const float4*)weight1;
    device const float4* weight24 = (device const float4*)weight2;

    float sum_src = 0.0f;
    for (uint i = tid; i < n4; i += 1024u)
        sum_src += dot(src4[i], src4[i]);
    sum_src = simd_sum(sum_src);
    if (lid == 0u) shared[sgitg] = sum_src;
    threadgroup_barrier(mem_flags::mem_threadgroup);
    if (tid == 0u) {
        float total = 0.0f;
        for (uint group = 0u; group < 32u; ++group) total += shared[group];
        shared[32] = rsqrt(total / float(n) + eps1);
    }
    threadgroup_barrier(mem_flags::mem_threadgroup);

    const float inv_src = shared[32];
    float sum_hidden = 0.0f;
    for (uint i = tid; i < n4; i += 1024u) {
        const float4 value = hidden4[i] + src4[i] * inv_src * weight14[i];
        hidden4[i] = value;
        sum_hidden += dot(value, value);
    }
    sum_hidden = simd_sum(sum_hidden);
    if (lid == 0u) shared[sgitg] = sum_hidden;
    threadgroup_barrier(mem_flags::mem_threadgroup);
    if (tid == 0u) {
        float total = 0.0f;
        for (uint group = 0u; group < 32u; ++group) total += shared[group];
        shared[32] = rsqrt(total / float(n) + eps2);
    }
    threadgroup_barrier(mem_flags::mem_threadgroup);

    const float inv_hidden = shared[32];
    for (uint i = tid; i < n4; i += 1024u)
        output4[i] = hidden4[i] * inv_hidden * weight24[i];
}

Read it as §12.3 twice, with a residual add between:

• Pass 1 reduces src (the sub-block’s output, e.g. the o_proj result) and computes inv_src with eps1 = 1e-8 — the post-norm.
• Pass 2 does the residual update hidden += src × inv_src × weight1 and, in the same loop, accumulates the sum of squares of the updated hidden — the read of hidden and the write and the reduction share one pass over the data.
• Pass 3 computes inv_hidden with eps2 = 1e-5 — the next pre-norm — and writes the next sub-block’s input.

The reduction here spans 32 SIMD groups (1,024 threads), not 4: lane 0 of each group writes shared[sgitg], and thread 0 sums 32 slots serially. Why 32 groups for 6,656 elements? The dispatch wrapper’s comment says it outright:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/norm.rs:236
// 32 SIMD groups keep the 6,656-wide Muse tail resident and match the
// accepted Ferrite geometry. 33 floats are padded to Metal's 16-byte
// dynamic-threadgroup-memory alignment.
encoder.set_threadgroup_memory_length(0, 144);
encoder.dispatch_thread_groups(MTLSize::new(rows as u64, 1, 1), MTLSize::new(1024, 1, 1));
}

n4 = 6,656/4 = 1,664 float4s; 1,024 threads means each thread owns one or two float4s — the whole vector stays resident in registers with no second strided loop iteration for most threads. The threadgroup allocation is 33 floats (32 partials + 1 broadcast slot) = 132 bytes, padded to 144 for Metal’s 16-byte alignment. This kernel is a Muser addition to the Ferrite file — the muser_ prefix marks it — because the ancestor had no dual-epsilon model to need it.

12.4.4 The exact batch-graph tail — …_batch_dual_eps

If the kernel above already does the triple work with two epsilons, why is there a fourth? Because that kernel agrees with nobody. It is correct; it is not identical — and the distinction between these last two variants is the most important routing fact in this chapter.

The serving batch graph uses muser_fused_norm_residual_rms_norm_batch_dual_eps (rmsnorm_batch_tail.metal:250). It does the same triple work, and then goes out of its way to reproduce the two pinned ggml reductions bit for bit: a 32-SIMD-group reduction, 1.0f / sqrt(...) instead of rsqrt, llama’s exact multiply/add expression shape, and even a forced threadgroup_barrier(mem_flags::mem_device) between the two norms, so that the f32 device-memory publication boundary of the split route survives the fusion (rmsnorm_batch_tail.metal:277-303). None of that buys speed. Its comment records what it does buy: “changing any of these moved public logprobs beyond the contract in the rejected four-group fusion” — the failure the tradeoffs section below tells in full.

The call sites confirm the split: decode.rs:4513/4656 (batch graph) take the exact variant; decode.rs:5807/5878 (the legacy one-token encode_token graph this book narrates) take the 32sg variant.

12.5 The Rust dispatch — where the norms run in one token

Four kernels exist; one token has to choose among them dozens of times. So where do the norms actually fire, and how many of them stand alone? Fewer than the sandwich diagram earlier would lead you to guess — and the reason is the whole point of fusing tails.

From encode_token (decode.rs:5515-5906), the norm dispatches of one token are:

Layers 1..51 receive their attention pre-norm fused into the previous layer’s tail — the (next_norm, next_output) selection at decode.rs:5869-5876 binds the next layer’s attn_norm as the tail’s second weight, and for layer 51 it binds output_norm and writes the final norm straight to the buffer the LM head reads. Layer 0’s attn norm is the only standalone pre-norm in the graph, plus the entry norm before it (whose γ is the all-ones vector of §12.2). Counting dispatches per token: 2 standalone + 2×52 fused = 106 norm-carrying dispatches, each folding two norms in the fused case — the number the gap accounting of §12.8 reconciles. Two more norms per layer run through the same encode_rms_norm_mul wrapper, and they belong in this count because they spend from the same dispatch budget: the per-head QK-norms of Ch 14 add 2 dispatches per layer (104 per token) at width head_dim = 128 (decode.rs:5599-5618) — same math, tiny n, per-head γ that turns out to be a constant broadcast. That is the preview; Ch 14 owns the full story.

12.6 The access pattern

Every kernel chapter in this Part asks the same two questions of its kernel: what does it read, and is it bandwidth-bound? For the norms the second answer is no — but the arithmetic is worth doing anyway, because why it is no is what decides how the kernels above are tuned.

One row of one stream norm at n = 6,656: read x (26,624 B) + two γ vectors (26,624 B each for the fused tail) and write hidden (26,624 B) + output (26,624 B) — roughly 106 KB per fused tail, of which the γ weights are read by all 52 layers’ different tensors (52 × 2 × 26 KB of distinct weights per token ≈ 2.7 MB) while the activations are re-touched per layer. Across a token: ~2×52 fused tails ≈ 5.5 MB of activation traffic plus ~2.7 MB of γ — against the ~16.76 GB artifact stream (lib.rs:14), that is ~0.05 %. The norms are latency-motivated (keep 1,024 threads’ worth of the vector resident, cross as few barriers as possible), not bandwidth-motivated. The γ bytes are the only weight traffic here, and they are one-thousandth of the per-layer matvec weights of Ch 13.

12.7 Tradeoffs

The rejected 104-group fusion — the measured heart of this chapter. Stand at the fork with us. The gap accounting of §12.8 counts one separated norm-boundary closure on each side of every layer’s sandwich, and the obvious optimization is to fuse more: sweep those +104 closures into the tails and let one dispatch do the work of three. We expected a cheap win. The reduction would be the same reduction, the epsilons the same epsilons; only the dispatch boundary moves.

It did not survive the gate. The retained-activation hybrid preserved greedy tokens — every sampled token identical, which is exactly the result that makes a change look safe — and then breached the public numerical contract underneath them. Full-logit max absolute error came in at 4.6300888e-4 and normalized logprob max error at 3.197146176834309e-4, both above the 1e-4 contract, with 201,970 of 202,048 logits differing. Tracing backwards, the first KV difference was a single f16 ULP in layer 1’s value plane (bits 39,892 vs 39,893). We kept the postmortem: [docs/decode-dispatch-gap-20260815.md §Rejected hybrid postmortem], and the runs behind it: muser-receipt://pinned-token-parity-20260814-v{3,4}/.

The hybrid was removed rather than hidden behind a widened tolerance, and what it taught is narrower and more useful than “fusion is dangerous”. It was not wrong about the arithmetic. It was wrong about the order of the arithmetic — and order is what the contract grades. The corrected fusion took that lesson literally. The …_batch_dual_eps kernel of §12.4.4 reproduces the pinned ggml reductions exactly; it kept the baseline’s full-logit SHA-256 and measured 39.274 ms GPU against the 40.330 ms baseline (655 vs 760 closures) in a single-run diagnostic: [docs/decode-dispatch-gap-20260815.md §Landed and rejected reductions]. The rule we came away with is the one this chapter keeps repeating: a fusion is eligible only if it is bit-exact, and “almost the same reduction” is a different reduction.

rsqrt vs 1.0f / sqrt — one function name, contract-sized consequences. The next fork is far smaller than the last one: not a dispatch, not a kernel, a single function name inside a single line. rsqrt is Metal’s fast-math reciprocal square root — one hardware instruction, a few ULP short of the IEEE-rounded result; 1.0f / sqrt(x) is fully rounded. A few ULP on one norm is nothing. A few ULP on every norm of every layer, each one feeding the multiply that follows it, is not nothing.

The ferrite-lineage port rms_norm_llamacpp.metal exists entirely to document and fix this difference: its header explains that Ferrite’s default rsqrt(mean + eps) “differs from 1/sqrt by a few ULP per call,” that those ULPs “compound into knife-edge logit flips past ~50 tokens,” and that llama uses the multi-simdgroup reduction and 1.0f / sqrt (shaders/ferrite/rms_norm_llamacpp.metal:14-19, 96-101).

Muser’s serving answer is blunter than porting that fix: route the standalone norms through llama’s own metallib kernel (§12.4.2) and make the fused batch tail reproduce llama’s reduction bit for bit (§12.4.4). That leaves the 32sg kernel’s rsqrt where it can do no public harm, and it is one reason that kernel is confined to the legacy one-token graph — which, per the routing comment at decode.rs:2085-2091, is itself confined to teacher-forced and profiling duty because its fused kernels’ rounding “diverges from the source-pinned llama Metal graph enough to breach public logprob tolerance.”

One fused dispatch vs three split dispatches. If fusing is this dangerous, why fuse at all? Because the tail that is exact still pays for itself, and the bookkeeping is easy to read. Unfused, each tail is: post-norm (read src+γ1, write normed), residual add (read hidden+normed, write hidden), pre-norm (read hidden+γ2, write output) — 5 reads and 3 writes over 26 KB buffers plus two extra dispatch boundaries. The fused kernel: read src, γ1, γ2, hidden; write hidden, output — one pass fewer over the residual, two fewer dispatches, ×104 per token. That is the +104-closure saving the rejected fusion chased; the exact tail captures it for the two norm pairs it covers without touching the reduction order.

The cross-vendor seam — when the norm must split in two. One last fork, and it reframes the other three. Everything so far measured “exact” against pinned llama.cpp. Point the engine at a different partner and the same fused kernel becomes the wrong answer.

Under MUSER_CROSS_VENDOR_QK, every norm wrapper here reroutes to a decomposed no-fast-math pair — unweighted RMS in one dispatch, explicit learned-weight multiply in a second, with a barrier between (norm.rs:25-50, norm.rs:115-146). The reason is the disaggregated lane: vLLM’s producer materializes the weightless norm output in F16 and applies its scale as a second F16 operation, so the seam-exact receiver must preserve that intermediate rounding point (norm.rs:280-284). The fused kernel “loses that intermediate rounding point and is therefore not seam-exact.”

That is the sentence to leave this section on. A fusion that is exact against llama is still wrong against CUDA, because exactness is never a property a kernel has by itself — it is always relative to an anchor.

12.8 Where the gap lives

This chapter is the most direct resident of the gap in Part IV. The one-token dispatch-gap reconciliation counts 104 separated norm-boundary closures (52 post-attention + 52 post-FFN boundary pairs) in the production graph’s +196-closure delta — the largest single family [docs/decode-dispatch-gap-20260815.md §Corrected closure-count diff].

The accounting’s sharpest lesson lives here too. Those 104 groups look like pure waste, and they are the first thing anyone reading the profile wants to delete — but every cheap removal we tried changed bits (§12.7’s 3.197e-4 logprob breach), so they are kept, classed “fusible adjacent ops — existing fusion is not exact; reject.” That is what the retained evidence buys: not a faster engine, but the right to leave an obvious-looking inefficiency in place and be able to say exactly why.

The gap survived bit-exactness until the anchor itself changed — the J0/J1 story of Ch 38 and Ch 40. When a later chapter says “the norm boundary is the gap,” this table row is what it means.

12.9 What comes next

The stream is normalized and sits in activations.post_norm. Four weight matrices are about to read it — Q, K, V, and the attention gate — in the one concurrent dispatch set where the token’s real bandwidth bill starts running. That is Ch 13, the hero chapter of Part IV.

References

• crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:1-33 — rms_norm_batch, the ferrite-lineage base kernel.
• crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:142-201 — muser_fused_norm_residual_rms_norm_32sg, the dual-eps fused tail.
• crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:250-322 — muser_fused_norm_residual_rms_norm_batch_dual_eps, the pinned-exact batch-graph tail (and its reproduced-publication comment at :277-303).
• crates/muser-engine/src/shaders/ferrite/rms_norm_llamacpp.metal:1-37, 49-109 — the bit-exact llama port; the rsqrt-vs-1/sqrt header and the accumulation-order commentary.
• crates/muser-engine/src/metal/encode/norm.rs:244-278 — encode_rms_norm_mul (ggml-pinned preference, 128-thread fallback).
• crates/muser-engine/src/metal/encode/norm.rs:53-94 — encode_ggml_rms_norm (kargs packing, llama’s thread-count rule).
• crates/muser-engine/src/metal/encode/norm.rs:163-241 — the 32sg wrapper and its 1,024-thread / 144-byte geometry comment.
• crates/muser-engine/src/metal/encode/norm.rs:25-50, 280-303 — the cross-vendor split-norm seam (vLLM F16 boundary).
• crates/muser-engine/src/metal/encode.rs:287 — kernel_rms_norm_mul_f32_4 from the pinned metallib.
• crates/muser-engine/src/config.rs:23-28 — MUSE_POST_NORM_EPS = 1e-8 and the muse-glimmer.cpp:67 provenance comment.
• crates/muser-engine/src/config.rs:117-120, 184-188 — rms_eps/post_norm_eps fields; the required GGUF epsilon key.
• crates/muser-engine/src/config.rs:300-314 — the four per-layer norm tensors asserted at load.
• crates/muser-engine/src/decode.rs:5534-5544, 5806-5818, 5869-5889 — the entry norm, both fused-tail call sites, and the next-norm selection.
• crates/muser-engine/src/decode.rs:2085-2091 — the routing comment that confines the 32sg graph to teacher-forced duty.
• crates/muser-engine/src/decode.rs:1216 — the all-ones entry-norm γ.
• [docs/decode-dispatch-gap-20260815.md] — the +196 reconciliation (104 norm-boundary groups), the rejected hybrid postmortem, and the landed-reductions table.
• muser-receipt://pinned-token-parity-20260814-v3/, -v4/ — retained evidence for the rejected hybrid.
• [arxiv:1910.07467] — Zhang & Sennrich, Root Mean Square Layer Normalization (RMSNorm; the LayerNorm-equivalence claim).
• Ch 2 — simd_sum, barriers, threadgroup memory.
• Ch 14 — the per-head QK-norm at width 128.
• Ch 38 — the parity gates the 1e-8 constant protects.
• [ferrite-book Ch 10] — the ancestor’s RMSNorm chapter (worked example and reduction pedagogy ported from it).

Chapter 13 — The QKV + gate matvec family

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 (SIMD groups, dispatch), Ch 6 (Q4_K/Q5_K/Q6_K blocks and the pinned ggml kernel_mul_mv_q*_K_f32 kernels), Ch 9 (GQA 32:2, the sigmoid gate), Ch 12 (the normed input these projections read). This is deliberately the longest kernel chapter so far: this one operation family is where the token’s bandwidth bill is actually incurred, and where Muser’s parity discipline is most visibly different from “write a fast kernel.”

13.1 What it computes

Chapter 12 left the stream normalized in activations.post_norm, ready for the four weight matrices that read it. Before any code appears, ask the question this chapter exists to answer: when a token spends its time, where does the time go? Overwhelmingly, it goes here. This is the family those matvecs belong to — and the place where the token’s bandwidth bill is actually incurred. A matrix–vector multiply — matvec, or GEMV — is:

y = W · x        y_i = Σ_{j=0..cols-1} W[i,j] · x[j]     for each output row i

W is a learned weight matrix, x the input vector, y the output. A matvec is rows independent dot products, one per output row, each of length cols. The twist is that W is not f32: it is packed as a kquant format (Q4_K’s 144 bytes per 256 elements, Q5_K’s 176, Q6_K’s 210 — Ch 6), so every W[i,j] must be dequantized on the fly:

W[i,j] = d × sc_sub(j) × nibble(i,j) − dmin × m_sub(j)      (Q4_K form)

Muser never materializes a dequantized weight. The nibbles fly from DRAM through the ALU into an accumulator without ever landing in a buffer.

Muse Glimmer runs four of these matvecs per layer, as one concurrent dispatch set, all reading the same normed input (decode.rs:5569-5598):

Those shapes are not arbitrary; they fall out of the geometry the architecture chapter fixed. hidden = 6,656 sets every matrix’s input side. n_heads × head_dim = 32 × 128 = 4,096 sets the output side of Q and of the gate. n_kv_heads × head_dim = 2 × 128 = 256 sets K’s and V’s, narrow because grouped-query attention shares each KV head across many query heads. We read those extents off the golden test, which is where the anchor is kept [crates/muser-engine/tests/muse_golden.rs:96-100], and config.rs:300-307 asserts them again at load, so a checkpoint whose tensors disagree fails before it can produce a wrong number. Muse Glimmer is not a fused-QKV architecture: there is no single [6,656 × 4,608] QKV matrix. There are four independent tensors, dispatched as four independent matvecs that share one read-only input and write disjoint outputs — “one concurrent set,” in the call site’s own words (decode.rs:5566-5568). The gate is the fourth, unusual member; Ch 17 explains what it gates.

13.2 Why it exists — decode is matvec, not matmul

Matvec vs GEMM. A GEMM multiplies a matrix by a matrix of many columns. Prefill is a GEMM: the whole prompt is one batch of tokens, so each weight byte is reused across the batch (Ch 36). Decode is a matvec: there is exactly one token, the batch collapses to one column, and every weight byte is used exactly once before being discarded until the next token.

That single-use property is the whole memory problem of Ch 1. With no reuse across a batch, the work is compute-light (one multiply-accumulate per weight) and memory-heavy (the entire matrix must stream from DRAM). Decode is bandwidth-bound by construction, and the matvec family is the workload that streams the weights: strip the norms, RoPE, attention, and activations out of a token and what remains, line for line, is a sequence of quantized matvecs — these four, the o_proj, the FFN’s three, and the LM head.

13.3 The matrix operation, explained from zero

If you have not multiplied a matrix by a vector recently, do this 2×2 by hand (Figure 13.1). Take

   W (2×2)           x (2×1)        y (2×1)
  ┌          ┐       ┌     ┐        ┌     ┐
  │  1    2  │   ×   │  1  │    =   │  y₀ │
  │  3    4  │       │  5  │        │  y₁ │
  └          ┘       └     ┘        └     ┘

  y₀ = 1·1 + 2·5 = 11
  y₁ = 3·1 + 4·5 = 23

Figure 13.1: a 2×2 matvec by hand. Two dot products of length 2; two outputs. Every matvec in this chapter is this, with cols = 6,656 and rows from 256 to 202,048, and with W’s entries unpacked from nibbles.

Now the storage fact that shapes every kernel here: GGUF weight matrices are row-major — row 0’s elements are contiguous, then row 1’s, and so on. One output row’s weights are therefore one contiguous span of DRAM, and contiguous spans are what DRAM serves efficiently. The naive mapping — one thread per output row, walking all cols bytes serially — is correct but starves the memory system: a single thread cannot keep enough loads in flight (memory requests the hardware has accepted but not yet answered) to saturate the bus. Every real kernel in this family instead splits each dot product across the 32 lanes of a SIMD group and splits rows across threadgroups. The exact split differs per kernel; the shared skeleton is “dequant-and-MAC in the inner loop, simd_sum to combine the 32 partials, lane 0 writes the output row.”

One more piece of vocabulary you need for §13.4: ggml’s kargs. llama.cpp’s Metal kernels do not take rows/cols as friendly scalars — they take a packed C struct of tensor extents and byte strides (ne00, nb01, …), the same struct the CPU backend passes. Muser builds that struct in Rust:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:1292
impl GgmlKargsMulMv {
    fn for_matmul(rows: usize, cols: usize, block_bytes: usize, nr0: i32) -> Self {
        let row_bytes = (cols / 256 * block_bytes) as u64;
        Self {
            ne00: cols as i32,          // elements per row
            ne01: rows as i32,          // rows
            // …(strides elided: nb00 = block_bytes, nb01 = row_bytes, …)
            nr0,                        // rows reduced per threadgroup
            // …
        }
    }
}
}

(Fields elided; see the file for the full 112-byte struct.)

13.4 The kernels — three sources, one family

Ch 4 established that Muser runs kernels from three libraries. All three meet in this chapter’s dispatch.

13.4.1 The pinned ggml matvec — kernel_mul_mv_q*_K_f32

Start with the question that decides everything else in this section: whose code actually runs when Muse Glimmer projects a token? Not ours. The primary decode matvec is llama.cpp’s own kernel, loaded from the metallib pinned by MUSER_GGML_METALLIB (crates/muser-engine/src/metal/context.rs:122-131) at llama.cpp commit 89e0aa6fd362… (PINNED.md). Registration names the exact functions:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode.rs:278
ggml_q4k: ggml_matvec_pipeline(context, "kernel_mul_mv_q4_K_f32")?,
ggml_q5k: ggml_matvec_pipeline(context, "kernel_mul_mv_q5_K_f32")?,
ggml_q6k: ggml_matvec_pipeline(context, "kernel_mul_mv_q6_K_f32")?,
}

We do not quote these kernels’ bodies in this book — they are not in the Muser tree; they are pinned binary provenance, compiled from llama.cpp’s source at the comparator commit and built by scripts/compile_llama_metallib.sh (scripts/ per the repo’s tooling). What Muser owns is the dispatch, and here it is, the tokens == 1 branch of encode_quantized_matmul:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:429
if tokens == 1 {
    if let Some(pipeline) = self.ggml_matvec(dtype) {
        let (block_bytes, rows_per_group) = match dtype {
            GgmlType::Q4_K => (144, 2),
            GgmlType::Q5_K => (176, 1),
            GgmlType::Q6_K => (210, 2),
            _ => unreachable!("ggml_matvec returned only for K-quant projections"),
        };
        let args =
            GgmlKargsMulMv::for_matmul(n_out, n_in, block_bytes, rows_per_group as i32);
        encoder.set_compute_pipeline_state(pipeline);
        set_value(encoder, 0, &args);
        encoder.set_buffer(1, Some(weights.metal()), weights.offset() as u64);
        encoder.set_buffer(2, Some(input.metal()), 0);
        encoder.set_buffer(3, Some(output.metal()), 0);
        let simdgroups = 2usize;
        encoder.dispatch_thread_groups(
            MTLSize::new(n_out.div_ceil(rows_per_group * simdgroups) as u64, 1, 1),
            MTLSize::new(32, simdgroups as u64, 1),
        );
        return;
    }
    // …(standalone fallbacks: §13.4.2)…
}
}

Read the geometry off the dispatch:

• threadgroup (32, 2) — 64 threads = 2 SIMD groups, llama’s own N_SG for these kernels (the function constants pinned at registration bake nsg=2, encode.rs:844-851).
• grid ceil(n_out / (rows_per_group × 2)) — each threadgroup owns rows_per_group output rows: 2 rows per group for Q4_K and Q6_K, 1 for Q5_K, matching llama’s per-format nr0.
• Buffers ride in at llama’s slots: kargs at buffer(0), weights at 1 (with the mmap-arena offset), input at 2, output at 3.

For the Q projection: n_out = 4,096, rows_per_group = 2 → 4,096 / 4 = 1,024 threadgroups of 64 threads. For K and V: n_out = 256 → 64 threadgroups each. For the gate: 1,024 again.

Why dispatch llama’s kernels instead of writing better ones? We arrived at this fork holding evidence for the other branch. The ancestor line had written its own Q4_K GEMV and measured it: it beat llama’s kernel by 6–10 % in isolated A/B, with bit-identical output [ferrite-book Ch 11] — a Ferrite-lineage measurement on A18 Pro, never a Muser result. The expectation we carried into Muser was that the same trick would transfer, and that owning the inner loop would be strictly better than borrowing one.

That expectation did not survive contact with Muser’s gate, and the reason has nothing to do with speed. Muser’s gate is stricter than Ferrite’s was: full-logit and logprob parity against a pinned comparator, not token parity. Put plainly, matching the reference is not a property of the answer, it is a property of the route to the answer — reduction order, rounding points, accumulation tree and all. A hand-written kernel must therefore prove bitwise equality with llama’s route, and “bit-identical” is a claim you re-earn at every shape, every dtype, every batch width, forever. That is the lesson: the cheapest way to match a reference’s floating-point behavior is to run the reference’s own compiled code.

It helps that the prize was small anyway. Even in the ancestor’s own telling, the inner loop was proven not to be the engine’s gap — the win we would have been defending was never the one that mattered. Under Muser’s constraint, “pin the kernel” dominates “beat the kernel.”

13.4.2 The standalone fallbacks — muser_matvec_q4k_4r2s and friends

What if the pinned metallib is not on the machine at all? Muser does not refuse to run — it falls back to kernels it owns. That fallback is also the only member of this family whose source we can open on the page, which makes it the place to learn what the pinned kernel is doing behind its compiled wall. Keep one thing in reserve while you read it, though: the tradeoffs section returns to argue that “falls back” and “runs the same computation” are not the same sentence.

When the metallib is absent, the same wrapper falls back to Muser-owned kernels (qkv.rs:451-474): Q4_K routes to muser_matvec_q4k_4r2s (grid n_out/8, 64 threads), Q5_K to muser_matvec_q5k_4sg (grid n_out, 128 threads). The 4r2s kernel — “4 rows, 2 SIMD groups” — is the Muser-authored adaptation of the Ferrite lineage, and its body is worth reading because it shows the dequant fused into the MAC concretely:

// crates/muser-engine/src/shaders/muse_reference.metal:735
kernel void muser_matvec_q4k_4r2s(
    device const uchar *weights [[buffer(0)]],
    device const float *input [[buffer(1)]],
    device float *output [[buffer(2)]],
    constant uint &rows [[buffer(3)]],
    constant uint &cols [[buffer(4)]],
    uint group [[threadgroup_position_in_grid]],
    uint lane [[thread_index_in_simdgroup]],
    uint simd [[simdgroup_index_in_threadgroup]]) {
    uint block_count = cols / 256;
    uint row_bytes = block_count * 144;
    uint base_row = group * 8 + simd * 4;
    if (base_row >= rows) return;
    uint active_rows = min(4u, rows - base_row);
    float accumulator[4] = {0.0f, 0.0f, 0.0f, 0.0f};
    device const uchar *row[4] = {
        weights + ulong(base_row) * ulong(row_bytes),
        // …(rows +1..+3 elided: same pattern)…
    };
    for (uint block_index = 0; block_index < block_count; ++block_index) {
        for (uint row_index = 0; row_index < active_rows; ++row_index) {
            device const uchar *block = row[row_index] + block_index * 144;
            uint delta = *reinterpret_cast<device const uint *>(block);
            float d = float(as_type<half>(ushort(delta & 0xffff)));
            float dmin = float(as_type<half>(ushort(delta >> 16)));
            uint sd0 = *reinterpret_cast<device const uint *>(block + 4);
            uint sd1 = *reinterpret_cast<device const uint *>(block + 8);
            uint sd2 = *reinterpret_cast<device const uint *>(block + 12);
            float d_scale[8];
            float neg_min[8];
            muser_decode_all_q4k_scales(d, dmin, sd0, sd1, sd2, d_scale, neg_min);
            uint input_base = block_index * 256;
            for (uint quant_group = 0; quant_group < 4; ++quant_group) {
                uint packed = uint(block[16 + quant_group * 32 + lane]);
                float low = input[input_base + quant_group * 64 + lane];
                float high = input[input_base + quant_group * 64 + 32 + lane];
                accumulator[row_index] +=
                    fma(d_scale[quant_group * 2], float(packed & 0x0f), neg_min[quant_group * 2]) * low;
                accumulator[row_index] +=
                    fma(d_scale[quant_group * 2 + 1], float(packed >> 4), neg_min[quant_group * 2 + 1]) * high;
            }
        }
    }
    for (uint row_index = 0; row_index < active_rows; ++row_index) {
        accumulator[row_index] = simd_sum(accumulator[row_index]);
    }
    if (lane == 0) {
        for (uint row_index = 0; row_index < active_rows; ++row_index) {
            output[base_row + row_index] = accumulator[row_index];
        }
    }
}

(Rows +1..+3 of the row[4] initializer elided — identical pointer arithmetic at muse_reference.metal:750-755.)

The anatomy, in the order the kernel uses it:

• Four output rows per threadgroup, four per SIMD group’s lane set — base_row = group * 8 + simd * 4: 8 rows per threadgroup (2 SIMD groups × 4), each lane holding four private accumulators, one per row.
• Pre-decoded scales — muser_decode_all_q4k_scales (muse_reference.metal:41-62) unpacks all eight sub-block scale/min pairs once per super-block per row, folding in d and pre-negating the min: d_scale[j] = d × sc_j, neg_min[j] = −(dmin × m_j). The inner loop then uses them as constants.
• Byte-wise nibble read with fma — lane lane reads one byte block[16 + quant_group*32 + lane]; its low nibble covers one 32-element sub-block, its high nibble the partner sub-block (low/high input elements). Each element costs one fused multiply-add — fma(d_scale, nibble, neg_min) computes the fully dequantized weight with a single rounding — plus one multiply-accumulate into the row’s accumulator.
• simd_sum per row, lane 0 writes — the 32 lanes each covered 32 of every 256 elements; one hardware instruction per row recombines them.

There was a genuine fork inside that inner loop, and the ancestor line took both branches. What you just read is the pre-decoded-scales design point [ferrite-book Ch 12]: unpack every scale up front, then run a loop with nothing in it but loads and multiply-adds. The sibling v4 kernel chose deferred scaling instead — accumulate the raw nibble products and apply the scales once per sub-block at the end, spending fewer multiplies for a more tangled loop. Neither branch is wrong. They are two ways to spend the same arithmetic, and on their own terms both are correct.

The reason to care is downstream of correctness. Fold the scales in early and you round early; defer them and you round late. Same function, same inputs, different last bit. So the Muser fallback is not a slower copy of the pinned kernel — it is a different kernel that happens to agree to within a rounding step. The tree makes that policy explicit rather than letting it happen quietly: encode.rs:370-385 gives Q6_K no standalone fallback at all, “so its math and dispatch remain comparator-exact.”

13.4.3 The other two libraries, briefly

The cross-vendor library (strict-f32, no-fast-math recompile of muse_reference + nvfp4, context.rs:111-121) supplies muser_cross_vendor_q4k/q5k/q6k — CUDA-parity scalar routes for the disaggregated lane, gated by MUSER_CROSS_VENDOR_QK (qkv.rs:300-335). And the native NVFP4 lane replaces this whole kquant family with muser_nvfp4_*/muser_f16_matvec_c* kernels (qkv.rs:68-227); Ch 7 owns that story. The four-projection graph structure is identical on every lane — only the inner-loop kernels change.

13.5 The Rust dispatch — one concurrent set of four

Four matvecs read the same vector. Do they have to take turns? Nothing in the math says so, and nothing in the encode says so either — the permission to overlap is expressed by where the calls sit, not by a scheduler. encode_token records all four projections inside a single dispatch closure. The comment at the call site is the design statement:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5566
// llama.cpp and Ferrite issue the four independent attention
// projections as one concurrent set. They share a read-only input
// and mapped weight arena but write disjoint activations.
dispatch(command, |encoder| {
    self.encode_projection(
        encoder,
        &layer.q,
        &self.activations.post_norm,
        &self.activations.q,
        1,
    );
    self.encode_projection(
        encoder,
        &layer.k,
        &self.activations.post_norm,
        &self.activations.k,
        1,
    );
    // …(v and gate: identical shape, into activations.v / activations.gate;
    //    elided — decode.rs:5584-5597)…
});
}

encode_projection (decode.rs:6044-6089) is the dtype router: F16 → encode_f16_matmul, NVFP4 → encode_nvfp4_matmul, kquant → encode_quantized_matmul of §13.4. Each of the four calls lands on the same 26,624-byte read of post_norm — cheap after the first, because by then it is L2-resident — and on its own disjoint weight span and its own output buffer. That disjointness is the whole permission slip: no kernel here writes anywhere another kernel reads, so none of them needs to wait.

One closure, four kernel dispatches. That asymmetry matters beyond this page, because it is the kind of thing that quietly corrupts a measurement. Count closures in a profile, call the count “dispatches,” and you are wrong by a factor on exactly this line — and the QKV line is one of the biggest in the layer, so the error does not stay small. The gap note says it in its own words, and we kept the correction: decode-dispatch-gap-20260815.md §Instrumentation correction — “the qkvg closure encodes four kernel dispatches but contributes one profiler count”.

13.6 The access pattern — the budget, itemized

This is where the bandwidth story lives. The question is blunt: what does one token cost in bytes, and how much of that bill do these four matvecs sign? Per layer, per token, the four matvecs read:

  q    : 6,656 × 4,096 = 27,262,976 params
  k    : 6,656 ×   256 =  1,703,936 params
  v    : 6,656 ×   256 =  1,703,936 params
  gate : 6,656 × 4,096 = 27,262,976 params
                     ─────────────────────
  attention projections total = 57,933,824 ≈ 57.9 M params

At Q4_K’s 0.5625 bytes/param that is ≈ 32.6 MB per layer. Resist writing that figure down as the number, though, because it is not a property of the architecture: which tensors ship at which dtype is a property of the loaded GGUF. The loader reads a dtype per tensor as it maps the file, and the only anchor the tree gives us is that the release artifact’s FFN gate/up are Q4_K. Both receipts are retained — decode.rs:1294-1310 for the per-tensor read, decode.rs:5820-5821 for the anchor.

So parametrize rather than assert. At Q4_K bitrate the four matvecs are ~32.6 MB/layer; at Q5_K (0.6875 B/param) ~39.8 MB; at Q6_K (0.8203 B/param) ~47.5 MB. Now put the same layer’s other matvecs beside them — o_proj (27.3 M params), FFN gate/up/down (3 × 132.9 M params). Against the layer’s ~483.8 M projection parameters the attention set is ≈ 12 %, o_proj ≈ 6 %, and the FFN ≈ 82 %: the family this chapter is about is the smaller share of the layer’s traffic, which is worth knowing before you spend a week optimizing it. Scale by 52 layers and add the ~1.34 B-parameter embedding and LM head each, and the per-token stream is the artifact’s 16,756,681,056 bytes (lib.rs:14) — the number Ch 1 turned into a token-time budget. Every one of those bytes crosses the DRAM bus exactly once per token; this chapter’s kernels are the mechanism for four of the nine per-layer fractions of it.

Two structural notes on how the bytes are touched. First, the weights are a view into the single mmap’d GGUF arena (weights.offset() in every dispatch) — no staging, no per-tensor copies (Ch 3). Second, each output row’s bytes are contiguous (row-major), each threadgroup reads whole rows, and llama’s nr0 = 2 pairing means consecutive rows stream together — the access pattern is as close to “linear read of a big buffer” as the layout allows, which is precisely what a bandwidth-bound kernel wants.

13.7 Tradeoffs

Four separate matvecs vs a fused QKV kernel. The obvious alternative — one kernel reading all four weight matrices and writing all four outputs — saves three dispatch boundaries per layer (×52) and would let one weight fetch serve four accumulators. Muse Glimmer cannot take it: there is no fused QKV tensor to read, and synthesizing one would change the checkpoint. But the deeper answer is in the routing comment that governs which graph serves decode at all:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:2085
// The legacy one-token graph uses Ferrite fused residual/norm and
// gate-up kernels whose rounding diverges from the source-pinned
// llama Metal graph enough to breach public logprob tolerance.
// The one-row batch graph dispatches the exact pinned kernels and
// has the same KV transition, so it is the serving correctness
// path until each fused kernel independently passes full-logit
// parity.
}

Read that comment as the war story it is. We had the fused kernels already — the Ferrite-lineage fused residual/norm and gate-up path, fewer dispatch boundaries per layer, exactly the win the fused-QKV argument above is reaching for. We expected a free speedup, on the intuition that fusing two exact operations ought to stay exact. It was not free. The fused rounding diverged from the source-pinned llama Metal graph by enough to breach public logprob tolerance: no bug, no lost precision anyone could point at, just a different order of operations arriving at a different last bit.

The lesson is the one this chapter keeps re-teaching in new words. Fusion is not free even when it is numerically exact in isolation, because the standard here is not “exact” — it is “exact against the pinned comparator, per kernel.” A fused kernel that is more accurate than the reference still fails. So serving routes single tokens through the one-row batch graph (forward_batch, decode.rs:2092) — the same op sequence, running the pinned kernels — while the legacy encode_token graph this book narrates survives for teacher-forced harnesses and phase profiling (decode.rs:2124-2182, gated by MUSER_METAL_PHASE_PROFILE). The cost is paid in engineering discipline, not in tokens: Muser keeps two graphs precisely so the cheap one can be held out of serving until it proves parity.

Batch-width boundaries are numerical boundaries. For multi-token inputs the same wrapper switches kernels by token count, and the switch comment is the campaign in one paragraph:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:476
// Match the source-pinned llama.cpp Metal dispatch boundary exactly:
// K-quant projections with four through eight activation rows use
// `mul_mv_ext`, with a token-count-specific number of rows per
// threadgroup. This changes the floating-point reduction order, so
// substituting repeated decode GEMVs here breaks embedding/logprob
// numerical parity even when every other layer is identical.
}

Decode never leaves the single-token rung, so it is fair to ask why the rest of the ladder belongs in a decode chapter. It belongs because parity is not a decode-only property: the verify path and the prefill path enter through this same wrapper, and every rung of it is a different arithmetic.

At tokens 4..=8 the pinned kernel_mul_mv_ext_{q4,q5,q6}_K_f32_r1_{2..5} pipelines run (encode.rs:959-1003); at 16 rows the M16 n32 tiles; at larger multiples the SGM batch matmuls; at 512-row prefill chunks, the kernel_mul_mm_* GEMMs (qkv.rs:482-623). Every one of these is a different floating-point reduction order, and the wrapper keeps llama’s exact boundaries so the comparator’s bits are reproducible at every batch width. The DFlash verify path leans on the same ladder — its 16-row matmuls are why the M16 tile family exists at all (Ch 33).

Pin the kernel vs write a better one. Covered in §13.4.1, but it belongs in the tradeoff ledger with its labels straight: the 6–10 % isolated-kernel win is Ferrite-lineage evidence [ferrite-book Ch 11] (A18 Pro, Qwen2.5-1.5B), never measured on Muser’s M3 Ultra, and Muser’s gate (full-logit parity against pinned llama.cpp) is not winnable by a hand-written kernel that must then prove bitwise equality against llama’s own. The pinned-kernel strategy converts a hard numerical problem into a build-provenance problem — solved by the metallib receipt of Ch 4.

Fallbacks are correct, not equivalent. The standalone 4r2s/4sg kernels compute the same function but round differently (different lane decomposition, pre-decoded scales). Running without MUSER_GGML_METALLIB is therefore a different numerical lane, not merely a slower one — Q6_K’s lack of any fallback (encode.rs:370-385) makes that policy explicit rather than accidental.

13.8 Where the gap lives

Does this family contribute to the decode gap? There are two answers, and holding them apart is the whole point of the section.

In the closure accounting of [docs/decode-dispatch-gap-20260815.md], the answer is no. These four matvecs sit in the common math row — 406 closures, production delta 0, “required math / Keep.” They are not among the +196 closure families the note calls out (104 norm boundaries, 39 SWA staging, 52 KV-publication splits, 1 copy). There is nothing here to delete: no repeated matvec closure doing identical arithmetic exists to remove.

In the byte accounting of Ch 1, the answer is yes, and emphatically so — this family is the bulk of the budget. The token’s time is the time to stream 16.76 GB through kernels like these, and how close they sit to the machine’s bandwidth ceiling is the recurring question the book returns to at Ch 38.

Both answers are correct, because they answer different questions. “Is there wasted work here?” is a closure question, and the closures say no. “Is this where the time goes?” is a byte question, and the bytes say yes. Keeping those two framings from contaminating each other is exactly the epistemics the gap note exists to enforce.

13.9 What comes next

Q, K, V, and gate now hold raw projection outputs — un-normalized, un-rotated, positionless. Before attention can use them, each head’s slice must be QK-normalized and, on the 39 sliding layers, rotated by RoPE. On the 13 NoPE layers nothing rotates at all — and that asymmetry is about to become the most consequential layout decision in the engine. That is Ch 14.

References

• crates/muser-engine/src/metal/encode/qkv.rs:414-450 — the tokens == 1 pinned-matvec dispatch (quoted in §13.4.1); :451-474 the fallbacks; :476-508 the mul_mv_ext boundary; :550-623 the M16/SGM/GEMM batch ladder; :1292-1317 GgmlKargsMulMv.
• crates/muser-engine/src/metal/encode.rs:278-280 — the pinned kernel names; :837-866 ggml_matvec_pipeline (nsg=2 function constants); :959-1003 the mul_mv_ext groups; :370-385 supports_projection (Q6_K pinned-only).
• crates/muser-engine/src/metal/context.rs:122-131 — MUSER_GGML_METALLIB loading of the pinned llama.cpp metallib.
• crates/muser-engine/src/shaders/muse_reference.metal:735-788 — muser_matvec_q4k_4r2s (quoted in §13.4.2); :41-62 the scale decoder; :790-831 muser_matvec_q5k_4sg.
• crates/muser-engine/src/decode.rs:5566-5598 — the four-projection concurrent set; :6044-6089 encode_projection; :2085-2093 the serving-routing comment; :1294-1310 per-tensor dtype gating at load.
• crates/muser-engine/tests/muse_golden.rs:96-101 — the geometry (6,656 / 32 / 2 / 128) this chapter’s shapes derive from.
• scripts/compile_llama_metallib.sh — builds the pinned metallib.
• [docs/decode-dispatch-gap-20260815.md] — closure-vs-dispatch instrumentation; the +196 reconciliation this chapter’s gap section cites.
• [docs/goal-parity-ledger-2026-08.md] — the parity gates the pinned kernels exist to satisfy.
• Ch 1 — the 16.76 GB/token budget this family incurs.
• Ch 4 — the three kernel sources and the metallib receipt.
• Ch 6 — the block formats the kernels unpack.
• Ch 17 — what the gate output does.
• Ch 36 — the GEMM side of the same weights.
• [ferrite-book Ch 11, Ch 12] — the ancestor’s v4/4sg GEMV chapters (2×2 worked example, deferred-vs-pre-decoded scaling, and the 6–10 % inner-loop verdict — Ferrite-lineage, labeled as such).

Chapter 14 — QK-norm and RoPE — rotating only the layers that rotate

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 9 (the two layer classes, GQA), Ch 12 (the RMSNorm machinery this chapter reuses), Ch 13 (the Q/K/V/gate projections that produce this chapter’s inputs). No prior exposure to positional encodings is assumed.

14.1 What this chapter computes

Chapter 13 ended with Q, K, V, and gate holding raw projection outputs — un-normalized, un-rotated, positionless. This chapter is what happens to Q and K next, and it turns on a question that sounds trivial until you try to answer it inside a kernel: how does a machine built entirely out of dot products know which token came first? Muse Glimmer answers that question twice, differently, in the same forward pass — and the second answer is the one the rest of the book keeps cashing in. Two operations run between the projections and the KV store, and a third deliberately does not:

1. Per-head QK-norm — an RMSNorm applied to each attention head’s 128-wide slice of Q and K separately (rms_norm_per_head family; the live dispatch reuses Ch 12’s wrapper). On Muse Glimmer it is parameterless in effect: the weight tensors are converter-synthesized constant broadcasts, verified at load.
2. RoPE (Rotary Positional Embedding) — a per-position rotation of Q and K, applied only on the 39 sliding-window layers.
3. NoPE — the 13 full-attention layers ({3, 7, …, 51}) apply no positional rotation at all (config.rs:51-60). Their position information comes entirely from the causal mask and the KV layout.

In one formula, for one pair of coordinates of a head vector at position pos on a sliding layer, with pair frequency θ_i:

┌ x₀' ┐   ┌ cos(pos·θᵢ)  −sin(pos·θᵢ) ┐ ┌ x₀ ┐
└ x₁' ┘ = └ sin(pos·θᵢ)   cos(pos·θᵢ) ┘ └ x₁ ┘

Nothing is added, nothing is learned at runtime; each 2D pair is spun by an angle that grows with position. The payoff — proved in §14.4 — is that attention’s Q·K ends up depending only on the position difference.

The two layer classes make this chapter’s structure unusual among Muse engines: the position question has two different answers in one model, and the second answer (NoPE) is what makes the disaggregated lane of Part VI possible.

14.2 Why position must be injected at all

This is the part that trips people up the first time. Attention — the one operation that reaches across tokens (Ch 16) — is permutation-invariant. At its core it is dot products and a weighted sum, and a dot product does not care which token came first. So to raw attention, the sentences

the cat sat on the mat
mat the on sat cat the

look identical — same tokens, order differs, attention cannot see order. But order is the whole point of language. Something must tell the model “this token is at position 0, this one at position 5.” Skip it and the model is order-blind; it will produce plausible-looking gibberish that no logprob gate would forgive. The three classical schemes, in one paragraph each:

1. Absolute embeddings — learn one vector per position, add it to the token embedding. Simple; generalizes poorly past trained positions, and distances must be learned.
2. Relative biases — add a per-pair bias to each attention score. Generalizes better; modifies the attention kernel itself.
3. RoPE — rotate Q and K so their dot product depends only on the position difference. Relative position falls out of the math for free, and the attention kernel is untouched.

Muse Glimmer picks RoPE for its sliding layers — and picks nothing for its full layers, trusting the window/mask structure to carry order where the context is unbounded. Why a hybrid like this works is a model-design claim the engine inherits [unverified]; what the engine must do is implement each layer’s choice exactly.

14.3 QK-norm first — the parameterless cousin

Before anything spins, a smaller question has to be settled: are all the heads speaking at the same volume? Attention’s softmax is a competition, and a head whose Q and K happen to leave the projection with a large magnitude wins that competition for reasons that have nothing to do with meaning. QK-norm is the answer to that, and on this model it is a strange one — a normalization with no learned parameters, wearing a learned parameter’s clothes. Immediately after the projections, before any rotation, each head of Q and K is RMSNormed across its own 128 dimensions:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5599
dispatch(command, |encoder| {
    self.kernels.encode_qk_norm(
        encoder,
        &self.activations.q,
        &layer.q_norm,
        &self.activations.q,
        cfg.head_dim,
        cfg.rms_eps,
        cfg.n_heads,
    );
    self.kernels.encode_qk_norm(
        encoder,
        &self.activations.k,
        &layer.k_norm,
        &self.activations.k,
        cfg.head_dim,
        cfg.rms_eps,
        cfg.n_kv_heads,
    );
});
}

encode_qk_norm (metal/encode/norm.rs:286-303) delegates to encode_rms_norm_mul with dim = head_dim = 128 and rows = 32 (Q) or 2 (K) — i.e. the per-head norm is dispatched as a batched 128-wide RMSNorm whose rows are heads, in place. The dedicated rms_norm_per_head kernel exists in the pipeline registry (shaders/ferrite/rms_norm_per_head.metal:15, one threadgroup per head, tree reduction in threadgroup memory), and it is the right place to read the algorithm — but the live decode path reaches the same math through the shared norm wrapper of Ch 12 (whose pinned-ggml preference applies at this width too).

The Muse-specific wrinkle is the γ. The upstream checkpoint has no learned q/k norm weights; the GGUF converter materializes full(qk_scale_factor) for attn_q_norm.weight and ones(...) for attn_k_norm.weight so llama.cpp’s weighted-RMSNorm op can carry a scalar (config.rs:383-395).

That leaves a fork at load time, and it is worth walking both branches. The comfortable one is to shrug and multiply: the tensors are present, the norm kernel already takes a weight vector, so read whatever is in the file and move on. The uncomfortable one is to ask what happens the day the file stops being what we assume — the day a converter writes a genuinely learned per-channel norm into those same two slots. Nothing would crash. The kernel would happily consume a vector where we expected a broadcast, the math would quietly become a different model’s math, and the output would stay fluent. That is the failure mode this book keeps meeting: plausible text is not evidence of a correct engine.

Muser takes the uncomfortable branch and proves the assumption instead of holding it. QkNormProbe fails the load unless both tensors are exactly the constant broadcasts the converter emits (config.rs:397-403), so a learned per-channel norm aborts startup instead of silently redefining the attention scores (loader.rs:28-37). The same probe pins the values it accepts: qk_scale_factor ≈ 3.87 and k_norm = 1.0 (config.rs:139-141), with an arithmetic companion test at config.rs:426-430 checking 3.87 × 1/√128 ≈ 0.342. The lesson outlives this one tensor: when a number reaches the engine from a converter rather than from training, the only safe way to depend on it is to assert it at the boundary.

So Q’s norm is “normalize, then scale by ≈3.87” and K’s is a plain normalize. And note what the scale is in addition to: the attention softmax scale is still 1/√128 ≈ 0.0883883 (config.rs:277-281), independent of the folded-in 3.87 — two different scales living at two different points of the graph, easy to conflate, asserted apart by test.

Back to the question this section opened with, now that the machinery is on the page. Per-head normalization exists to stabilize the score distribution within each head before the dot product, so that one hot-headed head cannot dominate the softmax on the strength of its magnitude alone. On this model the “learned” part of that stabilization was folded into a single scalar by training upstream [unverified] for the quality rationale — what is verified is the probe, the values, and where they are applied.

14.4 RoPE from zero

14.4.1 Pair up, rotate, at a per-pair frequency

So how do you tell a dot product where a token sits, without adding a parameter and without touching the attention kernel? You spin the vector. The mechanism is easier than RoPE’s reputation suggests; what ruins engines is never the rotation itself but the bookkeeping around it, so build the mechanism first and meet the bookkeeping immediately after.

Take one head’s 128-wide vector and split it into 64 pairs. Muse Glimmer uses the interleaved convention — pair i is the adjacent couple (x[2i], x[2i+1]) — not the half-split (“NEOX”) convention of Llama/Qwen. The shader file’s own comment block is the authoritative statement and is worth quoting whole:

// crates/muser-engine/src/shaders/ferrite/rope.metal:610
// ── NORM-convention RoPE (LLAMA_ROPE_TYPE_NORM) ─────────────────────────
//
// Every other kernel in this file uses the NEOX convention: rotate the pair
// (x[i], x[i + half_hd]). llama.cpp's `rope_norm` instead rotates the
// *interleaved* pair (x[2i], x[2i+1]). Both read the SAME frequency table —
// freq[j] = base^(-2j/head_dim) — so only the element pairing differs, and a
// model that needs one and gets the other still produces fluent text with
// silently wrong positions. That is why these are separate kernels rather
// than a runtime flag threaded through the NEOX ones.
//
// Muse Glimmer is the first NORM-rope architecture in the tree. Its GGUF
// converter un-permutes Q/K at conversion time precisely so the interleaved
// form is the correct one for the stored weights.

Read the warning twice: rotating with the wrong pairing still runs and still produces fluent text — with silently wrong positions. There is no crash to catch; only a parity gate against pinned llama.cpp can see it. A bug with no symptom cannot be found in production, so it has to be made unrepresentable at build time instead — the defense the tradeoffs section at the end of this chapter returns to.

Pair i has a fixed frequency, set once at load:

θ_i = rope_base_swa^(−2i/head_dim)     (i = 0..63)

rope_base_swa is read from the GGUF key rope.freq_base_swa (falling back to rope.freq_base, default 10,000 if absent, config.rs:120-129, 199-205). The pinned checkpoint’s value is not asserted anywhere in the tree [unverified] — what is verified is the key it is read from and the table formula built in Rust:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:1256
(0..cfg.head_dim / 2)
    .map(|index| {
        1.0 / cfg
            .rope_base_swa
            .powf(2.0 * index as f32 / cfg.head_dim as f32)
    })
    .collect::<Vec<_>>()
}

The powf happens once per engine; no kernel ever calls pow — that is what the _cached in the kernel name means.

14.4.2 The rotation, and one worked pair

The rotation of a 2D vector (v₀, v₁) by angle a is the 2×2 matrix every graphics programmer has memorized:

   v₀' = v₀·cos(a) − v₁·sin(a)
   v₁' = v₀·sin(a) + v₁·cos(a)

Length is preserved; the vector spins counterclockwise. Work one toy case, head_dim = 4 (two pairs), using the config-default base 10,000 so every number is real:

  θ₀ = 10,000^(−0/4) = 1.0        (fastest pair: 1 radian per position)
  θ₁ = 10,000^(−2/4) = 0.01       (100× slower)

  At pos = 2, take x = [1, 0, 1, 0]  (both pairs pointing at their x-axis):

    pair 0, angle 2×1.0  = 2.0 rad :
        (1, 0) → (cos 2.0, sin 2.0)          ≈ (−0.4161,  0.9093)
    pair 1, angle 2×0.01 = 0.02 rad :
        (1, 0) → (cos 0.02, sin 0.02)        ≈ ( 0.9998,  0.0200)

  x' ≈ [−0.4161, 0.9093, 0.9998, 0.0200]

Figure 14.1: RoPE at pos 2 on a head_dim-4 toy. The fast pair has been thrown nearly onto the negative x-axis; the slow pair barely moved. Every position leaves a different fingerprint of angles. At real width 128, frequencies θ_i span many orders of magnitude — a bank of 64 clocks at wildly different speeds, from “full turn every ~6 positions” (i = 0) to “one turn in ~10⁶ positions” for the slowest pair at base 10⁴ (and further out at larger bases). A larger base slows the slow clocks, keeping them from aliasing — completing a full turn and landing two distant positions on the same angle — within a long context. That is why long-context models choose large bases [arxiv:2104.09864]; which trade-off Muse Glimmer’s authors weighed for 131,072 positions is theirs, not the engine’s [unverified].

14.4.3 The magic: the dot product sees only (m − n)

Why rotate both Q and K? Suppose Q sits at position m, K at position n; RoPE spins pair i of Q by m·θᵢ and of K by n·θᵢ. Their dot product over one pair, with raw values (q₀,q₁) and (k₀,k₁), expands and — using cos(A−B) = cos A cos B + sin A sin B and sin(A−B) = sin A cos B − cos A sin B — collapses:

q_rot · k_rot  =  (q₀k₀ + q₁k₁)·cos((m−n)θᵢ)
               +  (q₀k₁ − q₁k₀)·sin((m−n)θᵢ)

Every m and n appears only as the difference m − n. Absolute positions have vanished; relative position falls out for free, with zero extra parameters and no change inside the attention kernel.

Worth restating in different words, because this is the idea the whole chapter hangs on: RoPE never tells attention where a token is. It arranges matters so that attention cannot ask anything else but how far apart two tokens are. The absolute coordinates go in, cancel against each other on the way through the dot product, and only the gap comes out.

That single identity is why RoPE is the decode-era default — and, in this book, it has a second life in §14.6: it is also the reason NoPE’s cache bytes are relocatable and RoPE’s are not.

14.5 The Metal kernel and the SWA-only dispatch

That is the theory; here is all of it in hardware. The kernel’s job is narrower than the derivation makes it sound — for this token, which floats get spun, and by how much? — and everything interesting in the listing is about answering that cheaply enough for the work to be invisible against the weight stream.

// crates/muser-engine/src/shaders/ferrite/rope.metal:624
kernel void rope_norm_batch_cached(
    device       float* Q          [[ buffer(0) ]],
    device       float* K          [[ buffer(1) ]],
    device const float* freq_table [[ buffer(2) ]],  // [half_hd]
    constant     uint&  n_heads    [[ buffer(3) ]],
    constant     uint&  n_kv_heads [[ buffer(4) ]],
    constant     uint&  head_dim   [[ buffer(5) ]],
    constant     uint&  start_pos  [[ buffer(6) ]],
    uint2 tgid [[ threadgroup_position_in_grid ]],
    uint  lid  [[ thread_index_in_simdgroup ]])
{
    const uint batch   = tgid.y;
    const uint pair_id = tgid.x * 32u + lid;

    const uint half_hd       = head_dim / 2u;
    const uint total_q_pairs = n_heads    * half_hd;
    const uint total_pairs   = total_q_pairs + n_kv_heads * half_hd;
    if (pair_id >= total_pairs) return;

    const uint pos = start_pos + batch;

    const bool is_q  = (pair_id < total_q_pairs);
    const uint local = is_q ? pair_id : (pair_id - total_q_pairs);
    const uint head  = local / half_hd;
    const uint pi    = local % half_hd;

    const uint q_stride = n_heads    * head_dim;
    const uint k_stride = n_kv_heads * head_dim;

    device float* base = is_q
        ? (Q + batch * q_stride + head * head_dim)
        : (K + batch * k_stride + head * head_dim);

    float angle = float(pos) * freq_table[pi];
    float cos_a = precise::cos(angle);
    float sin_a = precise::sin(angle);

    // NORM convention: the pair is adjacent, at 2*pi and 2*pi + 1.
    const uint i0 = 2u * pi;
    float v0 = base[i0];
    float v1 = base[i0 + 1u];
    base[i0]      = v0 * cos_a - v1 * sin_a;
    base[i0 + 1u] = v0 * sin_a + v1 * cos_a;
}

One thread per pair, in-place on Q and K:

• The decomposition pair_id → (is_q, head, pi) — flat index into “all pairs of Q then all pairs of K”; head = local / 64, pi = local % 64 at width 128. pi is exactly the i of θᵢ.
• No pow — one multiply against the cached table.
• precise::cos / precise::sin — Metal’s fast-math transcendentals can be a few ULP off; precise:: selects the higher-accuracy variant for these calls only, without giving up fast-math elsewhere in the library (Ch 4). At large bases and large positions the angle pos·θᵢ is itself large and its sine/cosine must be exact to the bit the comparator computed — accuracy, not speed, is the reason (the ancestor book carries the same lesson [ferrite-book Ch 4, Ch 13]).
• The interleaved addressing i0 = 2·pi — the NORM convention of §14.4.1, two adjacent floats rotated as one 2D vector, written straight back.

The dispatch (metal/encode/rope.rs:139-151) launches ceil(total_pairs/32) × batch threadgroups of 32 — one thread per pair, and with total_pairs = (32 + 2) × 64 = 2,176 for Muse Glimmer that comes to 68 threadgroups of 32 threads, once per sliding layer, per token.

The wrapper has more to decide than a grid size, though. The same rotation exists three times in the tree, and the copies are not interchangeable — same math, different provenance for the bits, and provenance is precisely the kind of difference the convention warning above says never surfaces in the output. When the pinned metallib is loaded, the wrapper prefers llama’s own kernel_rope_norm_f32 with a packed GgmlMetalKargsRope: same convention, llama’s own arithmetic, no daylight between the engine and the comparator it is scored against (rope.rs:88-138). Without that metallib, the ferrite kernel quoted above is the fallback. Under the cross-vendor flags the choice changes once more, to the no-fast-math NCO table route with explicit per-token positions (rope.rs:62-86) — the seam Ch 32 needs when the tensors on the other side of the handoff were produced by somebody else’s hardware.

The dispatch condition — SWA only

Which layers actually pay for all this? Not most of them — and the whole position apparatus hangs off a single predicate in the token graph:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5621
if cfg.layer_kinds[layer_index].uses_rope() {
    dispatch(command, |encoder| {
        self.kernels.encode_rope_norm_batch_cached(
            encoder,
            &self.activations.q,
            &self.activations.k,
            &self.rope_frequencies,
            cfg.n_heads,
            cfg.n_kv_heads,
            cfg.head_dim,
            position,
            1,
            // …(positions view + freq_base + n_ctx_orig elided)…
        );
    });
}
}

and uses_rope() is one line, with provenance to the comparator:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/config.rs:66
/// RoPE runs iff the layer is a sliding layer (`muse-glimmer.cpp:93`:
/// `const bool use_rope = hparams.is_swa(il);`).
pub const fn uses_rope(self) -> bool {
    matches!(self, Self::SlidingRope)
}
}

On layers {3, 7, …, 51} no rotation dispatch exists at all. Q and K flow from QK-norm straight to the KV store, positionless by design. The layer-kind partition itself is fail-closed: the loader panics rather than guess if the sliding_window_pattern key is missing, because “an all-full model runs and emits plausible text while being wrong” (config.rs:378-380). Notice what kind of defense that is. The loader is not saving itself from a crash — it is saving us from the absence of one, which is the same reason the QK-norm probe earlier in this chapter exists. Two different keys, two different tensors, one shared conviction: on this engine a missing assumption must stop the process, because the model will never complain on its own.

14.6 NoPE and relocatable KV — the consequence

Here is why the two-class asymmetry is the most consequential layout decision in the engine. A cached K row for a sliding layer is a function of its absolute position: the rotation angle pos·θᵢ is baked into the bytes. Move that row to a different position and the (m−n) identity of §14.4.3 breaks — the stored rotation is the wrong one. A cached K row for a NoPE layer is just a projection of the token’s hidden state: it carries no positional information whatsoever, so the same bytes are valid at any position. Two facts follow, both load-bearing for Part V and VI:

• NoPE tiles relocate by memcpy. A 512-token NoPE KV tile can be planted anywhere in any cache of the same identity — prefix reuse, warm sessions, remote handoff — with no recompute and no re-rotation. The engine’s own summary: the 13 NoPE layers are “position-free (relocate = memcpy) — the whole kvpack free lunch” (lib.rs:9-10), and kvpack’s layout keys enforce it fail-closed (the NoPE identity requires theta = 0; a RoPE layer cannot claim it — crates/muser-kvpack/src/layout.rs, K1 keys [crates/muser-engine/src/lib.rs:7-15]).
• SWA planes are position-bound but bounded. The 39 sliding layers only ever need the last 2,048 tokens (§14.7 previews the ring), so their “where” question is a window offset, not an absolute position — that is Ch 15’s whole subject.

When Ch 24 and Ch 26 move KV tiles around the lab, this section is the reason NoPE bytes move freely and SWA bytes move as logical tails with explicit origins.

14.7 Tradeoffs

Four decisions in this chapter could plausibly have gone the other way, and they share a family resemblance: in every one of them the wrong branch still runs, still returns floats, and still reads like language. That is what makes them worth walking rather than tabulating.

Interleaved vs half-split — a convention you must match, not choose. Same rotation math, different pairing (Figure 14.2). The checkpoint was trained with one; the GGUF converter un-permuted Q/K “precisely so the interleaved form is the correct one for the stored weights” (rope.metal:620-622), and llama.cpp’s rope_norm matches. Mixing conventions is the classic silent failure — fluent text, wrong positions (rope.metal:614-617) — and the defense is structural: separate kernels (rope_norm_batch_cached vs rope_batch_cached, rope.metal:566) rather than a runtime flag, so no configuration can accidentally cross-wire the two.

  head vector:   x0  x1  x2  x3 │ x4  x5  x6  x7          (head_dim = 8)
                                │
  INTERLEAVED (NORM / Muse):    │   HALF-SPLIT (NEOX / Llama-family):
  pair 0 : x0 ── x1             │   pair 0 : x0 ─────────── x4
  pair 1 : x2 ── x3             │   pair 1 : x1 ─────────── x5
  pair 2 : x4 ── x5             │   pair 2 : x2 ─────────── x6
  pair 3 : x6 ── x7             │   pair 3 : x3 ─────────── x7

Figure 14.2: the two RoPE pairings. Same frequency table, same rotation, different addressing. Match the checkpoint or corrupt position silently (rope.metal:610-618).

Rotate-before-store, and rotate Q and K only. RoPE runs before the KV store so the cache holds already-rotated keys — attention then needs no position argument beyond the mask, and the store kernel of Ch 15 is a pure copy. V is never rotated (position lives in the Q·K score, not in the payload), which is why V’s plane is pure payload bytes on both layer classes.

precise:: trig vs fast-math trig. The engine compiles its main library with fast-math on for speed (Ch 4) but pays for exact trig here, per call. The cost is a handful of cycles on 2,176 threads × 39 layers — nanoseconds — and the benefit is that sin(pos·θ) matches llama’s own evaluation of the same angle to the bit. There is no measured A/B in the ledger [unverified]; the choice is contract-driven, like most precision decisions in this book.

The QK-norm seam across the handoff. The producer side (vLLM on the GB10) materializes its weightless QK-norm in F16 and applies the scale as a second F16 operation; a fused RMS+weight kernel “loses that intermediate rounding point and is therefore not seam-exact” (metal/encode/norm.rs:280-284). Under MUSER_CROSS_VENDOR_QK the QK-norm therefore splits into two strict-f32 dispatches with a barrier between — slower, deliberately, to preserve the producer’s rounding. One more example of the book’s recurring rule: exactness is always exactness against a specific anchor.

14.8 Where the gap lives

Every kernel chapter owes the same accounting answer: does this work show up in the decode gap, or does it disappear into the noise of the layers around it? Here the arithmetic settles it quickly.

This kernel is not the gap. Bytes per token: Q 4,096×4 + K 256×4 read and written in place, ≈ 17.4 KB in and the same out, ×39 sliding layers ≈ 1.4 MB of traffic — six orders of magnitude under the weight stream of Ch 13. In the closure accounting of [docs/decode-dispatch-gap-20260815.md], the qk_norm and rope labels sit in the common-math row (delta 0). The chapter’s gap-adjacent fact is the one already banked in Ch 12: the norm-boundary families (+104) are where fusions go to die on exactness, and QK-norm’s dispatch shape is one of the boundaries that inherits that discipline.

14.9 What comes next

Q is rotated (on sliding layers), K likewise, V untouched — and the current token’s K and V must now be written into the cache that attention will read. One engine, two storage regimes: a 2,048-slot ring for the 39 sliding layers, a growing head-major plane for the 13 NoPE layers. That is Ch 15.

References

• crates/muser-engine/src/shaders/ferrite/rope.metal:610-667 — the NORM- convention comment block and rope_norm_batch_cached (quoted).
• crates/muser-engine/src/shaders/ferrite/rope.metal:566-608 — rope_batch_cached, the NEOX sibling kept deliberately separate.
• crates/muser-engine/src/metal/encode/rope.rs:45-152 — the dispatch: cross-vendor NCO route, pinned kernel_rope_norm_f32 route, and the ferrite fallback.
• crates/muser-engine/src/decode.rs:5599-5641 — QK-norm and the SWA-gated RoPE call sites.
• crates/muser-engine/src/decode.rs:1217-1270 — the frequency-table build (and the retained RoPE-cache file check).
• crates/muser-engine/src/config.rs:51-71 — MuseLayerKind, uses_rope() and the muse-glimmer.cpp:93 provenance.
• crates/muser-engine/src/config.rs:84-102, 336-381 — the il % 4 == 3 partition and the fail-closed pattern resolution.
• crates/muser-engine/src/config.rs:139-141, 277-281, 383-403, 426-430 — qk_scale_factor, attn_scale, QkNormProbe, and the scale test.
• crates/muser-engine/src/metal/encode/norm.rs:280-303 — encode_qk_norm and the vLLM F16 seam comment.
• crates/muser-engine/src/shaders/ferrite/rms_norm_per_head.metal:15-47 — the per-head tree-reduction kernel (algorithm reference).
• crates/muser-engine/src/lib.rs:7-15 — the position-free-NoPE / kvpack free-lunch summary.
• crates/muser-engine/src/loader.rs:28-37 — the fail-closed QK-norm probe at load.
• Ch 12 — the RMSNorm machinery QK-norm reuses.
• Ch 15, Ch 24, Ch 26 — where the NoPE/SWA asymmetry pays off.
• [arxiv:2104.09864] — Su et al., RoFormer (the RoPE origin paper).
• [arxiv:1910.07467] — RMSNorm (for §14.3’s normalization).
• [ferrite-book Ch 13] — the ancestor’s RoPE chapter (permutation- invariance motivation, (m−n) proof, convention warning — ported with Muse’s NORM convention and dual-class twist).

Chapter 15 — KV store and the ring

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 9 (the 39/13 layer split), Ch 14 (rotated K, unrotated V, and why NoPE bytes are position-free), Ch 13 (the K/V projections that produce this chapter’s inputs). This is an infrastructure chapter: the data structure every attention dispatch of Ch 16 reads.

15.1 What the KV store computes

Chapter 14 ended with Q rotated (on the sliding layers), K likewise, V untouched — and the current token’s K and V waiting to be written into the store attention will read. This chapter is that write.

Before any code, the question this chapter answers: where does a long session’s memory actually live, and what does keeping it cost? The answer starts one step earlier, with why a store has to exist at all.

To produce a token, attention must score the current query against the Key of every visible past token and take a weighted sum of their Values (Ch 16 does the math from zero). The Key and Value of past token i depend only on token i — they are projections of that token’s hidden state (Ch 13) — and once computed they never change. So every engine faces the same choice: recompute all past K/V every step (O(n²) work over a generation — token 100,000 recompute 99,999 keys it already computed) or compute each once, store it, reuse it forever. The store is the KV cache.

This chapter is about the write side: the per-layer GPU buffers, the two storage regimes Muse Glimmer’s architecture forces, the kernels that copy this token’s K and V into them, and the ring arithmetic that decides where the write lands. The read side is Ch 16; the portable-asset side is Part V.

The recurrence that motivates it all is per layer class. For the 13 NoPE layers, “every past token” grows without bound — the cache genuinely grows with context. For the 39 sliding layers, attention only ever sees the last sliding_window = 2,048 tokens (config.rs:16, 131), so the recompute-if-you-don’t-cache body is bounded at 2,048 keys — small enough that a fixed-size ring whose capacity never grows is the natural store. One model, two memory regimes, decided by layer % 4 == 3 (config.rs:84-93).

15.2 The planes in code — MetalKvPlane

Start with the object, because everything the ring does later is bookkeeping held in its fields — and it is worth knowing up front what a wrong field costs. A plane that loses track of which row is its oldest does not crash; it hands attention some other token’s key and keeps going. The per-layer cache object is seven fields:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:182
struct MetalKvPlane {
    key: GpuHalfBuffer,
    value: GpuHalfBuffer,
    capacity: usize,
    len: usize,
    origin_logical: usize,
    origin_physical: usize,
    head_major: bool,
}
}

• key / value — two f16 buffers; f16 on every Metal lane (the GpuHalfBuffer type is the constraint). F32 planes exist only in the kvpack interchange, never in live decode.
• capacity / len — the plane holds at most capacity token rows; len are currently live.
• origin_logical / origin_physical — the ring’s explicit bookkeeping: which logical token position sits at the front of the live window, and which physical slot it occupies. Said the other way round, the plane never asks “where does token n belong?”; it asks “how far has my window slid since I last knew where its front was?”
• head_major — which of the two layouts this plane uses (§15.3).

Those two origin fields are the whole difference between this ring and the obvious alternative, addressing straight by “position mod capacity”, and we did not choose the long way round for elegance. The ancestor engine took the obvious route and it bit: the extraction manifest records the naive form as a named out-of-bounds hazard — “Ferrite indexed by absolute position; the ring modulus was unwired/stubbed” — which is why the SWA ring-address translation is Muser-owned rather than inherited (docs/extraction-manifest.md). Carrying the origins explicitly costs two integers per plane and deletes that whole class of bug.

Allocation is per layer kind, and it is where the two regimes become concrete:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:1344
let mut cache = Vec::with_capacity(cfg.n_layers);
for layer in 0..cfg.n_layers {
    let capacity = match cfg.layer_kinds[layer] {
        MuseLayerKind::SlidingRope => max_context.min(cfg.sliding_window).max(32),
        MuseLayerKind::FullNoPe => max_context.max(32),
    };
    // Zero-filled on purpose: wrapped SWA rows must never expose
    // uninitialized storage during a sequence boundary transition.
    cache.push(MetalKvPlane::new(
        &shared.context,
        capacity,
        cfg.kv_dim(),
        matches!(cfg.layer_kinds[layer], MuseLayerKind::FullNoPe),
    )?);
}
}

A sliding layer gets min(max_context, 2,048) rows, token-major; a NoPE layer gets max_context rows, head-major.

The zero-fill comment sitting in that loop is worth a detour, because the reason it matters for anyone reading Muser’s docs beside this book is that we had the fact backwards in writing before we had it right. An engineering doc stated that Metal KV buffers were allocated without a CPU memset. It is a plausible claim — a buffer that is about to be overwritten row by row looks like a pure waste of a memset — and it stood long enough to be quoted. The audit that went checking found the opposite for live planes: they zero-fill on purpose, so that a wrapped SWA row can never expose uninitialized storage while a sequence boundary is in flight. Only detached remote-install generations take the uninitialized path. The correction is retained, and it is the corrected fact that should be cited, never the stale line: [docs/kvpack-merge-handoff §3 D2, per the 2026-08-20 audit].

15.3 The two layouts

Why should one model carry two layouts at all? Because a layout is never chosen for the writer’s convenience. It is chosen for whoever reads it most often — and the two layer classes have different readers, so they get different shapes.

kv_dim = n_kv_heads × head_dim = 2 × 128 = 256 halves per token row. The two planes lay those halves out differently:

  TOKEN-MAJOR (SWA ring), capacity C = 2,048:

    [token 0   ][token 1   ]…[token C−1 ]   each row = 256 halves
     k0h0 k0h1  k1h0 k1h1                      (h0 = KV head 0's 128,
                                                  h1 = KV head 1's 128)
    index: physical × kv_dim + element

  HEAD-MAJOR (NoPE growing), capacity C = max_context:

    KV head 0: [tok 0][tok 1]…[tok C−1]   128 halves per row
    KV head 1: [tok 0][tok 1]…[tok C−1]
    index: (kv_head × capacity + physical) × head_dim + dim

Figure 15.1: the two KV layouts. Token-major keeps one token’s two head rows adjacent — a ring rotation moves whole tokens. Head-major keeps each KV head’s sequence contiguous — a head’s whole history is one linear span, which is exactly the shape llama.cpp’s flash-attn vec kernel wants for its head-major ns10 = 128 addressing (metal/encode/attn.rs:503-511). The index formulas are the store kernels’ own (muse_reference.metal:1224, :1228); why each layout pairs with its layer class is §15.6’s tradeoff.

15.4 The store kernels

With the shapes settled, the write itself is an anticlimax, and that is the design. The useful question here is not “how does the store work?” but “how little is the store allowed to know?” — because every fact the GPU kernel is told about token positions is a fact that can be wrong on the GPU, where nothing checks it.

Two kernels write K/V; both are pure quantize-free copies — the values are already f32 in the activation buffers and f16 in the planes.

Single-token, token-major — the SWA ring’s steady-state write:

// crates/muser-engine/src/shaders/muse_reference.metal:979
kernel void muser_kv_store_f16(
    device const float *key [[buffer(0)]],
    device const float *value [[buffer(1)]],
    device half *key_cache [[buffer(2)]],
    device half *value_cache [[buffer(3)]],
    constant uint &kv_dim [[buffer(4)]],
    constant uint &write_index [[buffer(5)]],
    uint index [[thread_position_in_grid]]) {
    if (index < kv_dim) {
        uint destination = write_index * kv_dim + index;
        key_cache[destination] = key[index];
        value_cache[destination] = value[index];
    }
}

One thread per element; 256 threads; the row’s position (write_index) arrives pre-computed from the ring arithmetic of §15.5 — the kernel never sees an absolute token position. The f32→half conversion is implicit in the assignment. The wrapper binds buffers and dispatches dispatch_1d(key.len()) (metal/encode/attn.rs:635-655).

Batched, both layouts — the NoPE planes and prefill chunks:

// crates/muser-engine/src/shaders/muse_reference.metal:1203
kernel void muser_kv_store_batch_f16(
    device const float *key [[buffer(0)]],
    device const float *value [[buffer(1)]],
    device half *key_cache [[buffer(2)]],
    device half *value_cache [[buffer(3)]],
    constant uint &kv_dim [[buffer(4)]],
    constant uint &source_first [[buffer(5)]],
    constant uint &source_count [[buffer(6)]],
    constant uint &start_position [[buffer(7)]],
    constant uint &capacity [[buffer(8)]],
    constant uint &origin_logical [[buffer(9)]],
    constant uint &origin_physical [[buffer(10)]],
    constant uint &head_dim [[buffer(11)]],
    constant uint &head_major [[buffer(12)]],
    uint index [[thread_position_in_grid]]) {
    uint total = source_count * kv_dim;
    if (index < total) {
        uint source_token = source_first + index / kv_dim;
        uint element = index % kv_dim;
        uint logical = start_position + source_token;
        uint physical = (origin_physical + logical - origin_logical) % capacity;
        uint destination = physical * kv_dim + element;
        if (head_major != 0u) {
            uint kv_head = element / head_dim;
            uint dim = element % head_dim;
            destination = (kv_head * capacity + physical) * head_dim + dim;
        }
        uint source = source_token * kv_dim + element;
        key_cache[destination] = key[source];
        value_cache[destination] = value[source];
    }
}

The layout switch is the if (head_major) block: token-major lands at physical × kv_dim + element; head-major re-interleaves to (kv_head × capacity + physical) × head_dim + dim — Figure 15.1’s two index formulas, one kernel. Note that this kernel does see absolute positions (start_position, origin_logical) but only inside the explicit-origin translation physical = (origin_physical + logical − origin_logical) % capacity — never as logical % capacity.

15.5 Ring write-position arithmetic — append

Everything so far has deferred one question: which row does this token’s K and V land in? The stake is unusually quiet. Get the row wrong and nothing crashes — attention simply scores the query against some other token’s key, the logits shift, and the only symptom is generated text that is subtly worse than it should be, at a rate no test that checks for crashes will ever catch. So the arithmetic that answers the question is small, explicit, and refuses to guess. The CPU-side reservation that decides write_index is eleven lines and fail-closed:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:263
/// Reserve the physical row for `position` and advance explicit ring
/// metadata. No physical placement is derived from the absolute token ID.
fn append(&mut self, layer: usize, position: usize) -> Result<usize, MetalModelError> {
    let expected = self.origin_logical + self.len;
    if position != expected {
        return Err(MetalModelError::CacheDiscontinuity {
            layer,
            expected,
            got: position,
        });
    }
    if self.len < self.capacity {
        let write = (self.origin_physical + self.len) % self.capacity;
        self.len += 1;
        Ok(write)
    } else {
        let write = self.origin_physical;
        self.origin_logical += 1;
        self.origin_physical = (self.origin_physical + 1) % self.capacity;
        Ok(write)
    }
}
}

Walk it at the ring boundary, capacity 2,048:

  positions 0..2047 (fill):  write = origin_physical + len, len → 2048
                             origin_logical = 0, origin_physical = 0

  position 2048 (first wrap):
      len == capacity → write = origin_physical = 0     (overwrite slot 0)
      origin_logical  → 1        (the window now starts at token 1)
      origin_physical → 1        (slot 1 is the oldest live row)

  position 2049:  write = 1; origins → (2, 2).   And so on, forever.

Figure 15.2: the ring wrap. The write pointer, the logical origin, and the physical origin advance together; the plane’s physical layout is a rotation of the logical tail.

Three properties fall out of the wrap arithmetic of Figure 15.2:

1. Fail-closed continuity. position != origin_logical + len is an error, not a modulo — a skipped or replayed position can never silently alias a live row (decode.rs:117-121 defines the error; prefill’s module doc states the invariant: “physical placement is never derived from absolute position”, prefill.rs:15-17).
2. NoPE degenerates to append-only. A NoPE plane’s capacity is max_context; len < capacity always holds in a valid session, so the else branch never runs, origin_logical stays 0, and the modulo vanishes — the “ring” is a growing array.
3. append_batch (decode.rs:286-314) is the chunked form: it advances the origins by the overflow when a chunk crosses capacity and hands back which source rows are still live — the arithmetic behind prefill’s ring wrap.

The write_physical this returns is exactly the write_index the store kernel of §15.4 receives (decode.rs:5643, 5661-5667).

15.6 Why restore must preserve rotation

The plane’s rotation (where origin_physical points) looks like implementation detail. It is not — it is numerics. Attention scans rows in physical order and floating-point accumulation is order-sensitive, so two planes holding identical rows in different rotations produce different last-bit logits.

Put it plainly, because this is the idea in the chapter most worth re-reading: the same keys, stored starting at a different slot, are summed in a different order, and a different order of floating-point additions is a different number. The rotation is not where the data happens to sit. The rotation is part of the data. The restore path documents exactly this, with a test name in the comment:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:376
// Install at the rotation a sequentially-built live ring holds at this
// logical origin. Attention scans rows in physical order and float
// accumulation is order-sensitive, so a restore packed at origin 0
// can never replay a wrapped live session's logits bitwise (caught by
// real_model_wrap_boundaries_and_detached_restore_replay_exactly).
// NoPE planes never wrap (origin_logical is always 0), so their
// rotation is 0 and this reduces to the previous layout.
let rotation = snapshot.origin_logical % snapshot.capacity;
}

detached_from then scatters the snapshot’s logical rows back at (rotation + logical_offset) % capacity (decode.rs:384-407) — head-major per KV head, token-major as a split head/tail copy. Without this, a saved session restored on another process would produce bit-different logits from the same bytes, and the exactness gates of Ch 38 (and kvpack’s bit-identical warm hits, Ch 25) would fail for a reason no amount of weight-side care could fix. Bitwise replay requires the rotation to travel with the cache.

15.7 Footprint per token, derived by hand

Now the bill. Two questions a reader actually has — what does a session cost in memory, and where does that cost concentrate — are answerable with nothing but multiplication, so we would rather you re-derive them than take our word.

One K row per layer = n_kv_heads × head_dim × 2 B = 2 × 128 × 2 = 512 B; K and V together = 1,024 B per layer per token — topology-derived arithmetic, not a measured RSS [docs/memory-footprint.md §KV formula]. Derive the three numbers that matter:

  per token, whole model:  52 layers × 1,024 B = 53,248 B ≈ 52 KiB

  SWA steady state (bounded):  39 × 2,048 × 1,024 = 81,788,928 B ≈ 78 MiB
      — constant, whatever the context depth

  NoPE at max context:  13 × 131,072 × 1,024 = 1,744,830,464 B ≈ 1.66 GiB
      — grows linearly, unbounded to the 131,072 model limit

  one slot at 131,072:  81,788,928 + 1,744,830,464 = 1,826,619,392 B
                      ≈ 1.827 GB (decimal)

which is the memory-footprint table’s one-slot figure (1.827 GB; four slots 7.306 GB) [docs/memory-footprint.md, via the measured-numbers ledger §1k]. Note the asymmetry: at full depth, ~96 % of the KV footprint is the 13 NoPE layers. The same split shows up on the wire: the deep 130,815-token handoff’s 1,823,184,896 B payload decomposes as 130,814 × 13,312 B of NoPE (13 layers × 1,024 B per token) plus three pipe-safe SWA groups of 2,048 × 13,312 B — i.e. ≈95.5 % NoPE / ≈4.5 % SWA by these terms [receipt phase4-disagg-20260820/130815-g900091/; docs/kvpack-merge-handoff §6]. Ch 22 owns the full footprint treatment (2k/32k/131k tables, slot arithmetic); keep this chapter’s derivation as the per-layer-class split it is.

15.8 Tradeoffs

The store could have been built differently at four points — how the cells are compared, how the planes are laid out, what precision they hold, and whether the store is its own dispatch at all. Each alternative deserves a price rather than a dismissal. Start with the frame the rest of them are argued in.

A 2×2 that splits the confounds. The ancestor book’s sharpest KV device was a 2×2 layout × precision matrix whose ratios multiplied out to the combined effect, exposing that a flag named “addressing” was mostly a precision swap [ferrite-book Ch 14]. Muser’s KV matrix has the same shape with different axes — layout (token-major vs head-major) × layer class (SWA-RoPE vs NoPE) — and the decomposition that matters is the payload split of §15.7: at depth, the NoPE cells carry ≈95.5 % of the bytes and the SWA rings ≈4.5 %.

That split, not any kernel choice, is what shaped the disaggregated lane’s send schedule: stream the “SWA groups (~82 MB) early” and hold “the NoPE bulk (95.7 % of payload)” back until prefill finishes layer 51. The reasoning felt free — the small planes are ready first, so send them first and buy overlap for nothing. On the wire it was not free. A burst-then-wait shape is exactly the traffic pattern that lets an EEE-capable link drop into an idle state between bursts, and the pacing section of the merge handoff is where that collision is written down ([docs/kvpack-merge-handoff §6 "Pacing reality"]); Ch 31 tells the story properly. The lesson outlives the wire: the payload split is an input to schedules made three layers of abstraction away, so when you change one cell of this 2×2, re-derive the split before predicting anything downstream of it.

Ring + growing plane vs one layout, vs paging. Why not one layout for all 52 layers? A token-major growing NoPE plane would make each head’s history strided, defeating the head-major ns10=128 addressing of the pinned attention kernel; a head-major ring would rotate each head independently — same rotation, but the store/restore walk gets no contiguous tail and the llama vec path cannot read a wrapped ring as one span. The hybrid pairs each class with the layout its reader wants. Why not paging (16-token blocks behind a block table), the ancestor’s design? Because Muse Glimmer’s SWA layers bound themselves — a full ring is the whole live set, nothing to evict — and the NoPE layers’ prefix sharing is handled one level up, by kvpack’s content-addressed chunks (Ch 24), not by an in-GPU page table. The OS-paging analogy survives only as contrast [ferrite-book Ch 14].

f16, not quantized, not f32. The planes are f16 on every Metal lane (the GpuHalfBuffer field type, decode.rs:183-184). The ancestor quantized its cache to Q8_0 and measured an 80/15 precision/addressing split when it stopped [ferrite-book Ch 14] — Ferrite-lineage context, not a Muser decision point: Muser’s parity anchor (pinned llama.cpp) runs F16 KV, so Muser does too, and the lane table’s decode rows all say “FP16 KV” [docs/muser-architecture.md]. The kvpack interchange still carries both encodings (PlaneEncoding::{F16Le, F32Le}, crates/muser-engine/src/cache.rs:11-16) because the wire format serves producers and archives beyond the live planes. A quantized live cache would halve §15.7’s numbers and is exactly the kind of change the exactness contract makes a research lane, not a default.

Store-then-barrier vs fused store-in-attention. Muser’s decode stores K/V in its own dispatch and puts an explicit memory_barrier_with_resources between store and attention (decode.rs:5669-5670) when the pinned vec kernel follows. The ancestor fused the quantizing store into its attention kernel to save a dispatch [ferrite-book Ch 15]. Muser’s store is a 512-byte-per-plane copy with nothing to amortize, and the split keeps the pinned llama kernels untouched — the same pin-the-boundary reasoning as Ch 13. The ferrite interleaved fallback kernel does fuse its store (flash_attn_decode_vec_contiguous_ f16.metal:534-543) — on that route only, one simdgroup writes the current K/V as a side effect; see Ch 16.

15.9 Where the gap lives

The book’s running question about the decode gap is which dispatches are waste and which are structure. This chapter is where that question gets its least comfortable answer, twice.

Two of the four +196 families live here. 52 KV-publication splits (production’s separate kv_store + attention closures vs legacy’s combined kv_store_attention) are classed “session/publication structure — Keep; combining closures alone does not remove their kernel math” [docs/decode-dispatch-gap-20260815.md §Corrected closure-count diff]. And 39 SWA wrapped-ring staging groups are the batch graph’s multi-row ring feature — the staging shadow of Ch 36 — kept “until a bit-exact ring-aware replacement exists.” Neither is waste in the ordinary sense. Both are dispatches that exist because removing them would change a number the exactness contract does not allow to change — which is a harder thing to argue away than a merely inefficient loop. The note’s own ranked list still hopes for “a one-row ring-aware attention path that avoids SWA staging, gated by bitwise KV and full-logit equality at positions 1, 31, 32, 33, 2,047, 2,048, and 2,049.” This chapter is where those two rows of the table become concrete: the splits are the store dispatches you just read; the staging groups are what wrapping a 2,048-ring under a 512-token prefill chunk costs.

15.10 What comes next

The planes hold every visible token’s K and V — a ring for the sliding layers, a growing span for the full ones — and the current token’s Q is rotated and QK-normalized. Everything attention needs is in place. There is no single “attention kernel” to read next: there is a route ladder, selected per layer per token by alignment predicates. That ladder — and the Q·Kᵀ softmax V math it implements — is Ch 16.

References

• crates/muser-engine/src/decode.rs:182-190 — MetalKvPlane.
• crates/muser-engine/src/decode.rs:228-261 — zero-filled and uninitialized constructors.
• crates/muser-engine/src/decode.rs:263-314 — append / append_batch (the ring arithmetic; quoted).
• crates/muser-engine/src/decode.rs:374-417 — detached_from and the rotation-preserving restore (quoted).
• crates/muser-engine/src/decode.rs:1344-1358 — per-layer-kind capacity and layout allocation (quoted).
• crates/muser-engine/src/decode.rs:5643-5745 — the store + route call sites in encode_token; :117-121 CacheDiscontinuity.
• crates/muser-engine/src/shaders/muse_reference.metal:979-992 — muser_kv_store_f16 (quoted).
• crates/muser-engine/src/shaders/muse_reference.metal:1203-1234 — muser_kv_store_batch_f16, both layouts (quoted).
• crates/muser-engine/src/metal/encode/attn.rs:635-655, 786-827 — the two store wrappers.
• crates/muser-engine/src/prefill.rs:15-17 — the no-placement-from-absolute-position invariant.
• crates/muser-engine/src/cache.rs:11-16 — PlaneEncoding (interchange encodings).
• crates/muser-engine/src/config.rs:13-21, 84-93 — layer counts, window, the partition rule.
• crates/muser-engine/src/lib.rs:7-15 — the position-free-NoPE summary.
• [docs/memory-footprint.md] — the 1,024 B/row formula and the 1.827/7.306 GB slot table (§15.7’s tags).
• [receipt phase4-disagg-20260820/130815-g900091/] — the 1,823,184,896 B deep payload (payload_bytes verified).
• [docs/kvpack-merge-handoff §6] — the ~82 MB SWA / 95.7 % NoPE pacing decomposition.
• [docs/decode-dispatch-gap-20260815.md] — the 52 publication splits and 39 SWA staging rows (§15.9).
• Ch 16 — the read side.
• Ch 22, Ch 23, Ch 24 — footprint, server policy, the portable format.
• [ferrite-book Ch 14] — the ancestor’s paged-Q8 cache (the 2×2 device and the paging contrast, ported as contrast only).

Chapter 16 — Attention: the decode kernel ladder

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 (SIMD groups, simd_sum, barriers), Ch 9 (GQA 32:2, the two layer classes), Ch 14 (rotated Q/K), Ch 15 (the ring, the growing plane, and the explicit-origin arithmetic this chapter’s kernels index with). Attention math is built from zero here; no transformer background is assumed.

16.0 First: which kernel actually runs — the route ladder

The previous chapter left the keys and values sitting in the cache: appended, addressable through explicit origins, and so far unread. This chapter is where something finally reads them back. The natural way to write it would be to open the engine, find the attention kernel, and walk you through it line by line.

That is the one thing we cannot honestly do. There is no single attention kernel on Muser’s decode path. There are four routes, selected per layer, per token by predicates computed after the ring append of Ch 15. Presenting any one of them as the decode attention kernel would be this book’s fastest way to lie to you, and it is a lie with precedent: the ancestor book made exactly that mistake once and had to correct it in public [ferrite-book Ch 15]. So we start where the engine starts — with the code that chooses. Here is the selection, verbatim, from the middle of encode_token:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5643
let write_physical = self.cache[layer_index].append(layer_index, position)?;
let plane = &self.cache[layer_index];
let strict_attention = std::env::var_os("MUSER_CROSS_VENDOR_QK").is_some();
let llama_vec_rows = (strict_attention || self.kernels.has_llama_flash_attn_vec())
    && plane.len > 0
    // The pinned vec kernel rounds KV reads to a 32-row block.
    // A deliberately tiny raw session can have a smaller backing
    // allocation, so taking the vec path would read past it and
    // poison the full distribution with NaNs.
    && plane.capacity >= 32
    && (plane.origin_physical == 0 || plane.len == plane.capacity);
// Token-major SWA cannot use llama's pad kernel (nb11 is a full
// token row, not one head). Only take that path when the window
// is a multiple of 32 so the vec kernel never pads.
let llama_swa = llama_vec_rows && plane.len.is_multiple_of(32);
}

And the four routes those predicates pick between:

  layer class      vec-eligible?           route
  ──────────────   ─────────────────────   ─────────────────────────────────────────────
  SWA (39)         llama_swa               kv_store_f16 → barrier → llama vec (pinned)
  SWA (39)         not llama_swa           kv_store_f16 → splitk producer → splitk reduce
  NoPE (13)        llama_vec_rows          kv_store_batch_f16 → barrier → llama vec (pinned)
  NoPE (13)        not llama_vec_rows      ferrite interleaved producer → reduce_v2

Figure 16.0: the decode-attention route ladder. “Pinned” = a kernel from the llama.cpp metallib of Ch 4; the others are Muser-owned (splitk) or ferrite-lineage (interleaved). Decode the predicates, because each clause is a lesson:

• has_llama_flash_attn_vec() — is the pinned metallib loaded? No library, no vec route; the ladder falls to Muser’s own kernels.
• plane.len > 0 — attention needs at least one visible row.
• plane.capacity >= 32 — the pinned vec kernel rounds its KV reads to a 32-row block; a tiny diagnostic session with a smaller backing allocation would be read past its end, and the comment names the failure: “poison the full distribution with NaNs” (decode.rs:5648-5651). A fail-closed predicate against a silent OOB.
• origin_physical == 0 || len == capacity — the vec kernel reads a contiguous span of cache rows. An unwrapped plane (origin_physical == 0) is contiguous by construction; a full wrapped ring is also usable, because every slot is in-window and softmax is permutation-invariant (metal/encode/attn.rs:431-435). A partially-wrapped ring is not contiguous, so it never takes the vec path.
• llama_swa = … && plane.len % 32 == 0 — one more SWA-only clause: the token-major SWA plane’s row is a full token row (nb11 is 256 halves, not one head’s 128), so llama’s padding kernel cannot patch a ragged 32-row block; the vec path is taken only when the window length never pads (decode.rs:5654-5657).

One subtlety about that last-but-one clause, before we trust it too far: reading a full wrapped ring through the vec kernel is mathematically valid, but it is not bit-identical to llama’s own SWA addressing — which is why the serving batch graph, where bit-parity is the contract, stages wrapped SWA rows into llama’s absolute, 256-row-padded indices first, “so the pinned vec kernel sees the same reduction lanes rather than a mathematically equivalent compact permutation” (metal/encode/attn.rs:140-143). The teacher-forced graph this chapter narrates reads the ring directly. Two graphs, two answers to the same question — the recurring Part IV pattern.

It is worth flagging that distinction now, before a single kernel: the gap between the same answer and the same floating-point answer is the hinge every route decision in this chapter turns on. Keep it in view; it comes back as the sharpest tradeoff at the end.

16.1 What attention computes — from zero

We have just watched four routes argue with each other without saying what any of them computes. They all compute the same thing, so it is worth building that thing from nothing before we look at a kernel again.

Everything else in the transformer operates on one token’s vector. Attention is the one operation that reaches sideways: while generating token t, it is the moment token t may look at tokens 0..t and mix information out of them. The folklore naming is information retrieval: the Query (Q) is what this token seeks; each past token’s Key (K) is what it offers for matching; its Value (V) is the payload returned on a match.

Step 1 — the score is a dot product. For query head h, current query Q_h ∈ ℝ¹²⁸, and the key of past token t in the same head’s group K_t[h] ∈ ℝ¹²⁸:

score(t) = Q_h · K_t[h] = Σ_{d=0..127} Q_h[d] · K_t[h][d]

Step 2 — scale by 1/√head_dim. A sum of 128 random products has standard deviation growing like √128; unscaled, it would push softmax into saturation where one token takes all the weight. Muse Glimmer’s scale is 1/√128 ≈ 0.0883883 (config.rs:277-281; the companion test at config.rs:426-430 also checks it against the folded QK-norm factor). Note this is in addition to the qk_scale_factor ≈ 3.87 baked into the Q-norm weights (Ch 14) — two scales, two places, asserted apart.

Step 3 — softmax over tokens. With scaled scores s_t, the attention weights are a probability distribution:

w(t) = exp(s_t) / Σ_{t'} exp(s_{t'})

Step 4 — weighted sum of Values. The head’s output is out_h = Σ_t w(t) · V_t[h]. All weight on token 3 → copy token 3’s value; uniform weight → the mean. In one line:

out_h = softmax_t( (Q_h · K_t[h]) / √128 ) · V_t[h]

A fully worked example: head_dim = 2, three tokens

Smallest non-trivial case, every number by hand:

                    head_dim = 2, √head_dim ≈ 1.4142
  Query:    Q  = [1, 0]
  Keys:     K0 = [1, 0]   K1 = [0, 1]   K2 = [1, 1]
  Values:   V0 = [2]      V1 = [3]      V2 = [4]      (1-D values for readability)

  1) raw scores:      s0 = 1·1+0·0 = 1 ;  s1 = 1·0+0·1 = 0 ;  s2 = 1·1+0·1 = 1
  2) scaled ÷√2:      [0.7071, 0.0000, 0.7071]
  3) softmax:
       row max m       = 0.7071
       s_t − m         = [ 0.0000, −0.7071,  0.0000]
       exp(s_t − m)    = [ 1.0000,  0.4931,  1.0000]
       Z = Σ           = 2.4931
       weights = /Z    = [ 0.4011,  0.1978,  0.4011]      sums to 1 ✓
  4) output:
       out = 0.4011·2 + 0.1978·3 + 0.4011·4
           = 0.8022 + 0.5934 + 1.6044 = 3.0000

Figure 16.1: attention end to end at head_dim 2. Q=[1,0] sees only each key’s first dimension, so K0 and K2 tie at the top; the output sits between V0=2 and V2=4, pulled toward V1 by the middle weight. Step 3’s − m line is the max-subtraction trick: subtracting the row maximum before exp is algebraically exact (the exp(−m) cancels between numerator and denominator) and numerically mandatory — exp(200) is +inf in f32 and inf/inf is NaN. Every kernel below carries it.

16.2 Online softmax — the running-max table

The formula we just wrote has an inconvenient property: its denominator sums over every score, so no weight can be finalized until the last token has been seen. What does a kernel do about that? The literal answer is to keep everything. Materializing all visible scores and then doing softmax in a second pass (the ancestor’s score-buffer design) costs a buffer sized by context — comfortable at short depth, and a cost that grows with exactly the thing you want to grow. The flash/online formulation instead keeps three running quantities per worker — max M, denominator S, accumulator O — and folds each new score in:

  new_M = max(M, s)
  corr  = exp(M − new_M)              ← rescale: old stats were normalized
                                        against the wrong (smaller) max
  S     = S·corr + exp(s − new_M)
  O     = O·corr + exp(s − new_M)·V
  M     = new_M

Run it on Figure 16.1’s three scores in arrival order [0.7071, 0.0000, 0.7071]:

  start:        M = −∞,  S = 0,  O = 0
  token 0:      new_M = 0.7071; corr = exp(−∞−0.7071) = 0
                S = 0·0 + 1.0000 = 1.0000
                O = 0·0 + 1.0000·2 = 2.0000
  token 1:      new_M = 0.7071 (unchanged); corr = exp(0) = 1
                S = 1.0000 + 0.4931 = 1.4931
                O = 2.0000 + 0.4931·3 = 3.4793
  token 2:      new_M = 0.7071 (unchanged); corr = 1
                S = 1.4931 + 1.0000 = 2.4931   ← matches Figure 16.1's Z
                O = 3.4793 + 1.0000·4 = 7.4793
  output:       O / S = 7.4793 / 2.4931 = 3.0000 ✓

Figure 16.2: online softmax as a running table. The corr rescale fired only on the first row (max moved from −∞); had a later token beaten 0.7071, every earlier statistic would have been scaled down by exp(old − new) in one multiply. Two workers each running this over half the tokens merge the same way — each is a “token” whose (M, S, O) combines with the other’s — which is how the split across SIMD groups and workgroups below stays exact.

Say that the other way round, because it is the load-bearing idea of the whole chapter. The triple (M, S, O) is a complete summary of every token a worker has looked at; nothing else about those tokens is ever needed again. So attention can be cut into arbitrary pieces, chewed in any order, on any number of workers, and reassembled with no approximation anywhere. That single property is what lets three different kernels share one mathematics while disagreeing about nearly everything else — and, as we will see, it is also what makes them disagree in the last bits of the result.

16.3 GQA 32:2 — sixteen query heads per KV head

Muse Glimmer has 32 query heads and 2 KV heads — grouped-query attention at 32:2, i.e. heads_per_kv = 16 (config.rs:274-276; geometry from muse_golden.rs:97-99). Queries fan in; the cache does not:

  query heads:  Q0 … Q15        Q16 … Q31
                   └── KV head 0 ┘└── KV head 1 ┘
  K/V planes:        K0,V0              K1,V1      (2 planes, not 32)

Figure 16.3: the 16:1 fan-in. Every kernel maps a query head to its KV head by kv_head = head / heads_per_kv (muse_reference.metal:1077-1078, flash_attn_decode_vec_contiguous_f16.metal:519). The bandwidth win is the point: the KV cache — and every attention read of it — is 16× smaller than full multi-head attention at the same head count [arxiv:2305.13245]. It is also why the store kernels of Ch 15 write 256-element rows (two heads’ worth), not 4,096.

16.4 SWA masking against the ring position

Which past tokens are visible? For a sliding layer at absolute position position with window W = 2,048:

  visible      = min(position + 1, W)          (this token included)
  logical_start = position + 1 − visible        (the window's first token)

and each visible logical token maps to its ring slot through Ch 15’s explicit origins:

  physical = (origin_physical + logical − origin_logical) % capacity

These three lines — not position % capacity — are how every Muser-owned attention kernel addresses the cache; they appear verbatim inside the splitk kernel below (muse_reference.metal:1079-1102). For a NoPE layer the same code with window = 0 degenerates to visible = position + 1, logical_start = 0 — the growing plane. Masking here is addressing: tokens outside the window are never read, so they need no explicit −inf mask. (Explicit masks do exist on Muser’s paths — for the pinned llama prefill kernel’s causal blocks and the staged SWA decode route — metal/encode/attn.rs:266-317; the decode ladder itself masks by address.)

16.5 The kernels — three rungs, one math

The rungs come in teaching order here, not in the order the predicates test them. Muser’s own splitk pair goes first, because its source is in the tree and it wears the online-softmax table on its sleeve. Then the pinned llama kernel that outranks it wherever bit-parity is the contract. Then the ferrite-lineage pair that catches NoPE layers when the metallib is missing. Read the first rung as the reference implementation and the other two as departures from it: everything that differs between them — who owns the reduction order, who reads the current token, who can address a wrapped ring — falls out of the situation each was built for.

16.5.1 The Muser splitk producer + reducer (SWA fallback)

This is the default SWA route whenever the pinned vec path is not eligible, and the clearest exhibit of §16.1–16.2 in code. The producer’s grid is (n_heads, n_workgroups) and its threadgroup is (32, n_simdgroups):

// crates/muser-engine/src/shaders/muse_reference.metal:1052
kernel void muser_attention_decode_splitk_f16(
    device const float *query [[buffer(0)]],
    device const half *key_cache [[buffer(1)]],
    device const half *value_cache [[buffer(2)]],
    device float *partials [[buffer(3)]],
    constant uint &n_heads [[buffer(4)]],
    constant uint &n_kv_heads [[buffer(5)]],
    constant uint &position [[buffer(6)]],
    constant uint &capacity [[buffer(7)]],
    constant uint &origin_logical [[buffer(8)]],
    constant uint &origin_physical [[buffer(9)]],
    constant uint &window [[buffer(10)]],
    constant uint &n_workgroups [[buffer(11)]],
    constant uint &n_simdgroups [[buffer(12)]],
    constant float &attention_scale [[buffer(13)]],
    threadgroup float *shared [[threadgroup(0)]],
    uint2 group [[threadgroup_position_in_grid]],
    uint lane [[thread_index_in_simdgroup]],
    uint simdgroup [[simdgroup_index_in_threadgroup]]) {
    const uint head = group.x;
    const uint workgroup = group.y;
    if (head >= n_heads || simdgroup >= n_simdgroups) {
        return;
    }
    const uint head_dim = 128;
    const uint heads_per_kv = n_heads / n_kv_heads;
    const uint kv_head = head / heads_per_kv;
    const uint visible = window > 0 ? min(position + 1, window) : position + 1;
    const uint logical_start = position + 1 - visible;
    const uint block_count = (visible + 31) / 32;
    const uint vector_offset = lane * 4;
    if (simdgroup == 0) {
        *((threadgroup float4 *)(shared + vector_offset)) =
            *((device const float4 *)(query + head * head_dim + vector_offset));
    }
    threadgroup_barrier(mem_flags::mem_threadgroup);
    const float4 q = *((threadgroup float4 *)(shared + vector_offset));

    float running_max = -3.402823466e+38f;
    float running_sum = 0.0f;
    float4 accumulator = 0.0f;
    for (uint block = workgroup * n_simdgroups + simdgroup;
         block < block_count;
         block += n_workgroups * n_simdgroups) {
        const uint first_offset = block * 32;
        const uint count = min(32u, visible - first_offset);
        float scores[32];
        for (uint item = 0; item < count; ++item) {
            const uint logical = logical_start + first_offset + item;
            const uint physical =
                (origin_physical + logical - origin_logical) % capacity;
            const uint base = (physical * n_kv_heads + kv_head) * head_dim;
            const float4 key =
                float4(*((device const half4 *)(key_cache + base + vector_offset)));
            scores[item] = simd_sum(dot(q, key)) * attention_scale;
        }
        float block_max = scores[0];
        for (uint item = 1; item < count; ++item) {
            block_max = max(block_max, scores[item]);
        }
        const float next_max = max(running_max, block_max);
        const float old_factor = exp(running_max - next_max);
        accumulator *= old_factor;
        running_sum *= old_factor;
        running_max = next_max;
        for (uint item = 0; item < count; ++item) {
            const float weight = exp(scores[item] - running_max);
            running_sum += weight;
            // …(V pass: same logical→physical walk, accumulator += weight * V;
            //    elided — muse_reference.metal:1117-1127)…
        }
    }
    // …(threadgroup merge of the n_simdgroups partials and the per-workgroup
    //    [max, sum, weighted-V] write to `partials`; elided —
    //    muse_reference.metal:1130-1166)…
}

(V-pass and merge elided as marked; the elided lines are the same online-softmax steps against the value plane and the Figure 16.2 merge.) The anatomy:

• Q staged once — SIMD group 0 copies the head’s 128-wide Q into threadgroup memory as float4s; every lane then holds one float4 slice (lane * 4), reused for all 32-token blocks.
• The score — dot(q, key) is a 4-lane partial; simd_sum folds the 32 lanes’ partials into one score. 32 lanes × 4 = the whole 128-dim dot product per token.
• Logical 32-token blocks, distributed round-robin — block index workgroup * n_simdgroups + simdgroup, stepping by n_workgroups * n_simdgroups: each SIMD group owns every (n_workgroups × n_simdgroups)-th block, so every block is owned exactly once (asserted by test, metal/encode/attn.rs:1001-1019).
• Online softmax per block — block max, then the old_factor rescale of Figure 16.2, then weights and the V-accumulate.
• Partials [max, sum, weighted-V] — one per (head, workgroup), stride 2 + 128 floats (attn.rs:742-743).

Where do the workgroup and SIMD-group counts come from, and why cap them at all? splitk_geometry (attn.rs:888-896) answers the first half: blocks of 32 visible tokens, workgroups capped at MAX_DECODE_SPLIT_WORKGROUPS = 32, SIMD groups growing 1→4 as visibility demands. The second half is a fork we lost. The geometry inherited from Ferrite put occupancy first and capped workgroups far above llama’s fixed launch, on the reasonable theory that more resident work hides more latency. On the deep, growing planes it did the opposite: the extra workgroups arrived with too little to chew on. The comment on the constant records the result in the codebase’s own words:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:41
// llama.cpp's Metal `flash_attn_ext_vec` always launches `nwg = 32` and only
// grows simdgroups (1→4) once `2 * nwg * nsg * 32 < visible`. Ferrite's
// occupancy-first cap of 96 oversubscribed the 13 full/NoPE planes and is
// the depth-rent we lose to llama as context grows. Keep short-context
// `nwg = min(blocks, 32)` so TG512 does not pay empty workgroups.
pub(crate) const MAX_DECODE_SPLIT_WORKGROUPS: usize = 32;
}

Read that comment twice. One half is the setting we shipped and the llama launch rule it now matches. The other half is a confession: the earlier cap oversubscribed the deep planes, and what we still lose to llama as context grows is named out loud, in the source, as rent. Nobody had to write that second half down — a tidier codebase would have left it in a commit message nobody reads. Muser’s house style is that the deficit lives next to the constant that causes it.

The reducer then finishes the job the producer deliberately left open. Each workgroup handed up a partial, and the reducer merges those partials with exactly the combine the producer used inside its own blocks: correction = exp(part[0] − global_max), then global_sum += part[1] * correction, then accumulate the weighted values, then divide — once, at the very end. That final single divide is why the producer normalized nothing along the way: an early normalization would only have to be undone. The code is muser_attention_decode_splitk_reduce_f32, sibling of muser_attention_decode_splitk_reduce_f16 (muse_reference.metal:1169-1201), dispatched at attn.rs:776-783. One detail there is worth stealing for your own encoders: the barrier before it is scoped to the partials buffer alone, “instead of stalling every buffer used by the 52-layer command buffer” (attn.rs:771-773).

16.5.2 The pinned llama vec kernel — kernel_flash_attn_ext_vec_f16_dk128_dv128

Why would an engine that has a perfectly good attention kernel of its own hand two of its four routes to somebody else’s binary? Park the question through the bullets; the answer arrives at the end of the section, and it is not about speed.

The vec-eligible routes dispatch llama.cpp’s own flash-attention decode kernel from the pinned metallib. Like Ch 13’s matvec, the body is pinned binary provenance — not in the Muser tree, not quoted here — and what Muser owns is a meticulously shaped dispatch (metal/encode/attn.rs:437-633):

• Strides describe the plane. Head-major NoPE planes bind as ns10 = 128 (one head row per KV row); token-major SWA rings as ns10 = 256 (a full token row per KV row) — the two GgmlMetalKargsFlashAttnExtVec layouts at attn.rs:503-511.
• Split-K by construction. llama’s kernel launches nwg = 32 workgroups per head (LLAMA_FA_NWG, encode.rs:1066) and grows SIMD groups 1→4 the same way Muser’s splitk does (attn.rs:496-499); each writes a partial, and llama’s own kernel_flash_attn_ext_vec_reduce merges them (attn.rs:616-630).
• Ragged visibility handled llama’s way. If visible % 32 ≠ 0, a kernel_flash_attn_ext_pad dispatch first rounds the tail (attn.rs:525-559) — which is exactly what the SWA len % 32 predicate avoids needing on the token-major plane, where llama’s pad kernel “would read past” the wrong-shaped row (decode.rs:5654-5657).

Now the parked question. The pinned-vs-own distinction is not performance vanity — nobody clocked llama’s kernel as faster and surrendered. It is arithmetic identity. The parity ledger’s Stage A close-out records two findings that close the door together: llama’s vec kernel “uses an intentionally different reduction DAG” than any Muser/Ferrite kernel, and “no untried llama scheduling transplant [was] compatible with the fixed production hash” [ledger, Stage A close-out]. Read them side by side — you cannot reschedule your way to llama’s bits, and the hash that defines production will not move to meet you. Matching llama’s bits meant adopting llama’s kernels on the routes where llama runs them.

16.5.3 The ferrite interleaved fallback (NoPE)

The last rung catches the case neither of the others can: a NoPE layer with no eligible vec route. Here the ladder falls to a ferrite-lineage pair kept deliberately unmodified — the producer flash_attn_decode_vec_f16_gqa_interleaved (shaders/ferrite/flash_attn_decode_vec_contiguous_f16.metal:494) and the reducer flash_attn_decode_reduce_v2 (shaders/ferrite/flash_attn_decode_reduce_v2.metal:4). The wrapper states the terms of that inheritance in one line: “Exact Ferrite a85048a90 cache-interleaved producer + LSE reducer for the growing NoPE planes. These planes are head-major and never wrap; SWA rings remain on Muser’s explicit-origin kernel” (attn.rs:185-187). That is a division of labour, not a preference: the ferrite pair is trusted exactly where the plane’s shape matches the assumption it was written under, and nowhere else.

Its signature is the prelude ABI — function constants bake head_dim = 128 and the SIMD-group count into the pipeline at PSO build (encode.rs:798-835). Its grid is (n_heads, n_workgroups), launched so that sibling Q heads land adjacent; its own header calls this a “schedule-only interleaved sibling” (…contiguous_f16.metal:487-493), and “schedule-only” is the load-bearing half of that phrase — the interleaving changes which head runs beside which, never what any of them computes. Each workgroup then merges its SIMD groups into one legacy [M, S, O] partial, and reduce_v2 combines those with the same precise::exp(p[0] − global_max) correction of §16.2. (The “LSE” in that quoted header is log-sum-exp — the (M, S, O) statistics of §16.2 under their textbook name.)

This route also owns the current-token KV bypass: the producer takes k_cur/v_cur as buffers 4–5, one simdgroup stores them into the plane as a side effect (…contiguous_f16.metal:534-543), and — per the code’s own comment — “Every workgroup still reads the current token from k_cur/v_cur”: the freshest row is consumed as f32 from the activation buffer, never round-tripped through the f16 plane. The pinned-vec and splitk routes do not bypass: they store first and read the plane back (the vec route with an explicit barrier between, decode.rs:5669-5670). So the answer to “does Muser bypass the current token?” is yes on the ferrite rung only — a property of which ladder step you stand on, not of the engine.

16.6 The Rust dispatch — the ladder in one table

Everything above lands in four call sites in encode_token, one per rung. Read the table as the ladder seen from the encoder — and read the last column first, because that is where the previous chapter’s store and this chapter’s read have to be ordered against each other, and each route answers that differently:

That last column is the fused-versus-staged argument compressed into four cells. Three routes store the row and then read it back, so they need an ordering guarantee — a full barrier, or the narrow one scoped to the partials. The ferrite rung needs neither, because the value never leaves the kernel that produced it.

The vec wrapper call itself carries the whole story in its argument list — visible, capacity, origin_physical, head_major, the pad and partials scratch — decode.rs:5671-5692 (SWA, head_major = false) and decode.rs:5748-5769 (NoPE, head_major = true).

16.7 The access pattern — when attention starts to matter

Now the question this chapter owes the bandwidth story: at what context length does attention stop being a rounding error and become the thing that costs? The derivation is short enough to do in full, so do it in full rather than trusting anyone’s intuition about it.

Per token per layer, the attention read is the visible window’s KV: visible × n_kv_heads × 2 planes × head_dim × 2 B = visible × 1,024 B (the per-layer row cost of Ch 15). Derive the shape across depth:

  SWA layers (window caps at 2,048):  39 × 2,048 × 1,024 ≈ 81.8 MB  (constant)
  NoPE layers at depth D:             13 × D × 1,024
  weights per token:                  16,756,681,056 B ≈ 16.76 GB    (constant)

  at D = 2,048:    KV ≈ 0.17 GB   ≈ 1 % of the weight stream
  at D = 32,768:   KV ≈ 0.44 GB + 0.08 GB ≈ 2.7 %
  at D = 131,072:  KV ≈ 1.74 GB + 0.08 GB ≈ 1.83 GB ≈ 10.9 %
  crossover with weights: 13 × D × 1,024 = 16.76 GB ⇒ D ≈ 1.26 M tokens
      — beyond the model's 131,072 limit

So unlike smaller models (the ancestor’s 1.5 B crossed over near 65 K context [ferrite-book Ch 14] — Ferrite-lineage arithmetic), weights dominate Muse Glimmer’s decode at every context the model can hold; attention’s KV read grows linearly but never overtakes within the limit. That is derived arithmetic from the geometry, not a measurement — and it is also why the splitk workgroup cap of §16.5.1 (“the depth-rent we lose to llama as context grows”, decode.rs:41-46) is about latency and occupancy at depth, not about bytes: 13 NoPE layers × 131 K rows of strided f16 is plenty to expose an under-parallel kernel even at 10 % of the byte budget.

16.8 Tradeoffs

A ladder, not a kernel — and the evidence for each rung. The ancestor faced this same question and got it wrong first: its position ladder was ground-truthed by route logging and then corrected once [ferrite-book Ch 15]. Muser’s ladder is at least legible, since the predicates are in source and quoted at the top of this chapter. Legible is not the same as justified, though, so here is how the rungs earned their places.

The fork came while the route question was still open and attention was the prime suspect for the decode gap. The attractive move was to specialize. Decode issues exactly one query, so a one-query GQA FA2 kernel ought to beat a general flash-attention kernel that is still carrying prefill’s machinery around with it. We wrote that kernel and ran it on the streamed diagnostic, fully expecting the specialist to win. It came in at 28.290 tok/s median against llama’s 33.428 — a ratio of 0.8463×, the specialist losing to the generalist it was built to beat. We kept the measurement: it sits in the landed-and-rejected table [docs/decode-dispatch-gap-20260815.md §Landed and rejected reductions], banked as evidence rather than shipped as a claim.

What the loss taught is that the kernel’s shape was never the lever. The lever was the reduction order — and the only way to get llama’s reduction order is to run llama’s kernel. Transplanting the attention DAG bit-exactly is what moved decode from 0.781× (single-sample, 2026-08-14) toward the six-depth matrix above parity [ledger, Arc 1]. So the ladder is neither a hedge nor an accident of history. It exists because each rung is the exact kernel for its situation: llama’s for parity-critical contiguous reads, Muser’s splitk for wrapped rings the pinned kernel cannot address, ferrite’s for the growing head-major planes without the metallib.

Wrapped-ring vec read vs staging — mathematical validity vs bit-parity. §16.0’s full-ring clause is sound mathematics (permutation-invariant softmax, attn.rs:431-435) and still not what serving does: the batch graph stages wrapped SWA into llama’s padded indices so “the pinned vec kernel sees the same reduction lanes rather than a mathematically equivalent compact permutation” (attn.rs:140-143). The 39 staging groups that decision costs live in the gap accounting (§16.9). This is the book’s exactness-vs-equivalence distinction at its purest: two routes with identical softmax outputs in exact arithmetic, one of which reproduces llama’s floating-point reduction order and one of which does not. Put plainly, serving pays dispatch groups for an answer it already had — because it needs that answer to arrive in llama’s order, not merely to be correct.

Splitk’s own kernel vs pinned-everything. Why keep a Muser-owned attention kernel at all when the metallib is loaded? Because the pinned vec kernel cannot read a partially-wrapped token-major ring contiguously — the predicate excludes it — and re-staging every decode token (rather than every wrapped prefill chunk) would add a per-token copy dispatch. The splitk producer walks the ring’s explicit origins natively (muse_reference.metal:1100-1102). The price is a different reduction DAG from llama’s on that rung — acceptable on the teacher-forced/diagnostic graph, and one reason the serving route prefers eligibility for the pinned kernel whenever the predicates allow.

32-token blocks, 32-workgroup cap. The block size matches the pinned kernel’s read granularity (ncpsg = 32, encode.rs:1067) and the workgroup cap matches llama’s fixed nwg = 32 — with the in-source admission that an earlier occupancy-first cap “oversubscribed the 13 full/NoPE planes and is the depth-rent we lose to llama as context grows” (decode.rs:41-46). Owning the tradeoff in a constant’s comment is this codebase’s house style; the rent itself is the kind of measured deficit Ch 40 catalogs.

16.9 Where the gap lives

Which of this chapter’s dispatches actually show up in the decode-gap accounting, and could any of them be removed? Two families again, plus one experiment. The 52 KV-publication splits — store dispatch + attention dispatch as separate closures in production — are “session/publication structure; Keep” [docs/decode-dispatch-gap-20260815.md]: combining the closures would not remove either kernel’s math, and the splits are what make the pinned-kernel-per-route discipline auditable. The 39 SWA staging groups are this chapter’s wrapped-ring story — staging exists because bit-parity with llama’s reduction lanes outranks the cheaper direct read (§16.8). And the attention-shaped row in the landed-and-rejected table — the one-query GQA FA2 at 0.8463× — is the measured trace of the route hunt that J0/J1 eventually resolved by changing the anchor. Attention is not the gap’s largest family; it is the gap’s most instructive one.

16.10 What comes next

activations.attention now holds 32 heads × 128 floats of weighted-past mixture. Muse Glimmer does something unusual with it before the output projection: multiplies it element-wise by sigmoid(gate), where gate is the fourth matvec of Ch 13 that has been waiting in activations.gate all along. That sigmoid gate — and the o_proj that follows it — is Ch 17.

References

• crates/muser-engine/src/decode.rs:5643-5792 — the route predicates (quoted) and all four dispatch branches.
• crates/muser-engine/src/shaders/muse_reference.metal:1052-1167 — muser_attention_decode_splitk_f16 (quoted in §16.5.1; V-pass and merge elided as marked).
• crates/muser-engine/src/shaders/muse_reference.metal:1169-1201 — the splitk reducer.
• crates/muser-engine/src/metal/encode/attn.rs:437-633 — the pinned vec wrapper (strides, ns10 split, pad, reduce).
• crates/muser-engine/src/metal/encode/attn.rs:707-784 — the splitk wrapper and its scoped barriers; :888-896 splitk_geometry (with the pinned-schedule test at :976-983).
• crates/muser-engine/src/metal/encode/attn.rs:185-257 — the ferrite interleaved wrapper; :102-183 the SWA staging kernels and their llama-lanes comment.
• crates/muser-engine/src/metal/encode.rs:149-167, 1066-1072, 1079-1249 — LlamaFlashAttnPipelines, LLAMA_FA_NWG/NCPSG, and the vec/pad/reduce pipeline construction.
• crates/muser-engine/src/shaders/ferrite/flash_attn_decode_vec_contiguous_f16.metal:487-554 — the interleaved producer: header, fused store, and the k_cur/v_cur bypass comment.
• crates/muser-engine/src/shaders/ferrite/flash_attn_decode_reduce_v2.metal:4-48 — the LSE reducer.
• crates/muser-engine/src/decode.rs:41-46 — MAX_DECODE_SPLIT_WORKGROUPS and the depth-rent comment (quoted).
• crates/muser-engine/src/config.rs:274-281 — heads_per_kv, attn_scale; :426-430 the scale test.
• [docs/decode-dispatch-gap-20260815.md] — the closure families of §16.9 and the 0.8463× GQA-FA2 diagnostic row.
• [ledger] (docs/goal-parity-ledger-2026-08.md) — Arc 1 (0.781× → the six-depth matrix) and the Stage A close-out’s pinned-kernel audit.
• [docs/memory-footprint.md] — the 1,024 B/row KV constant §16.7 derives from.
• [arxiv:1706.03762] — Vaswani et al., Attention Is All You Need (the scaled dot-product formula).
• [arxiv:2305.13245] — Ainslie et al., GQA (the 16:1 bandwidth lever).
• Ch 15 — the planes and origins this chapter indexes; Ch 17 — the gate that consumes this chapter’s output; Ch 36 — the prefill-side attention routes.
• [ferrite-book Ch 15] — the ancestor’s attention chapter (the worked-example and online-softmax devices ported; its position-ladder correction is the reason §16.0 leads this one).

Chapter 17 — The sigmoid gate and the output projection

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 (threads, SIMD groups, dispatch_threads), Ch 9 (the attention-output gate in the architecture), Ch 12 (the sandwich norms and the fused dual-eps tail), Ch 13 (the pinned ggml matvec family — this chapter reuses it verbatim), Ch 16 (what attention writes into activations.attention). Sigmoid is defined from zero here; no prior exposure is assumed.

17.1 What it computes

Ch 16 ended with attention writing one vector: activations.attention, the concatenation of 32 head outputs of 128 elements each — attn_dim = 32 × 128 = 4,096 floats (config.rs:268-270). In most transformer families that vector goes straight to the output projection. Muse Glimmer does something extra first: before the attention result is allowed to rejoin the residual stream, the layer asks a second, learned question about every one of those channels — should this one be heard at all?

The answer arrives as another vector. A fourth attention projection — the gate, sibling of Q, K, and V from Ch 13 — produced a second 4,096-vector, and this chapter’s kernel multiplies the two element-wise:

attn_out[i]  ←  attn_out[i] · σ(gate[i])        for i in [0, 4096)

where σ is the logistic sigmoid, defined from zero:

σ(x) = 1 / (1 + e^(−x))

The sigmoid maps any real number into the open interval (0, 1):

Table 17.1: the sigmoid at five points. Positive gate values pass the attention output through nearly unchanged; negative values suppress it; the transition is smooth, never a hard zero.

So the gate is a learned, per-channel volume knob on the attention result, and it lives in attention-output space — MuseConfig::attn_dim’s doc comment calls 4,096 “the space the attention-output gate lives in” (crates/muser-engine/src/config.rs:268-270). After the gate, the output projection (o_proj, the tensor attn_output.weight) maps the gated 4,096-vector back into the 6,656-wide residual stream:

delta = W_o · (attention ⊙ σ(gate))        W_o : [6656 × 4096], Q4_K
residual ← residual + post_norm(delta)     (the fused tail — §17.7)

Two operations, one chapter: a pointwise gate and a matvec you already know. Keep an eye on the asymmetry between them, because it is the chapter’s point. The cheap operation is the one that turned out to need a careful safety argument; the expensive one is the one with nothing to confess.

17.2 Why it exists — a gated attention output

There are two questions hiding in this section’s title, and they have very different evidence behind them. What is the gate, mechanically? is settled by reading the tree. Why did somebody train a model to want one? is not in the tree at all. Separating them is the whole discipline of this book, so we do it out loud: the verifiable facts first, the honest shrug second.

The gate is part of the model contract, asserted in the engine’s own header — “Parameterless QK-RMSNorm, sigmoid attention-output gate, Gemma-2-style sandwich norms” (crates/muser-engine/src/lib.rs:12). It is a real learned projection with its own tensor blk.{l}.attn_gate.weight of shape [6656, 4096] (config.rs:307), and it rides in the same concurrent four-projection set as Q, K, and V (Ch 13; decode.rs:5569-5598). The CPU oracle applies it strictly between attention and o_proj:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/reference.rs:446
// ── sigmoid gate, then o_proj ─────────────────────────────────
for g in gate.iter_mut() {
    *g = 1.0 / (1.0 + (-*g).exp());
}
// … (capture elided) …
for (a, g) in attn_out.iter_mut().zip(gate.iter()) {
    *a *= *g;
}
matmul(
    &self.w(&format!("blk.{il}.attn_output.weight")),
    &attn_out,
    t,
    &mut proj,
);
}

Now the second question. Why gate the attention output at all? The structural reading — the same decoupling you will meet again in the FFN of Ch 18 — is that the gate separates whether a channel’s attention result flows onward from what that channel carries. A sigmoid-squashed multiplier in (0, 1) can near-zero a head-output channel that the layer has decided is noise for this token, while passing useful channels through. But the model’s own training rationale is not in the Muser tree, and this book does not invent psychology: [unverified] why the authors chose a sigmoid (rather than tanh or ReLU) gate for Muse Glimmer specifically. What the code proves is the wiring, the shape, and that skipping it is not an option — remove the gate and every downstream bit changes, because the oracle order above is the parity specification the Metal graph must reproduce (reference.rs is “the correctness gate”, lib.rs:99-109).

A second-order effect worth naming: because σ(x) > 0 always, the gate can damp a channel but never invert it — and because the gate is applied before o_proj, it scales each attention-head channel while they are still separate, before the heads are mixed by W_o.

17.3 The operation, explained — a worked gate by hand

Before trusting a kernel, it is worth doing its job once by hand, at a size small enough to check on paper. The gate makes that easy: it is as simple as GPU work gets — element-wise, no reduction, no mixing, every output element depending on exactly one attention value and one gate value. A hand example with a 4-element slice of the two vectors:

  i :    0      1      2      3
  gate:  +2.0   −1.0   0.0    −4.0
  σ(g):  0.881  0.269  0.500  0.018

  attn:  1.50   2.00   0.25   8.00
  out:   1.32   0.54   0.125  0.14     ← attn[i] · σ(gate[i])

Every element is independent — element 3’s attention value of 8.0 is large, but its gate is nearly closed (σ(−4) = 0.018), so 98.2 % of it is suppressed. That is the whole gate.

The operation that follows is where all the bytes are, and it asks nothing new of you. The o_proj after the gate is the matvec of Ch 13 at a new shape — 6,656 output rows, each a dot product over 4,096 inputs, which is 4096/256 = 16 Q4_K super-blocks of 144 bytes = 2,304 bytes of weight per row. No new math; the shapes, in Figure 17.1:

  gated attn [4096]        W_o [6656 × 4096] (Q4_K)          projected [6656]
  ┌           ┐   ┌──────────────────────────────┐   ┌            ┐
  │  4,096 f32│ × │ 6,656 rows × 2,304 B/row    │ = │ 6,656 f32  │
  └           ┘   └──────────────────────────────┘   └            ┘
    16 KiB read        15,335,136 B ≈ 15.34 MB          26 KiB write
                       (read once per layer)

Figure 17.1: The gate (pointwise, 4096 wide) feeding the o_proj matvec (4,096 → 6,656, Q4_K). The weight stream dominates: 15.34 MB against 42 KiB of activation traffic.

17.4 The Metal kernel — sigmoid_gate_inplace

How much Metal does a per-channel volume knob need? Less than the paragraph that describes it. Here is the whole kernel, verbatim — it is nine lines and worth reading as one piece before we take it apart:

// crates/muser-engine/src/shaders/ferrite/sigmoid_gate.metal:4
// Element-wise sigmoid gating: attn_out[i] *= sigmoid(gate[i])
// dispatch: (ceil(n/1024), 1, 1) × (1024, 1, 1)

kernel void sigmoid_gate_inplace(
    device       float* attn_out [[ buffer(0) ]],
    device const float* gate     [[ buffer(1) ]],
    constant     uint&  n        [[ buffer(2) ]],
    uint gid [[ thread_position_in_grid ]])
{
    if (gid < n) {
        attn_out[gid] *= 1.0f / (1.0f + exp(-gate[gid]));
    }
}

Line by line:

• Buffer 0, attn_out — the attention result, and note it is device float*, not const. This kernel mutates its input in place: the gated value overwrites the ungated one. There is no third buffer.
• Buffer 1, gate — the gate projection’s output, read-only.
• Buffer 2, n — the element count (4,096 on this model), bound as a 4-byte inline constant.
• gid — the global thread index; each thread owns exactly one element gid, and the if (gid < n) guard covers the ragged tail when the grid rounds up past n.
• The body — literally the formula of §17.1: one exp, one add, one divide, one multiply-into-place. Per element: two global reads (attn_out, gate), one global write (attn_out).

That *= is the only subtlety, and §17.7 is about why it is safe.

Now back to the top of that listing, because its header comment is a small trap and it is worth walking into deliberately once. We read dispatch: (ceil(n/1024), 1, 1) × (1024, 1, 1) as a specification and went to the Rust wrapper expecting to find threadgroups of that width being launched. They are not there. dispatch_1d calls dispatch_threads with threadgroup width min(n, 256) (crates/muser-engine/src/metal/encode.rs:1337-1342), so the comment describes a dispatch shape the engine no longer uses. The detour taught more than the discrepancy is worth on its own: the half of the comment that constrains correctness — one thread per element, ragged-tail guarded — holds either way, and the half that drifted is the half nothing checks. So we flag it code-wins style and move on. A kernel header is a hypothesis about the code, dated the day it was typed; we kept the pointer to both sides of this one [crates/muser-engine/src/shaders/ferrite/sigmoid_gate.metal:5 versus crates/muser-engine/src/metal/encode.rs:1337].

One digression before the wrapper, and it announces its own relevance: the pattern it shows up in recurs in nearly every kernel chapter after this one. This kernel has a twin. A strict-arithmetic sibling exists for the cross-vendor comparison lane: muser_cross_vendor_sigmoid_gate (shaders/muse_reference.metal:680) computes the same thing through the no-fast-math library’s controlled expf so a CUDA producer’s bytes can be matched bit for bit — same formula, different compile flags (gate.rs:14-15 selects it under MUSER_CROSS_VENDOR_QK). That is what an exactness lane looks like throughout Muser: never a different algorithm, always the same algorithm with the fast paths refused.

17.5 The Rust dispatch

The kernel is trivial, so the interesting question moves to the encoder: what must the wrapper guarantee before it is allowed to launch anything? Two things. The two vectors have to be the same length, or the element-wise product is quietly meaningless; and the launch has to land between attention and o_proj and nowhere else, because that is the order the CPU oracle fixed. The wrapper is four statements:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/gate.rs:7
pub fn encode_sigmoid_gate(
    &self,
    encoder: &ComputeCommandEncoderRef,
    values: &GpuBuffer,
    gate: &GpuBuffer,
) {
    debug_assert_eq!(values.len(), gate.len());
    if std::env::var_os("MUSER_CROSS_VENDOR_QK").is_some() {
        encoder.set_compute_pipeline_state(&self.cross_vendor_sigmoid_gate);
    } else {
        self.bind(encoder, "sigmoid_gate_inplace");
    }
    encoder.set_buffer(0, Some(values.metal()), 0);
    encoder.set_buffer(1, Some(gate.metal()), 0);
    set_value(encoder, 2, &(values.len() as u32));
    dispatch_1d(encoder, values.len());
}
}

In prose: bind the PSO (the strict twin only under the cross-vendor flag), bind values = activations.attention and gate = activations.gate (both 4,096 floats — the debug_assert enforces the equal lengths the element-wise product requires), push n as an inline constant, and launch one thread per element via dispatch_threads, i.e. grid (4,096, 1, 1) with 256-wide threadgroups and a guarded ragged tail (encode.rs:1337-1342).

Its call site in the token graph sits exactly where the oracle puts it — one dispatch closure after the attention route, before o_proj:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5793
dispatch(command, |encoder| {
    self.kernels.encode_sigmoid_gate(
        encoder,
        &self.activations.attention,
        &self.activations.gate,
    );
});
self.project(
    command,
    &layer.output,
    &self.activations.attention,
    &self.activations.projected,
);
}

self.project (decode.rs:5909-5917) is the same wrapper every projection uses: it routes by the loaded tensor’s dtype to encode_projection (decode.rs:6044), which for a Q4_K attn_output.weight and one token dispatches the pinned llama.cpp metallib kernel kernel_mul_mv_q4_K_f32 — the exact kernel family of Ch 13:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/qkv.rs:429
if tokens == 1 {
    if let Some(pipeline) = self.ggml_matvec(dtype) {
        let (block_bytes, rows_per_group) = match dtype {
            GgmlType::Q4_K => (144, 2),
            // … (Q5_K (176, 1), Q6_K (210, 2) elided) …
        };
        let args =
            GgmlKargsMulMv::for_matmul(n_out, n_in, block_bytes, rows_per_group as i32);
        encoder.set_compute_pipeline_state(pipeline);
        set_value(encoder, 0, &args);
        encoder.set_buffer(1, Some(weights.metal()), weights.offset() as u64);
        encoder.set_buffer(2, Some(input.metal()), 0);
        encoder.set_buffer(3, Some(output.metal()), 0);
        let simdgroups = 2usize;
        encoder.dispatch_thread_groups(
            MTLSize::new(n_out.div_ceil(rows_per_group * simdgroups) as u64, 1, 1),
            MTLSize::new(32, simdgroups as u64, 1),
        );
        return;
    }
    // … (fallback to Muser's own muser_matvec_q4k_4r2s elided;
    //     reached only when the metallib is absent) …
}
}

The PSO comes from the pinned metallib (encode.rs:278-280 registers ggml_q4k: kernel_mul_mv_q4_K_f32 and siblings). For o_proj, n_out = 6,656, rows_per_group = 2, simdgroups = 2, so the launch is 6,656 ÷ 4 = 1,664 threadgroups of 64 threads (two SIMD groups, two rows per group — every output row covered exactly once). The per-dtype table Q4_K → (144 B, 2 rows), Q5_K → (176 B, 1), Q6_K → (210 B, 2) is the same one Ch 13 introduced; the o_proj is Q4_K on the release artifact (attn_output 4096->6656 q4k, crates/muser-bench/src/m16.rs:163-166).

Put that another way, because it matters for everything the rest of the chapter argues: the expensive half of this stage is not a Muser kernel. Muser computes the shape arguments, binds three buffers, picks a grid — and then hands the actual arithmetic to a pipeline state compiled from someone else’s pinned metallib. Whatever the engine can be blamed for here happens before set_compute_pipeline_state, never inside the loop.

17.6 The access pattern — where the bytes go

Where does the time go in this stage? For every decode kernel in this book the honest first answer is bandwidth rather than arithmetic, so the way to read a stage is to count what it drags across the bus. Start with the cheap half. The gate kernel, per layer per token:

  read  gate[4096]        16,384 B   (16 KiB)
  read  attn[4096]        16,384 B   (16 KiB)
  write attn[4096]        16,384 B   (16 KiB)
                       ─────────────────────
                        49,152 B   (48 KiB)

The o_proj matvec after it:

  read  W_o (Q4_K)       6,656 rows × 16 blocks × 144 B = 15,335,136 B ≈ 15.34 MB
  read  gated attn        16,384 B  (16 KiB)
  write projected[6656]   26,624 B  (26 KiB)   (f32; 6,656 × 4)

The arithmetic for the weight figure, shown so you can re-derive it: a row of W_o spans cols = 4,096 inputs; Q4_K packs 256 elements per 144-byte super-block (Ch 6), so a row is 4096/256 × 144 = 2,304 B; there are rows = 6,656 of them; 6,656 × 2,304 = 15,335,136 B. Across all 52 layers that is 52 × 15,335,136 = 797,427,072 B ≈ 797 MB of the pinned 16,756,681,056-byte artifact — 4.76 % (arithmetic against the artifact size asserted at lib.rs:14). For scale, the whole five-projection attention block (Q + gate + K + V + O) is ≈ 48.4 MB per layer, while the FFN of Ch 18 is 224–259 MB per layer — Figure 17.2 itemizes the layer:

  per-layer weight read (kquant lane, derived arithmetic):
    attn_q      15.34 MB  (Q4_K)      ffn_gate  74.76 MB  (Q4_K)
    attn_gate   15.34 MB  (Q4_K)      ffn_up    74.76 MB  (Q4_K)
    attn_k       0.96 MB  (Q4_K)      ffn_down  74.76 MB  (Q4_K; 109.03 MB on
    attn_v       1.40 MB  (Q6_K)                 the Q6_K-down layers, Ch 19)
    attn_o      15.34 MB  (Q4_K)
    ──────────────────────            ─────────────────────────────
    attention  ≈ 48.4 MB              FFN      ≈ 224–259 MB

Figure 17.2: The weight budget of one Muse Glimmer layer, from the shape/dtype table of crates/muser-bench/src/m16.rs:139-198. The o_proj this chapter covers is a mid-size slice; the FFN is ~4.6–5.3× the whole attention block. The gate’s own projection (attn_gate) costs as much as the Q projection — gating is not free, it is another 15.34 MB stream.

The pattern to internalize: the gate’s 48 KiB is rounding error against the 15.34 MB that follows it. Like every decode kernel in this book, this stage lives or dies on weight bandwidth, and the gate barely moves bytes.

17.7 Why the in-place *= is safe — ownership and ordering

The attn_out[gid] *= … is a read-modify-write on a shared buffer. Two questions: can two threads collide inside the dispatch, and can another dispatch read attention while the gate is still writing it? Both are worth answering carefully, because a wrong answer does not crash — it produces a plausible token, sometimes, on some runs, which is the most expensive kind of bug this engine can have.

In-dispatch: no collision by construction. Each global thread index gid touches exactly one element, attn_out[gid], and no two threads share a gid. The write set is partitioned as finely as it can be — one element per thread. There is no reduction, no simd_sum, no threadgroup_barrier, and none is needed. (Contrast the attention kernels of Ch 16, where many threads cooperate per head and partials must be combined.)

Across dispatches: ordering is explicit and tracked. The token graph runs all 52 layers inside one command buffer on one concurrent compute encoder — the source records the contract at the top of the token route:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5449
// One concurrent encoder owns the complete token. Graph dependencies
// are explicit barriers; independent projection groups share a barrier
// interval and may overlap, matching the accepted Ferrite/llama route.
}

Muser’s buffers are allocated StorageModeShared with Metal’s automatic hazard tracking. That is a deliberate default, and we know it is deliberate because the other branch of the fork was taken first. Untracked buffers promise less driver bookkeeping between dispatches, and this graph already declares its dependencies explicitly, so we expected the tracking to be redundant work that could simply be switched off — free latency, no behaviour change. It was not redundant. With tracking off, the engine “empirically changed DFlash conditioning” — no kernel had been edited, and the observable behaviour moved anyway — so the experiment was reverted. The allocator still carries the name of the mode that survived [crates/muser-engine/src/metal/buffer.rs, shared_tracked]. The lesson generalizes well past this kernel, which is why it belongs in a chapter about a nine-line gate: in a fail-closed engine, an optimization that changes bits is not an optimization.

So the tracking stays, and a dispatch that reads attention after the gate’s write is ordered by the driver’s tracked-resource dependencies; where the graph needs finer control it issues targeted memory_barrier_with_resources calls (you saw them around the KV store in Ch 16, decode.rs:5669-5670). The o_proj that consumes the gated attention is therefore sequenced after the gate, and the next layer’s attention (which overwrites attention) after that — a producer→consumer chain, which is the dominant hazard shape of the whole decode graph (Ch 35 formalizes the taxonomy).

What Muser deliberately does not do here is fold the residual add into the o_proj matvec itself. This is a real fork, and the ancestor took the other branch of it: the Ferrite book taught exactly that device — a y[row] += dot write-back in the matvec kernel [ferrite-book Ch 16] — and it is a good device, cheaper by a dispatch and a buffer. Muser’s gate fuses nothing into the pinned matvec. The o_proj writes to a scratch buffer, activations.projected, and the residual add happens one dispatch later inside the fused dual-eps tail (encode_fused_norm_residual_rms_norm_32sg, decode.rs:5806-5818), where the in-place mutation actually lives: that kernel reads activations.normed (the running residual), adds the post-normed projection into it, and writes the result back in place.

Why give up the cheaper shape? The reason is the book’s recurring one, and the source states it outright: the legacy one-token graph with its Ferrite-lineage fused kernels “diverges from the source-pinned llama Metal graph enough to breach public logprob tolerance”, so serving routes one-token work through the batch graph that “dispatches the exact pinned kernels” (decode.rs:2085-2091). Keeping the o_proj a stock pinned-metallib matvec — unmodified write-back, = not += — is what makes its bytes match llama.cpp’s. Turn that around and it becomes the sentence to carry into the rest of the book: Muser buys exactness with dispatches. Every fusion the ancestor took for free, this engine pays for at the encoder, because it does not own the kernel it has to agree with. The fusion temptation is paid for elsewhere (Ch 19 prices it).

17.8 Tradeoffs

Three forks meet at this stage. Two of them were the engine’s to decide; the third was decided by whoever trained the checkpoint, and knowing which is which saves an afternoon of re-litigating a choice you cannot touch.

In-place gate vs materialized sigmoid buffer. The alternative — write σ(gate) to a fresh buffer, then a separate multiply — would traffic 16 + 16 + 16 + 16 + 16 = 80 KiB per layer instead of 48 KiB, and add one dispatch. The in-place form saves 32 KiB/layer ≈ 1.66 MB/token across 52 layers — real but tiny against the ~15.34 MB/layer o_proj weight stream, and tiny again against the ~800 MB/token whole-model read. This is a “no downside, take it” fusion of the pointwise kind: no reduction is moved, no rounding changes (each element’s arithmetic is identical; only the buffer destination differs), so it carries none of the exactness risk the norm fusions of Ch 19 do. [unverified] whether the 32 KiB/layer saving is individually measurable in end-to-end tok/s — no retained A/B isolates this kernel, and at 0.002 % of per-layer bytes it would be below noise by construction.

A separate gate closure vs fusing the gate into attention or o_proj. The gate could in principle be folded into the attention kernel’s epilogue (compute σ(gate[h·128+d]) while writing the head) or into the o_proj’s input load. The intuition pulls hard toward doing it: this book’s whole gap story is told in dispatch counts, and here is a dispatch that moves rounding error and computes one exponential per channel. Muser keeps it a standalone closure on every route anyway — teacher forced (decode.rs:5793-5799), batched serving, and the packed decode group (decode.rs:5112-5116, where per-row attention/gate buffers are gathered into shared batch buffers and one gate serves all rows).

We went to the closure-count accounting expecting to find the gate implicated somewhere, and it is not there. The measured consequence is structural rather than a timing number: in the one-token dispatch-gap accounting, the sigmoid-gate closures fall in the “common math closures” family that is identical in both the production and legacy graphs — 406 closures on each side, delta 0 [docs/decode-dispatch-gap-20260815.md, closure-count table]. So fusing the gate would trade a bit-exact match against the pinned graph for no reduction in the +196-closure gap at all, on the route that actually serves traffic. It stays separate because separate is exact.

Gate before o_proj vs gate after o_proj. The third fork was never ours to take, and it is worth saying so before anyone spends a week on it. The model applies the gate in attention space (4,096 channels, pre-mixing) rather than in residual space (6,656 channels). This is the checkpoint’s choice, mirrored by the oracle at reference.rs:446-464; an engine has no say in it. The consequence for the engine is only that the gate kernel is 4,096 wide rather than 6,656 wide — and that the gate projection [6656 → 4096] is one of the four concurrent matvecs of Ch 13, not a fifth sequential one.

17.9 Where the gap lives

Every kernel chapter in this part owes the same answer, and the question behind it is one of suspicion: does this stage help explain the extra dispatch closures the campaign is hunting? Here the answer is short, and it is the boring one.

This kernel is not the gap. Both of this chapter’s operations appear in the “Common math closures (including LM head/softcap)” row of the corrected closure-count diff — 406 production versus 406 legacy, delta 0 [docs/decode-dispatch-gap-20260815.md]. The one-token graphs differ in norm boundaries, SWA staging, KV publication, and one copy — none of them here. The o_proj streams 15.34 MB/layer through the same pinned llama.cpp matvec the comparator itself runs (Ch 13), so there is no engine-specific bandwidth story to tell about it either. The gate’s 48 KiB is four orders of magnitude below the o_proj’s weights. When the dispatch-gap chapters (Ch 35, Ch 40) hunt the +196, this stage is already exonerated.

The attention half of the layer is closed: context was gathered, gated, and mixed back toward the residual stream. What remains of the layer is the part with most of the bytes — the feed-forward block, two 74.76 MB Q4_K streams and a learned gate of its own. Ch 18 teaches the FFN from zero.

References

• crates/muser-engine/src/shaders/ferrite/sigmoid_gate.metal:7-16 — sigmoid_gate_inplace, the whole kernel (this chapter’s primary source; the header dispatch comment at :5 is stale versus the wrapper — dispatch_1d uses dispatch_threads, encode.rs:1337-1342).
• crates/muser-engine/src/metal/encode/gate.rs:7-23 — encode_sigmoid_gate, the Rust dispatch (cross-vendor twin selected at :14-15).
• crates/muser-engine/src/shaders/muse_reference.metal:680-688 — muser_cross_vendor_sigmoid_gate, the strict no-fast-math sibling.
• crates/muser-engine/src/decode.rs:5793-5805 — the gate dispatch and o_proj call in encode_token; :5449-5451 the one-encoder contract; :2085-2091 the serving-versus-legacy exactness comment; :5112-5116 the packed decode group’s shared gate.
• crates/muser-engine/src/decode.rs:6044-6091 — encode_projection, the dtype router; crates/muser-engine/src/metal/encode/qkv.rs:429-450 the one-token pinned-metallib matvec path; :451-459 the muser_matvec_q4k_4r2s fallback.
• crates/muser-engine/src/metal/encode.rs:278-280 — ggml_q4k/q5k/q6k PSO registration against the pinned llama.cpp metallib; :1337-1342 dispatch_1d.
• crates/muser-engine/src/config.rs:266-273 — attn_dim/kv_dim (“the space the attention-output gate lives in”); :300-314 the per-layer tensor shape contract including attn_gate.weight [h, attn] and attn_output.weight [attn, h].
• crates/muser-engine/src/lib.rs:7-15 — the model-facts header asserting the sigmoid attention-output gate.
• crates/muser-engine/src/reference.rs:446-464 — the CPU oracle’s gate-then-o_proj order (the parity specification).
• crates/muser-bench/src/m16.rs:139-198 — the per-projection shape/dtype table (attn_q/gate 6656->4096 q4k, attn_v-q6k 6656->256, attn_output 4096->6656 q4k) backing §17.6’s budget.
• [docs/decode-dispatch-gap-20260815.md] — the closure-count reconciliation; this stage is common math (406 = 406).
• Ch 13 — the pinned ggml matvec family this chapter reuses; Ch 16 — what produced activations.attention.
• Ch 12 — the fused dual-eps tail that consumes this chapter’s projected output.
• Ch 6 — the Q4_K 144-byte super-block behind the 15.34 MB arithmetic.
• Ch 35 — the hazard taxonomy behind §17.7’s ordering argument.
• [ferrite-book Ch 16] — the ancestor’s residual-fused o_proj matvec; the device Muser deliberately does not port onto the pinned kernel (§17.7).

Chapter 18 — The SwiGLU feed-forward block

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 2 (SIMD groups, simd_sum, threadgroup memory), Ch 5 and Ch 6 (blocks, scales, the Q4_K super-block), Ch 12 (the dual-eps tail that produces this chapter’s input), Ch 13 (the matvec family and the pinned ggml kernels), Ch 17 (attention just closed into the residual stream). The FFN and its gated activation are taught from zero here.

18.1 What it computes

Attention has just folded its result back into the residual stream and handed the layer a vector. What does the layer do with it next? Half the layer’s work is still ahead, and it is the half that owns most of the weights on disk — so before any kernel, the question to settle is what shape that computation has and what it does to the vector it is given.

Each Muse Glimmer layer does two things to the residual stream: attention (Ch 13–Ch 17) mixes information across tokens; the feed-forward network (FFN) transforms the vector within a token — one position at a time, no cross-token mixing at all. Concretely it is two matvecs and a nonlinearity between them:

x : [6656]                        (hidden_dim — the ffn_norm'ed stream, Ch 12)
      │  gate projection   W_gate : [6656 → 19968]   (Q4_K)
g : [19968]                       (intermediate_dim)
      │  SiLU·Hadamard with the up branch
h : [19968]                       ("ffn_mid")
      │  down projection  W_down : [19968 → 6656]   (Ch 19)
out : [6656]

The vector is blown up 6,656 → 19,968 — a 3× wider “thinking space” (the checkpoint declares muse-glimmer.feed_forward_length, config.rs:180) — and squeezed back down in Ch 19. The widening is where most of the model lives: the two projections this chapter covers are 74.76 MB each of Q4_K per layer (§18.7), against 48.4 MB for the entire attention block (Ch 17 Figure 17.2).

The combination rule between the two branches is SwiGLU — Figure 18.1 shows its wiring:

h = silu(W_gate · x)  ⊙  (W_up · x)

Read the symbols: W_gate · x and W_up · x are two independent [6656 → 19968] matvecs over the same input x; silu(·) is the activation of §18.4 applied only to the gate branch; ⊙ is the element-wise (Hadamard) product, (a ⊙ b)[j] = a[j] × b[j], no mixing across the 19,968 coordinates.

18.2 Why it exists — the “thinking” half of the layer

Attention is the block’s mechanism for gathering context: it decides which earlier tokens this token looks at. The FFN is the mechanism for reasoning over what was gathered: a learned, position-independent transformation applied to the vector attention assembled. Every token at every layer walks the same FFN weights; what differs is the vector it walks in with. That is the “thinking” framing — and the size framing is starker: at ~30 B parameters, the FFN pair-plus-down is roughly 224–259 MB of the ~273–307 MB per-layer weight read (§18.7), so when Ch 1 said decode is ~99 % reading weights, this is where most of the weights are.

Why gated? A plain FFN (out = down(act(up(x))), the original transformer’s shape [arxiv:1706.03762]) applies its activation unconditionally per element — for feature j, “should this feature fire?” and “what does it carry?” are the same number. A gated FFN splits them into two learned projections: the gate branch answers should it fire, the up branch answers what does it carry. The silu turns the gate into a smooth on/off ramp. This structural claim is standard SwiGLU motivation [arxiv:2002.05202]; the quality delta for Muse Glimmer specifically is [unverified] here — this book does not retrain the model to A/B its own architecture. The cost side, however, is exact arithmetic: gating means two widening matrices instead of one, ~50 % more FFN parameters than the ungated shape — and it is why W_gate and W_up both stream past every token.

flowchart LR
    x["x [6656]<br/>ffn_norm'ed stream"]
    wg["W_gate<br/>[6656→19968] Q4_K"]
    wu["W_up<br/>[6656→19968] Q4_K"]
    g["g [19968]"]
    u["u [19968]"]
    sg["silu(g)"]
    had["⊙ Hadamard"]
    out["ffn_mid [19968]"]
    x --> wg --> g --> sg --> had
    x --> wu --> u --> had
    had --> out

Figure 18.1: The SwiGLU dataflow. The fusion this chapter covers collapses the two projections, the silu, and the Hadamard into one kernel; the down projection that consumes ffn_mid is Ch 19.

18.3 The matrix operation, by hand

Trusting a fused kernel gets easier once you have done its arithmetic by hand. So: what exactly is computed for a single output coordinate, and where is the redundancy that a fusion could take away?

For output element j (0 ≤ j < 19,968):

g_j = Σ_{i=0..6655} W_gate[j, i] · x[i]      one dot product over hidden_dim
u_j = Σ_{i=0..6655} W_up  [j, i] · x[i]      another, independent, same x
h_j = silu(g_j) · u_j                        the gated combination

Two dot products over the same x, then a pointwise combine. A toy worked example with hidden = 2, one output element, invented numbers:

x = [1.0, −2.0]
W_gate row = [0.5, 0.25]  →  g = 0.5·1 + 0.25·(−2) = 0.0
W_up   row = [2.0, −1.0]  →  u = 2.0·1 + (−1.0)·(−2) = 4.0
silu(0.0) = 0.0 · σ(0.0) = 0.0 · 0.5 = 0.0
h = 0.0 · 4.0 = 0.0          ← gate fully closed: nothing flows

Had the gate row been [1.0, 0.0] instead, g = 1.0, silu(1.0) ≈ 0.731, and h ≈ 0.731 · 4.0 ≈ 2.92 — the same up-value flows, scaled by how open the gate is. That decoupling is the whole idea.

The observation that motivates a fused kernel: both dot products read the same x[k]. In an unfused pair of matvecs, x is fetched twice and both 19,968-wide intermediates (g and u) make a round trip through memory. One kernel that loads x[k] once and updates two accumulators in lockstep eliminates both — Figure 18.2 shows the loop shape.

18.4 SiLU — the sigmoid linear unit

Two branches go in and one comes out, and the function on the gate branch is the only nonlinearity in the whole block. It decides how much of the up branch survives. So it is worth a section of its own: what is it, and how does it behave at the edges where it will actually be asked to decide?

The activation on the gate branch is SiLU (a.k.a. Swish, [arxiv:1710.05941]):

silu(x) = x · σ(x)  =  x / (1 + e^(−x))        (σ from Ch 17 §17.1)

Behavior at the extremes — like ReLU at the ends, unlike it in the middle:

Table 18.1: SiLU vs ReLU at five points (computed by hand from the formula). Large positive x passes through nearly unchanged; large negative x is suppressed — but small negative x lets the gate subtract a little, not merely go silent.

Two properties matter downstream. First, SiLU is smooth and its gradient is nonzero everywhere (ReLU has a dead zone for x < 0) — whether that is the reason gated FFNs ship SiLU is [unverified] here; we inherit the checkpoint’s choice. Second, it is cheap but not free: one exp, one add, one divide per element. At 19,968 elements per layer it runs 19,968 times — fully parallel, no reduction, and in every kernel this chapter quotes it is folded into the final write.

The CPU oracle states the whole combination in two loops (crates/muser-engine/src/reference.rs:511-513):

#![allow(unused)]
fn main() {
for (a, b) in ffn_a.iter_mut().zip(ffn_b.iter()) {
    *a = silu_fast(*a) * *b;
}
}

The helper it calls is silu_fast(x) = x / (1.0 + (−x).exp()) — this section’s formula written as a one-liner, inherited from Ferrite along with the rest of the shader lineage. We kept the provenance: crates/muser-engine/src/quant/helpers.rs:70-74, tracked through the extraction manifest [docs/extraction-manifest.md].

18.5 The Metal kernel — ffn_q4k_gate_up_silu_4r2s

There are two gate+up routes in the tree, and which one runs is the chapter’s real story (§18.6). The fused kernel first — it is the cleanest expression of the SwiGLU fusion, a wholesale port of Ferrite’s accepted 897a6256b kernel, and both the call site and the wrapper say so in their own comments (decode.rs:5823-5825, ffn.rs:7-8). Below: the signature, the x-load, and the lockstep MAC, verbatim; the Q4_K decode helper it calls is summarized after.

// crates/muser-engine/src/shaders/ferrite/ffn_fused_tail.metal:485
// ── ffn_q4k_gate_up_silu_4r2s ─────────────────────────────────────────────
//
// 4-row-per-TG fused gate+up variant using the V4 x-load pattern.
// 64 threads (2 SIMDs × 32), 4 output rows per TG (2 per SIMD).
// Each thread loads x into local registers yl[16]/yh[16] via stride-4
// block sub-groups (same as matvec_q4k_f32_v4), reusing x for both
// gate and up weight rows.
//
// Activation selected at PSO build time via FC_FFN_ACTIVATION.
// dispatch_thread_groups( (ceil(i_dim/4), 1, 1), (64, 1, 1) )
//
kernel void ffn_q4k_gate_up_silu_4r2s(
    device const uchar* W_gate [[ buffer(0) ]],
    device const uchar* W_up   [[ buffer(1) ]],
    device const float* x      [[ buffer(2) ]],
    device       float* out    [[ buffer(3) ]],
    constant     uint&  rows   [[ buffer(4) ]],
    constant     uint&  cols   [[ buffer(5) ]],
    uint tgid [[ threadgroup_position_in_grid ]],
    uint lid  [[ thread_index_in_simdgroup ]],
    uint sgid [[ simdgroup_index_in_threadgroup ]])
{
    const uint n_blocks    = cols / 256u;
    const uint block_bytes = 144u;
    const uint row_bytes   = n_blocks * block_bytes;

    // 2 rows per SIMD, 2 SIMDs = 4 rows per TG
    const uint base_row = tgid * 4u + sgid * 2u;
    if (base_row >= rows) return;

    // V4 thread partitioning: 4 sub-groups of 8 threads for block stride
    const uint ix = lid / 8u;   // 0..3 — block stride index
    const uint it = lid % 8u;   // 0..7 — position within block
    const uint iq = it / 4u;    // 0 or 1 — half-block selector
    const uint ir = it % 4u;    // 0..3 — quarter within half

    // x-vector pointer: each thread reads 8 positions per block (stride-4 blocks)
    device const float* xp = x + ix * 256u + 64u * iq + 8u * ir;

    float yl[16], yh[16];
    float gate_sumf[2] = {0.f, 0.f};
    float up_sumf[2]   = {0.f, 0.f};

    for (uint ib = ix; ib < n_blocks; ib += 4u) {
        // Load x slice into registers (8 elements × 4 positions)
        float4 sumy = {0.f, 0.f, 0.f, 0.f};
        for (uint i = 0u; i < 8u; i++) {
            yl[i]     = xp[i];       sumy[0] += yl[i];
            yl[i + 8] = xp[i + 32];  sumy[1] += yl[i + 8];
            yh[i]     = xp[i + 128]; sumy[2] += yh[i];
            yh[i + 8] = xp[i + 160]; sumy[3] += yh[i + 8];
        }

        // Gate weight: 2 rows starting at base_row + sgid*2
        device const uchar* gate_blk = W_gate + (ulong)base_row * (ulong)row_bytes
                                       + (ulong)ib * block_bytes;
        q4k_v4_dual_row_mac(gate_blk, row_bytes, yl, yh, sumy, iq, ir, gate_sumf);

        // Up weight: same 2 rows
        device const uchar* up_blk = W_up + (ulong)base_row * (ulong)row_bytes
                                     + (ulong)ib * block_bytes;
        q4k_v4_dual_row_mac(up_blk, row_bytes, yl, yh, sumy, iq, ir, up_sumf);

        xp += 4u * 256u;  // advance by 4 blocks (stride)
    }

    // Reduction across all 32 threads in each SIMD
    const float gr0 = simd_sum(gate_sumf[0]);
    const float gr1 = simd_sum(gate_sumf[1]);
    const float ur0 = simd_sum(up_sumf[0]);
    const float ur1 = simd_sum(up_sumf[1]);

    if (lid == 0u) {
                                    out[base_row]      = apply_activation(gr0, ur0);
        if (base_row + 1u < rows)   out[base_row + 1u] = apply_activation(gr1, ur1);
    }
}

Line by line:

• Bindings (496–505). Two weight buffers (W_gate slot 0, W_up slot 1), the input x (slot 2), the output out (slot 3), and rows/cols as inline constants (19,968 / 6,656 on this model).
• Row ownership (511–513). base_row = tgid·4 + sgid·2: one threadgroup of 64 threads = 2 SIMD groups, each SIMD group owns two whole output rows. Four rows per threadgroup; the grid of §18.6 covers 19,968 rows.
• The V4 lane decomposition (515–522). The 32 lanes split into 4 sub-groups of 8 (ix), each striding a different Q4_K super-block (ib += 4); within a sub-block, iq/ir select which of the eight 32-element runs this lane dots. This is the same x-in-registers pattern as the QKV matvec family of Ch 13 — the header says so (“same as matvec_q4k_f32_v4”).
• The register x-cache (524–536). Each thread pulls its 32 x-slice elements (yl[16], yh[16]) and their quarter-sums sumy into registers once per block iteration. The four sumy partial sums exist for the Q4_K min-term subtraction (w = d·sc·nib − dmin·m, Ch 6).
• The lockstep MAC (538–548). This is the fusion. The same register-held yl/yh/sumy feed two calls to q4k_v4_dual_row_mac — once against the gate rows, once against the up rows at the same base_row:

                       ┌─► gate_sumf[r] += dequant(W_gate[base_row+r, k]) · x[k]
  x[k] (registers)  ───┤
                       └─► up_sumf[r]   += dequant(W_up  [base_row+r, k]) · x[k]

Figure 18.2: The lockstep MAC. Every x element is loaded from device memory once (into yl/yh) and consumed by both the gate and the up accumulator for two rows each — one load, four dot-product contributions.

q4k_v4_dual_row_mac itself (shaders/ferrite/_q4k_helpers.metal:34-88) decodes two consecutive Q4_K rows’ block: it unpacks the 6-bit scale strip with the 0x3F3F/0x0F0F/0xC0C0 masks, accumulates nibble·x products into four float4 lanes per row, and folds them into the d·(Σ…) − dmin·(Σ…) super-block epilogue — the deferred-scaling schedule of Ch 13, here applied to two rows and two matrices at once.

• Reduction and activation (551–560). simd_sum collapses each accumulator across the 32 lanes; lane 0 of each SIMD group writes the two finished rows via apply_activation(g, u). There is no cross-SIMD combine and no threadgroup_barrier — whole-row ownership per SIMD group means each group’s simd_sum is the final answer for its rows. The ownership implies synchronization freedom, the same argument as Ch 17 §17.7.

apply_activation is compiled, not branched. The helper is selected at PSO build time through a Metal function constant:

// crates/muser-engine/src/shaders/ferrite/ffn_fused.metal:15
// 0 = SiLU (default), 1 = GELU
constant uint FC_FFN_ACTIVATION [[function_constant(32)]];
constant bool HAS_FC_FFN_ACTIVATION = is_function_constant_defined(FC_FFN_ACTIVATION);

// Activation helper — compiled away at pipeline creation time (zero runtime cost)
inline float apply_activation(float g, float up) {
    if (HAS_FC_FFN_ACTIVATION && FC_FFN_ACTIVATION == 1u) {
        // … (GELU branch elided) …
    } else {
        // SiLU (default): g * sigmoid(g) * up
        return (g / (1.0f + exp(-g))) * up;
    }
}

Build the PSO without slot 32 and the dead-code eliminator removes the GELU path entirely; the shipped Muse kernel contains only the SiLU line — silu(g) · u, the chapter’s formula, as the kernel’s last instruction (ffn_fused.metal:14-30, the Ch 4 function-constant mechanism).

18.6 The Rust dispatch — and the opt-in flag story

Two questions decide what this section owes you. How does the host launch that kernel? And — since the tree carries both routes — which one actually runs when the engine serves a token? The first has a short answer. The second is this chapter’s war story.

The wrapper first:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/metal/encode/ffn.rs:7
/// Ferrite 897a6256b Q4_K SiLU gate+up route: four rows per threadgroup,
/// two SIMD groups, with the input vector shared by both projections.
pub fn encode_ffn_q4k_gate_up_silu_4r2s(
    &self,
    encoder: &ComputeCommandEncoderRef,
    gate_weights: GpuByteView<'_>,
    up_weights: GpuByteView<'_>,
    input: &GpuBuffer,
    output: &GpuBuffer,
    intermediate_dim: usize,
    hidden_dim: usize,
) {
    let row_bytes = hidden_dim / 256 * 144;
    debug_assert_eq!(gate_weights.len(), intermediate_dim * row_bytes);
    // … (up/input/output length asserts elided) …
    self.bind(encoder, "ffn_q4k_gate_up_silu_4r2s");
    encoder.set_buffer(0, Some(gate_weights.metal()), gate_weights.offset() as u64);
    encoder.set_buffer(1, Some(up_weights.metal()), up_weights.offset() as u64);
    encoder.set_buffer(2, Some(input.metal()), 0);
    encoder.set_buffer(3, Some(output.metal()), 0);
    set_value(encoder, 4, &(intermediate_dim as u32));
    set_value(encoder, 5, &(hidden_dim as u32));
    encoder.dispatch_thread_groups(
        MTLSize::new(intermediate_dim.div_ceil(4) as u64, 1, 1),
        MTLSize::new(64, 1, 1),
    );
}
}

In prose: grid (19,968 ÷ 4, 1, 1) = (4,992, 1, 1) threadgroups of (64, 1, 1) threads — 4,992 × 4 = 19,968 rows covered exactly. The two weight views are bound at their offsets into the mmap’d GGUF (Ch 3); rows/cols ride as inline constants.

Now the fork. We had a kernel that was better on paper in every dimension we knew how to count — fewer dispatches, fewer bytes moved, the same algebra — and it arrived as a port of something Ferrite had already accepted and run. The obvious move was to make it the FFN path and move on. We expected to do exactly that. Instead: the fused kernel is opt-in, and the default is the unfused control. The gate at the call site shows both branches side by side, which is why it is worth reading whole:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5819
if self.ferrite_ffn_gate_up
    && layer.ffn_gate.layout.dtype == GgmlType::Q4_K
    && layer.ffn_up.layout.dtype == GgmlType::Q4_K
{
    // Port Ferrite 897a6256b wholesale: the four-row/two-SIMD-
    // group kernel reads the normalized input once for both Q4_K
    // projections and writes the final SiLU(gate) * up row.
    dispatch(command, |encoder| {
        self.kernels.encode_ffn_q4k_gate_up_silu_4r2s(
            encoder,
            layer.ffn_gate.view(&self.mapped_weights),
            layer.ffn_up.view(&self.mapped_weights),
            &self.activations.post_norm,
            &self.activations.ffn_gate,
            cfg.intermediate_dim,
            cfg.hidden_dim,
        );
    });
} else {
    // Exact upstream-matvec control and non-Q4_K fallback.
    dispatch(command, |encoder| {
        self.encode_projection(
            encoder,
            &layer.ffn_gate,
            &self.activations.post_norm,
            &self.activations.ffn_gate,
            1,
        );
        self.encode_projection(
            encoder,
            &layer.ffn_up,
            &self.activations.post_norm,
            &self.activations.ffn_up,
            1,
        );
    });
    dispatch(command, |encoder| {
        self.kernels.encode_silu_mul(
            encoder,
            &self.activations.ffn_gate,
            &self.activations.ffn_up,
        );
    });
}
}

ferrite_ffn_gate_up is set from the environment — MUSER_FERRITE_FFN_GATE_UP (decode.rs:1334) — so by default the engine takes the else branch: two pinned ggml matvecs (W_gate, W_up, the exact kernel_mul_mv_q4_K_f32 of Ch 13) plus one pointwise activation kernel. That third kernel is muser_silu_mul_inplace, which does gate[i] = silu(gate[i]) · up[i] — the same formula as the fused tail, materialized:

// crates/muser-engine/src/shaders/muse_reference.metal:4
kernel void muser_silu_mul_inplace(
    device float *gate [[buffer(0)]],
    device const float *up [[buffer(1)]],
    constant uint &count [[buffer(2)]],
    uint index [[thread_position_in_grid]]) {
    if (index < count) {
        float value = gate[index];
        gate[index] = (value / (1.0f + exp(-value))) * up[index];
    }
}

dispatched one-thread-per-element over 19,968 (ffn.rs:38-60).

So why is the better kernel the one that does not run? Because when the fused tail was put on the serving route, the numbers coming out of the model stopped matching the numbers the pinned comparator produced. Not wrong — different, and different by more than the public tolerance allows. That result is written down where it belongs, in the routing comment that governs serving decode:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:2085
// The legacy one-token graph uses Ferrite fused residual/norm and
// gate-up kernels whose rounding diverges from the source-pinned
// llama Metal graph enough to breach public logprob tolerance.
// The one-row batch graph dispatches the exact pinned kernels and
// has the same KV transition, so it is the serving correctness
// path until each fused kernel independently passes full-logit
// parity.
}

That is the contract discipline in one comment. The fused kernel’s reduction order — two interleaved accumulators folded per super-block — is mathematically SwiGLU, but it is not bit- the same as llama.cpp’s graph of independent mul_mv nodes plus a pointwise silu-mul, and Muser’s public commitment is logprob parity with the pinned comparator.

Say it the other way round, because this is the sentence the rest of the book keeps coming back to. Floating-point addition is not associative, so changing when you add changes what you get; a contract written against a specific graph is therefore a contract against a specific order of additions, not against the algebra those additions approximate. A kernel that computes the same function by a different schedule is, by that measure, a different kernel — and the measure is the one the public promise is written in.

The lesson we kept is the uncomfortable one: fusion is never free, even when it is free, because what it spends is exactness, and here exactness is the product. The fused kernel therefore lives on as a qualified-off fast path, runnable under the flag for the teacher-forced/diagnostic route (encode_token) and never on the serving default. The dtype guards in the if are a smaller version of the same care — any non-Q4_K FFN tensor falls back automatically, and on the release artifact ffn_gate/ffn_up are Q4_K. We kept that evidence too: ffn_gate/up 6656->19968 q4k, crates/muser-bench/src/m16.rs:171-175.

18.7 The access pattern — the largest weight read in the layer

Where does the time go in this block? Not into the silu, and not into the Hadamard: both are pointwise, and both vanish into the write. It goes into dragging two very large matrices past the arithmetic units, once for every token the model emits. It is worth knowing exactly how large, and worth knowing what the fusion actually buys against that background — because the answer decides whether the flag we just described gave up something expensive or something cheap.

All arithmetic derived from the verified shapes, shown step by step:

  W_gate: 19,968 rows × (6,656 / 256 = 26 blocks) × 144 B = 19,968 × 3,744
        = 74,760,192 B ≈ 74.76 MB        (Q4_K: 0.5625 B/element)
  W_up  : same shape, same dtype          = 74,760,192 B ≈ 74.76 MB
                                          ────────────────────────
  gate + up pair per layer                ≈ 149.5 MB   ← read once per token
  (for scale: the whole attention block   ≈  48.4 MB;  o_proj alone 15.34 MB,
   Ch 17 Figure 17.2; the down projection is Ch 19: 74.76 MB Q4_K /
   109.03 MB Q6_K)

Activation traffic, fused vs unfused (per layer):

  FUSED (one kernel):
    read  x [6656] f32                    26,624 B  (registers thereafter)
    read  W_gate + W_up                 149,520,384 B
    write ffn_mid [19968] f32             79,872 B

  UNFUSED (two matvecs + silu_mul — the default):
    read  x twice                          53,248 B
    read  W_gate + W_up                  149,520,384 B
    write g [19968], u [19968]            159,744 B   ← intermediates born
    read  g + u                           159,744 B   ← …and read back
    write ffn_mid                          79,872 B
                                          ──────────
    extra activation traffic vs fused:    ≈ 345,856 B ≈ 338 KiB/layer

Hold the two ledgers side by side and the same lesson falls out that the ancestor’s FFN chapter [ferrite-book Ch 17] drew, re-derived here for this geometry. Reading x once is mostly a cache effect — x is 26 KiB and would have been sitting in cache for the second matvec anyway. The hard saving is the one that never shows up as a load at all: in the unfused route the two 19,968-wide intermediates are born, written, and read back; in the fused route they simply never exist.

Then scale it up before deciding how much to care. 338 KiB/layer × 52 layers ≈ 18.0 MB/token of avoided activation traffic — about 12 % of one layer’s FFN weight read. Real, then, but second-order; and on the serving route it is deliberately not taken (§18.6). What is neither second-order nor optional is the weight stream underneath it: 149.5 MB per layer, 7.78 GB across 52 layers, irreducible at Q4_K bitrate no matter which route dispatches it. That is the number Ch 1’s bandwidth argument leans on.

18.8 Tradeoffs

Fused 4r2s vs the unfused control — bytes versus bits. The fusion saves ~338 KiB/layer of activation round-trips and one dispatch: three closures become one. The control gives up both of those and buys back llama.cpp’s exact per-node arithmetic, and with it the public logprob contract. Muser ships the control as the default and gates the fusion behind MUSER_FERRITE_FFN_GATE_UP, and the reason lives in the source, where the fused kernels’ rounding divergence “breache[s] public logprob tolerance” against the source-pinned llama Metal graph (decode.rs:5819-5836 and :1334 for the flag, decode.rs:2085-2091 for the reason).

That verdict was not invented here. The dispatch-gap investigation had already made the discipline explicit for a whole family of these fusions: they were “removed, not hidden behind a tolerance” [docs/decode-dispatch-gap-20260815.md, Rejected hybrid postmortem]. A flag is the gentlest form of that same judgement — the code survives, the default does not move. What we still cannot tell you is what the flag would be worth end to end: no retained A/B quotes a tok/s delta for this specific flag [unverified]. The burden the fused route has to clear is full-logit parity first, per the comment, and at the pin that gate is not recorded as passed.

Two accumulators in lockstep vs two kernels. The 4r2s design doubles down on the V4 pattern of Ch 13: the register x-cache is shared across two matrices and two rows simultaneously — one x load feeds gate_sumf[0..1] and up_sumf[0..1]. The cost is register pressure (yl[16], yh[16], sumy, four running accumulators) and a kernel that only exists for Q4_K (the q4k_v4_… helper is Q4_K-specific; Q5_K/Q6_K tensors would need their own decoders — hence the dtype guard at decode.rs:5820-5821). The payoff is the traffic ledger of §18.7.

The design did not arrive in that shape, either. An older 4-SIMD-group variant — ffn_q4k_gate_up_silu_4sg, one row per threadgroup, 128 threads — sits in the same shader file at ffn_fused_tail.metal:295, with a threadgroup-x-cache sibling at :392. Both are the same idea at a coarser grain of x-reuse, and both were kept rather than deleted; the 4r2s port at :496 is the one Muser wired.

SiLU vs GELU vs ReLU. The choice is the checkpoint’s, not the engine’s; the function-constant mechanism (§18.5) exists so one .metal source could serve either at zero runtime cost. Muse Glimmer is SiLU — every route (fused apply_activation, control muser_silu_mul_inplace, CPU oracle silu_fast) implements x·σ(x).

The normed-quant tail variants — present but unwired. The shader library carries a family that goes one step further than 4r2s: fuse the norm into the gate+up read, so that the FFN would consume the raw residual and normalize in-kernel. The kernel is ffn_q4k_gate_up_silu_normed, at shaders/ferrite/ffn_fused_normed_quant.metal:296, with Q5_K siblings at :1 and :190. We went looking for whoever calls it, and found nobody: at the pinned commit no Rust wrapper binds any kernel from that file (verified: no reference in crates/muser-engine/src). It is not on a live path — retained as Ferrite-lineage research material.

The reason that belongs in a tradeoffs section rather than a footnote is that it names the ceiling of the whole fusion strategy. Its fate is the story of §18.6 taken one step further: the more arithmetic you fold across a norm boundary, the harder bit-exactness becomes (Ch 19 §19.9 makes that tradeoff precise).

18.9 Where the gap lives

The gate+up stage is not the gap — but its fusion is one of the casualties of the exactness contract. In the one-token closure accounting, the FFN gate-up and swiglu closures are “common math” — identical counts in the production and legacy graphs (the 406 = 406 row of [docs/decode-dispatch-gap-20260815.md]); the +196-closure gap lives in norm boundaries, SWA staging, KV publication, and one copy, not here. The FFN’s connection to that story is the reverse direction: the fused kernel this chapter teaches is part of the legacy route’s fusion set, and the serving graph pays extra closures (two matvecs + silu_mul instead of one) precisely to keep llama.cpp’s node-for-node arithmetic (decode.rs:2085-2091). When Ch 35 and Ch 40 audit what was measured and rejected, this is a standing example: a structurally sound bandwidth win, held out of serving by a logprob tolerance, exactly as the campaign’s fail-closed culture requires.

The FFN is half closed: ffn_mid [19968] holds the gated, activated intermediate. One projection remains — the squeeze back to 6,656, the residual add, and the fused tail that hands the next layer its normed input. That tail is also where this book’s central tradeoff — dispatch count versus the logprob contract — gets priced to the last ULP. It is Ch 19.

References

• crates/muser-engine/src/shaders/ferrite/ffn_fused_tail.metal:485-561 — ffn_q4k_gate_up_silu_4r2s, the fused Q4_K SwiGLU kernel (primary source; the _4sg ancestor variant at :295, _4sg_tgcache at :392).
• crates/muser-engine/src/shaders/ferrite/ffn_fused.metal:14-30 — FC_FFN_ACTIVATION function constant + apply_activation (PSO-build-time SiLU/GELU selection).
• crates/muser-engine/src/shaders/ferrite/_q4k_helpers.metal:34-88 — q4k_v4_dual_row_mac, the dual-row Q4_K MAC both projections share.
• crates/muser-engine/src/metal/encode/ffn.rs:7-36 — encode_ffn_q4k_gate_up_silu_4r2s (grid (i_dim/4, 1, 1) × 64 threads); :38-60 encode_silu_mul (control-route activation).
• crates/muser-engine/src/shaders/muse_reference.metal:4-13 — muser_silu_mul_inplace, the pointwise control kernel.
• crates/muser-engine/src/decode.rs:5819-5862 — the flag/dtype gate, the fused dispatch, and the unfused control; :1334 the MUSER_FERRITE_FFN_GATE_UP env read; :2085-2091 the serving-exactness comment; :5165-5185 the packed decode group’s unfused FFN.
• crates/muser-engine/src/reference.rs:493-516 — the CPU oracle’s gate/up/silu·mul order; quant/helpers.rs:70-74 silu_fast.
• crates/muser-engine/src/config.rs:180 — intermediate_dim from muse-glimmer.feed_forward_length.
• crates/muser-bench/src/m16.rs:171-175 — ffn_gate/up 6656->19968 q4k (release-artifact dtype evidence).
• crates/muser-engine/src/shaders/ferrite/ffn_fused_normed_quant.metal:1,190,296 — the normed-input fused FFN family; no Rust binder at the pin (§18.8).
• [docs/decode-dispatch-gap-20260815.md] — closure accounting (common math) and the rejected-hybrid postmortem’s remove-don’t-tolerate discipline.
• [docs/extraction-manifest.md] — silu_fast and the shader lineage from Ferrite 83cfd55…/a85048a9….
• Ch 13 — the V4 lane decomposition and the pinned ggml matvec the control route dispatches.
• Ch 6 — the Q4_K super-block and dequant formula behind the MAC helper.
• Ch 1 — the per-token weight-read arithmetic this block dominates.
• [ferrite-book Ch 17] — the ancestor’s SwiGLU chapter; the fused-vs-control honesty pattern and the intermediate-buffer byte ledger ported here.
• [arxiv:1706.03762] — Vaswani et al., Attention Is All You Need (the ungated ReLU FFN).
• [arxiv:1710.05941] — Ramachandran et al., Searching for Activation Functions (Swish/SiLU).
• [arxiv:2002.05202] — Shazeer, GLU Variants Improve Transformer (SwiGLU).

Chapter 19 — The down projection + residual

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Ch 6 (Q4_K and Q6_K block layouts), Ch 12 (the dual-eps sandwich and muser_fused_norm_residual_rms_norm_32sg), Ch 13 (the pinned ggml matvec family), Ch 17–Ch 18 (the layer so far; ffn_mid is waiting). This chapter closes the layer — and then prices what closing it costs in dispatch groups, which is where this book’s central tradeoff becomes concrete.

19.1 What it computes

Ch 18 left a [19,968] vector in activations.ffn_gate — the gated, activated FFN intermediate. It is the widest thing a layer ever holds, and it is the wrong shape to hand back to the loop. So the closing question of every layer is a plumbing question: how does that wide intermediate get folded back into the narrow residual stream, and who prepares the stream for the layer that comes next? Two operations answer it:

1.  projected = W_down · ffn_mid                W_down : [6656 × 19968]
2.  residual += post_norm(projected)            (eps 1e-8)
    next_input = rms_norm(residual, next_norm)  (eps 1e-5)

Operation 1 is a matvec — the last one of the layer, and on some layers the most expensive single weight read in it (§19.7). Operation 2 is the second fused dual-eps tail: the sandwich of Ch 12 paying off, where the same kernel that adds the FFN delta into the residual also computes the next layer’s normalized input (or, after layer 51, the final norm that feeds the LM head of Ch 20).

After operation 2, activations.normed holds the residual stream plus layer l’s attention delta and FFN delta, normalized for layer l+1 — and the 52-layer loop takes its next turn.

19.2 Why it exists — closing the block and opening the next

The down projection is the FFN’s exit: without it the layer would emit a 19,968-wide vector into a 6,656-wide stream — wrong shape, and the next layer’s attention would have nothing to read. The residual add is what makes deep transformers deep: each layer contributes into a running sum rather than replacing it, so the gradient path to early layers stays near-identity (∂(x + f(x))/∂x = 1 + f'(x)). And the tail’s second norm exists because Muse Glimmer sandwiches every sub-block between a pre-norm and a post-norm (Ch 12) — the post-FFN norm (1e-8) scales the delta before it lands in the residual, and the next pre-norm (1e-5) prepares the stream for the next attention. Figure 19.1 lays out the whole layer with both tails.

  ┌────────────── layer l ────────────────────────────────────────────┐
  │  [Ch 17] attention ─► gate ─► o_proj ─► TAIL#1: residual +=       │
  │                                              post_norm(o) (1e-8); │
  │                                              ffn_in = norm (1e-5) │
  │  [Ch 18] ffn_in ─► gate·x, up·x ─► silu⊙ ─► ffn_mid [19968]       │
  │  [Ch 19] ffn_mid ─► W_down ─► TAIL#2: residual +=                 │
  │                                    post_ffn_norm(·) (1e-8);       │
  │                                    next_in = norm(residual, 1e-5) │
  └───────────────────────────────────────────────────────────────────┘
                                    │ after layer 51: next_in feeds
                                    ▼ the final norm → LM head (Ch 20)

Figure 19.1: One layer, two tails. This chapter is the W_down matvec and TAIL#2.

19.3 The matrix operation — and the Q6_K wrinkle that is real here

Start with the question we actually wanted answered here: how many bytes does this matvec read? The mechanism is the family you already know — 6,656 output rows, each a dot product over 19,968 inputs — so we expected to multiply one row size by one row count and be done in a paragraph. The artifact would not give a single answer. What distinguishes this projection is its dtype mix: it is the one weight in the layer that is not stored in a single format.

On the kquant release artifact, ffn_down tensors come in both Q4_K and Q6_K — the verify-shape table lists ffn_down-q4k 19968->6656 q4k and ffn_down-q6k 19968->6656 q6k side by side (crates/muser-bench/src/m16.rs:179-194), and the quickstart warns that a build without the pinned llama.cpp metallib “fails closed … because Q6_K tensors route through” it (docs/quickstart.md:16). The per-layer split — which of the 52 layers carry which variant — lives in the GGUF tensor headers and is not recorded in the repo docs this book cites [unverified]; what is verifiable is that both variants exist on live paths and both dispatch through the pinned metallib.

The two formats in bytes, step by step (layouts from Ch 6):

  W_down row = 19,968 inputs = 19,968/256 = 78 super-blocks
  Q4_K: 78 × 144 B = 11,232 B/row  →  6,656 × 11,232  =  74,760,192 B ≈  74.76 MB
  Q6_K: 78 × 210 B = 16,380 B/row  →  6,656 × 16,380  = 109,025,280 B ≈ 109.03 MB
                                        ─────────────────────────────────────
  Q6_K / Q4_K = 210/144 = 1.4583      →  +45.8 % bytes for the 6-bit format

So a Q6_K-down layer reads ~258.6 MB of FFN weights (gate 74.76 + up 74.76 + down 109.03) against a Q4_K-down layer’s 224.3 MB. The ancestor book’s Q4_K_M mix table [ferrite-book Ch 18] taught exactly this device — spend extra bits on the projection whose output lands directly in the residual stream — and Muse Glimmer’s artifact realizes the same idea with its own (per-layer, GGUF-internal) split. The intuition is worth saying twice, because it is the entire reason a mixed-format checkpoint exists at all: an error made anywhere else in the layer is consumed and largely forgotten inside that layer, but an error made in ffn_down is added into the residual stream, and every remaining layer carries it forward. Bits spent on this projection buy quiet downstream. Whether the quality payoff justifies the +45.8 % on the layers that take it is the checkpoint author’s call, inherited not measured [unverified].

The tail is this chapter’s kernel, so operation 2, the layer’s real exit, deserves a worked example small enough to check by hand. Watch what the two epsilons actually do to a vector: the first normalization shrinks the delta before it lands, the second renormalizes the sum after it has landed. Take n = 4, residual = [1, 2, 3, 4], projected = [4, 0, 0, 0], post_weight = [1,1,1,1], next_weight = [1,1,1,1], both eps tiny:

post_norm(projected, 1e-8):  rms = √(16/4) = 2  →  [2, 0, 0, 0]
residual +=                 →  [3, 2, 3, 4]
next_norm(residual, 1e-5):  rms = √((9+4+9+16)/4) = √9.5 ≈ 3.082
next_input                  →  [0.973, 0.649, 0.973, 1.297]

One kernel, two normalizations, one add — with a device-memory publication between them that turns out to be load-bearing (§19.9).

19.4 The Metal kernel — muser_fused_norm_residual_rms_norm_32sg

What does one kernel have to do to be simultaneously a layer’s exit and the next layer’s entrance? It has to normalize with one epsilon, add, and normalize again with a different epsilon — and it has to do all of that without ever letting the two reductions see each other’s rounding. Here is the tail kernel, verbatim. It is the decode-only member of the sandwich family Ch 12 introduced; read it here as the layer-exit machine:

// crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:142
// Decode-only dual-epsilon tail fusion:
//   hidden += rms_norm(src, eps1) * weight1
//   output  = rms_norm(hidden, eps2) * weight2
// Muse uses eps1=1e-8 for sandwich post-norms and eps2=1e-5 for the
// following pre-norm, so the older single-epsilon batch kernel is not valid.
kernel void muser_fused_norm_residual_rms_norm_32sg(
    device float* hidden [[buffer(0)]],
    device const float* src [[buffer(1)]],
    device float* output [[buffer(2)]],
    device const float* weight1 [[buffer(3)]],
    device const float* weight2 [[buffer(4)]],
    constant uint& n [[buffer(5)]],
    constant float& eps1 [[buffer(6)]],
    constant float& eps2 [[buffer(7)]],
    uint row [[threadgroup_position_in_grid]],
    uint tid [[thread_index_in_threadgroup]],
    uint sgitg [[simdgroup_index_in_threadgroup]],
    uint lid [[thread_index_in_simdgroup]],
    threadgroup float* shared [[threadgroup(0)]]) {
    const uint n4 = n >> 2u;
    device float4* hidden4 = (device float4*)(hidden + row * n);
    device const float4* src4 = (device const float4*)(src + row * n);
    device float4* output4 = (device float4*)(output + row * n);
    device const float4* weight14 = (device const float4*)weight1;
    device const float4* weight24 = (device const float4*)weight2;

    float sum_src = 0.0f;
    for (uint i = tid; i < n4; i += 1024u)
        sum_src += dot(src4[i], src4[i]);
    sum_src = simd_sum(sum_src);
    if (lid == 0u) shared[sgitg] = sum_src;
    threadgroup_barrier(mem_flags::mem_threadgroup);
    if (tid == 0u) {
        float total = 0.0f;
        for (uint group = 0u; group < 32u; ++group) total += shared[group];
        shared[32] = rsqrt(total / float(n) + eps1);
    }
    threadgroup_barrier(mem_flags::mem_threadgroup);

    const float inv_src = shared[32];
    float sum_hidden = 0.0f;
    for (uint i = tid; i < n4; i += 1024u) {
        const float4 value = hidden4[i] + src4[i] * inv_src * weight14[i];
        hidden4[i] = value;
        sum_hidden += dot(value, value);
    }
    sum_hidden = simd_sum(sum_hidden);
    if (lid == 0u) shared[sgitg] = sum_hidden;
    threadgroup_barrier(mem_flags::mem_threadgroup);
    if (tid == 0u) {
        float total = 0.0f;
        for (uint group = 0u; group < 32u; ++group) total += shared[group];
        shared[32] = rsqrt(total / float(n) + eps2);
    }
    threadgroup_barrier(mem_flags::mem_threadgroup);

    const float inv_hidden = shared[32];
    for (uint i = tid; i < n4; i += 1024u)
        output4[i] = hidden4[i] * inv_hidden * weight24[i];
}

Three passes, two barriers apiece:

• Pass 1 (sum of squares of src). Every thread strides the row’s float4s (i += 1024); each of the 32 SIMD groups reduces with simd_sum, writes one partial to shared[sgitg]; thread 0 sums the 32 partials in order and posts rsqrt(mean + eps1) to shared[32]. This 32-partial, fixed-order reduction is the pinned llama.cpp shape.
• Pass 2 (the residual add — the in-place mutation). Each thread computes value = hidden + src·inv_src·weight1, writes it back into hidden, and accumulates dot(value, value) for the second norm. This is where hidden += … physically happens: the same buffer is read, added into, and rewritten, float4 by disjoint float4.
• Pass 3 (the next norm). With inv_hidden from the second reduction, output[i] = hidden[i] · inv_hidden · weight2. weight1 is the layer’s post_ffw_norm; weight2 is the next layer’s attn_norm (or the final output_norm after layer 51) — chosen by the caller (§19.5).

That “in order” in the first pass looks like pedantry, and it is the part of the kernel we got wrong first. The fork was how to fold a threadgroup’s partial sums: an earlier version of the tail reduced across four SIMD groups rather than the full complement, gathering fewer partials in whatever order they arrived. Same summands, same mean, same rsqrt — we expected the same bits out. We did not get them. The dispatch-gap investigation found that earlier four-SIMD-group rsqrt variant “not exact” and replaced it with precisely the shape quoted above [docs/decode-dispatch-gap-20260815.md, “The corrected fusion”]. The lesson is one this chapter keeps circling back to: when a kernel’s job is to match another engine bit for bit, the order of a floating-point reduction is part of the interface, not an implementation detail. Add the same numbers in a different sequence and you get a different number.

Safety of the in-place add, in full: one threadgroup owns one row (grid = rows), and within the threadgroup each thread owns a disjoint strided set of float4 slots — no two threads touch the same i in any pass, so no in-dispatch race on hidden. The barriers order the shared rsqrt handoffs, not the data writes (each thread’s pass-3 reads the same slots it wrote in pass 2). Put it plainly: the kernel is allowed to scribble on the residual stream in place because, for the duration of the dispatch, nothing else is looking at it — no other thread, no other threadgroup. Across dispatches, the consumer of hidden/ output is the next layer’s first kernel, sequenced by the tracked-buffer ordering of the single-encoder token graph that Ch 17 §17.7 introduced; Ch 35 formalizes that taxonomy.

One geometry note from the wrapper, because it explains the kernel’s odd proportions: 32 SIMD groups keep “the 6,656-wide Muse tail resident” — 1,024 threads, one row per threadgroup (crates/muser-engine/src/metal/encode/norm.rs:236-240), with 33 floats of threadgroup memory padded to 144 bytes for alignment.

19.5 The Rust dispatch — the layer exit in source

Two dispatches close the layer, and only one of them is interesting. The down projection is the stock project wrapper — the same pinned ggml matvec the rest of the layer uses, with the dtype routed automatically. Q4_K or Q6_K ffn_down both land on kernel_mul_mv_q{4,6}_K_f32, and the rows-per-group table that picks the launch geometry — the (144, 2)/(210, 2) rows Ch 13 walked through — is read out of qkv.rs:429-450. The mixed dtype we made so much of above costs the dispatch code nothing at all; it is a table lookup.

The interesting dispatch is what follows — the tail and its next_norm selection, which is where the layer decides who it is handing off to:

#![allow(unused)]
fn main() {
// crates/muser-engine/src/decode.rs:5863
self.project(
    command,
    &layer.ffn_down,
    &self.activations.ffn_gate,
    &self.activations.projected,
);
let (next_norm, next_output) = if layer_index + 1 < cfg.n_layers {
    (
        &self.layers[layer_index + 1].attn_norm,
        &self.activations.post_norm,
    )
} else {
    (&self.output_norm, &self.activations.hidden)
};
dispatch(command, |encoder| {
    self.kernels.encode_fused_norm_residual_rms_norm_32sg(
        encoder,
        &self.activations.normed,
        &self.activations.projected,
        next_output,
        &layer.post_ffn_norm,
        next_norm,
        cfg.hidden_dim,
        cfg.post_norm_eps,
        cfg.rms_eps,
    );
});
}

Read the buffer wiring carefully — it is the residual-stream bookkeeping of the whole graph in one call:

• hidden ← activations.normed: the running residual, updated in place.
• src ← activations.projected: the down-proj output (scratch, like o_proj’s output in Ch 17).
• next_output ← activations.post_norm for layers 0..50 (the next layer’s normalized input), but activations.hidden for layer 51 — after the last layer the tail’s second norm is the final norm, writing the vector the LM head consumes (Ch 20 picks it up there). That is what “fused into last tail (decode.rs:5875)” means: there is no separate final-norm dispatch on this route.
• weight1 ← layer.post_ffn_norm (1e-8), weight2 ← the next layer’s attn_norm (1e-5) — the sandwich hand-off.

The wrapper itself (norm.rs:163-241) binds the six buffers, pushes n, eps1, eps2 inline, sets 144 bytes of threadgroup memory, and dispatches (rows, 1, 1) × (1024, 1, 1) — one row per threadgroup, 32 SIMD groups. The first tail of the layer (post-attention, decode.rs:5806-5818) is the identical call with post_attn_norm/ffn_norm — same kernel, both boundaries.

19.6 The access pattern

Where does the time go in this pair of dispatches? The answer is lopsided enough to be worth stating before the arithmetic: one of the two operations moves essentially all of the bytes, and the other is free. Down projection per layer:

  Q4_K: read W_down 74,760,192 B   read ffn_mid 79,872 B   write projected 26,624 B
  Q6_K: read W_down 109,025,280 B  (same activation traffic)
  weight : activation ratio ≈ 74.8 MB : 106 KiB ≈ 700:1  — pure weight stream

Tail per layer:

  read  src (projected)      26,624 B
  read+write hidden (normed) 26,624 B read + 26,624 B written (+ reread in pass 3)
  write output (post_norm)   26,624 B
  read  two weight vectors   2 × 6,656 f32 = 53,248 B
                          ≈ 181 KiB total — rounding error next to W_down

The layer’s total weight read: 224.3 MB (Q4_K-down) or 258.6 MB (Q6_K-down), of which the down projection is a third to 42 %. Across 52 layers the FFN family is the plurality of the 16.76 GB artifact — the arithmetic of Ch 18 §18.7 plus this chapter’s down numbers.

Turn that around and it says something uncomfortable about the shape of this chapter. The tail kernel — three passes, two reductions, an in-place mutation, the longest section here — moves less traffic than a rounding error on the matvec that precedes it. Expensive and interesting are not the same property. The matvec is where the bytes are; the tail is where the boundary is, and boundaries are what the rest of the chapter is about.

19.7 Tradeoffs

Q6_K vs Q4_K on the down projection — +45.8 % bytes on the layers that take it. The arithmetic of §19.3: 109.03 MB vs 74.76 MB per layer, a deliberate precision spend on the last projection before the residual — the same reasoning the ancestor’s Q4_K_M mix table documented for Qwen [ferrite-book Ch 18], realized differently here. Both engines pay it equally (both read the same GGUF through equivalent pinned kernels), so it is a quality-vs-bytes decision, not a parity hazard.

What we did not know going in was whether it was also a speed hazard. A wider quant means a fatter block to decode — more shifts, more scale unpacking per weight — so we expected the six-bit path to be slower per dispatch by something more than its byte ratio, and we went looking for that penalty. The M16 verify-shape bench measured the two side by side in its candidate sweep, and the penalty was not there: Q4_K ffn_down 0.891 → 0.533 ms and Q6_K ffn_down 0.897 → 0.539 ms per dispatch under the winning n32 tile. The ledger keeps that sweep [ledger Stage B close-out, L0 “Winner (n32)”]. Read the two rows next to each other and the conclusion is hard to miss: the formats cost nearly the same time per byte on this GPU, so the +45.8 % is a byte bill, not a kernel-efficiency bill. That is a comfortable result — it means the checkpoint author’s precision choice can be argued about purely in terms of bandwidth, with no hidden decode tax to price in.

One fused tail vs three separate kernels. Unfused, the layer exit is: norm the projection (1e-8), add into the residual, norm the residual (1e-5) — three dispatches and a materialized post-norm intermediate. The 32sg tail is one dispatch that never materializes the intermediate. But note precisely what it preserves: the source comment at decode.rs:1328-1330 says the fused kernel “reproduces the two pinned ggml f32x4 norm reductions and their intervening f32 device-memory boundary” — pass 2 writes hidden to device memory and pass 3 reads it back, exactly where llama.cpp’s graph publishes between nodes. It is a fusion of dispatches, not of arithmetic boundaries; that restraint is why it can be exact at all, and the diagnostic split route survives behind MUSER_NO_FUSED_PREFILL_DUAL_NORM (decode.rs:1331) as the control. State it as the rule the next section tests to destruction: a fusion may delete dispatches, but it may not delete rounding points. Wherever the reference graph writes a float to memory and reads it back, a value gets truncated to storage precision — and that truncation is part of the answer, not an artifact standing in front of it.

The 104-group question — fusion rejected where it would matter most. This is the tradeoff this chapter exists for; §19.8 gives it its own section with the numbers.

19.8 Where the gap lives — the 104 norm-boundary groups

Every kernel chapter in this part ends by asking where its kernel sits in the decode gap. This one has a real answer, and it is the least comfortable answer in the book: the layer boundary — the thing the previous sections spent their length building — is the largest single identified block of extra dispatch work in the serving graph, and the obvious way to remove it is the one thing we are not allowed to do. Here is how we found that out, and what we took instead.

Start with the census. The one-token dispatch-gap investigation reconciled the production (serving) graph’s 760 profiling closures against the legacy fused route’s 564 — a difference of +196 — into four families plus one copy [docs/decode-dispatch-gap-20260815.md, “Corrected closure-count diff at position 2,048”]:

The three norm-boundary rows are the 104 separated norm-boundary groups (+51 +52 +1): they are this chapter’s boundaries — the layer exits and entries that TAIL#2 fuses on the teacher-forced route but that the serving graph keeps separated, publishing each norm as its own node through the pinned ggml kernels. The instrument’s own label diff names them: production carries layer.*.post_attn_norm + layer.*.ffn_norm where the legacy route has one post_attn_ffn_norm, and layer.*.post_ffn_norm + output_norm where legacy has post_ffn_next_norm [docs/decode-dispatch-gap-20260815.md, label table].

So we tried to take them. The fork is the one any reader of that table would take: the tail kernel we just read is proof that a dual-eps boundary can be fused exactly, the serving graph keeps those same boundaries separated, so fuse them there and the largest rejectable family in the diff simply goes away. We built the hybrid and we expected it to come out bit-identical — the corrected fusion had already come out bit-identical, on the same arithmetic, in the same kernel family.

It did not. The fusion that would remove those 104 groups exists and was rejected because it changes logprobs beyond contract. The seductive part is that the greedy token survived: reusing a retained-activation schedule with fast fused boundaries picked the same word, so a casual sample of the model looks identical to the baseline. The logits did not survive, and the hybrid postmortem is exact about how far beyond contract they went — a full-logit maximum absolute error of 4.6300888e-4, a normalized-logprob maximum error of 3.197146176834309e-4 against the 1e-4 contract, with 201,970 of 202,048 logits differing and the first KV divergence in layer 1, value plane element 524,115, one f16 ULP apart (bits 39,892 vs 39,893). The runs that proved it are retained [docs/decode-dispatch-gap-20260815.md, “Rejected hybrid postmortem”; receipts muser-receipt://pinned-token-parity-20260814-v{3,4}/].

Follow that divergence back and it is a single rounding difference in the first layer’s residual chain — one bit, in exactly the structure this chapter has been describing — amplified through 51 more layers until it breaks a public numerical commitment. That is the lesson, and it is why the fusion rule of the previous section is stated as a rule rather than a preference: the boundary you may fuse is the one that keeps its rounding points, and this hybrid quietly dropped one. The decision followed from the lesson without much argument. The attempt “was removed rather than hidden behind a tolerance or shipped as an alternate route” — and the routing comment of Ch 18 §18.6 (decode.rs:2085-2091) is the standing consequence: serving decode takes the batch graph with the exact pinned kernels because the fused boundaries breach tolerance.

So what survived the rejection? Exactly one removal, and it is deliberately small: the last-row copy — one closure and one 6,656-element f32 copy, worth −0.136 ms GPU (−0.34 %) in a single-run diagnostic, with no wall-time claim (the +4.380 ms wall sample was submit/wait noise) [docs/decode-dispatch-gap-20260815.md, “Landed and rejected reductions”]. The corrected exact dual-norm fusion (the 32sg kernel’s pinned-reduction form) matched the baseline’s full-logit SHA-256 but was retained as “historical self-consistency only” — useful, not sufficient. The distinction is worth holding onto, because it is easy to mistake one for the other: matching your own previous bytes proves you have not broken yourself, and proves nothing about whether you agree with the engine you are being measured against. By that point the agreement was the whole game. The five-sample streamed serving number after it was 28.290 tok/s against llama’s 33.428 (ratio 0.8463), Stage A still open at that point [same doc]. The gap closed later only when the anchor itself changed — J0 made llama’s own bytes the gate and J1 transplanted llama’s attention DAG [ledger Stage A close-out, Arc 1 of the campaign; the parity outcome is the six-depth matrix at or above 1.0 of Ch 38].

So the honest accounting for this chapter: the layer boundary is where the +196 lives, and the fix that looks obvious is the one the contract forbids. Every closure in those 104 groups does required math; the investigation’s own conclusion is that “no repeated closure performing provably identical arithmetic was found” — the boundary cost is real but the only cheap removals change bits, and changing bits is the one thing Muser’s public logprob contract cannot buy. Ch 35 carries the full hazard framing; Ch 40 files this as the canonical measured rejection.

The loop is closed: 52 times you have watched a layer normalize, attend, gate, project, feed forward, and fold its delta into the stream — and the last tail wrote activations.hidden through the final norm. One question is left in Part IV: what does the model actually say? The 6,656-vector now sitting in hidden is about to meet the largest matvec in the engine — 202,048 output rows — and the soft cap that bounds what it may score. Ch 20.

References

• crates/muser-engine/src/shaders/ferrite/rmsnorm_batch_tail.metal:142-201 — muser_fused_norm_residual_rms_norm_32sg, the dual-eps tail kernel (primary source; the comment block states the eps split and why the single-epsilon kernel is invalid).
• crates/muser-engine/src/metal/encode/norm.rs:163-241 — the 32sg wrapper and its batch form (1,024 threads, 144 B threadgroup memory, “6,656-wide Muse tail” geometry note); :97-160 the batch dual-eps sibling; the strict decomposed route under MUSER_CROSS_VENDOR_QK.
• crates/muser-engine/src/decode.rs:5863-5889 — down projection + tail + next_norm selection in encode_token; :5806-5818 the first (post-attention) tail; :1328-1334 the fusion-control flags; :2085-2091 the serving-exactness routing comment.
• crates/muser-engine/src/metal/encode/qkv.rs:429-450 — the one-token pinned-metallib matvec path (Q4_K/Q6_K rows-per-group table); crates/muser-engine/src/metal/encode.rs:278-280 PSO registration.
• crates/muser-bench/src/m16.rs:179-198 — ffn_down-q4k / ffn_down-q6k shape-dtype evidence for the mix; docs/quickstart.md:16 the Q6_K fail-closed routing note.
• crates/muser-engine/src/reference.rs:526-541 — the oracle’s post-FFN norm, residual add, and l_out-{il} capture point.
• [docs/decode-dispatch-gap-20260815.md] — the 760/564 reconciliation, the 104 norm-boundary groups, the rejected hybrid postmortem (3.197e-4 vs the 1e-4 contract), the −0.136 ms exact copy removal, and the 0.8463× Stage-A-open snapshot.
• [receipt muser-receipt://pinned-token-parity-20260814-v3/], [receipt muser-receipt://pinned-token-parity-20260814-v4/] — retained evidence for the rejected hybrid.
• [ledger Stage B close-out] — docs/goal-parity-ledger-2026-08.md, L0 “Winner (n32)”: the Q4_K/Q6_K ffn_down per-dispatch timings.
• Ch 12 — the sandwich and the tail family’s first appearance; Ch 13 the pinned matvec; Ch 18 the FFN budget this chapter completes.
• Ch 35, Ch 38, Ch 40 — the downstream chapters that build on §19.8.
• [ferrite-book Ch 18] — the ancestor’s Q4_K_M mix table and the “where the bytes are, not where the gap is” device, ported with Muse’s real mix.