Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Chapter 32 — Precision across the handoff

status: polished · path: Muse Glimmer, pinned Muser tree

Prerequisites: Chapters 5–7 (the two weight lanes and their arithmetic), Chapter 20 (the soft cap and why it changes logit gaps), Chapters 28–31 (the producer, the transport, and the wire it rides). This chapter assumes you believe the bytes arrive intact — Chapter 31 bought you that — and asks the harder question: are they right?

Chapter 31 ended on a deliberately sharp distinction: intact is not the same as correct. The Handoff V2 machinery of Chapter 30 proves the bytes on the Mac are the bytes the producer signed; nothing in a MAC tells you the producer’s arithmetic was worth signing. This chapter is the one this book was explicitly demanded to contain — the user’s own words were that what they wanted read out of this codebase was “the special care we had for precision while bringing prefill from vLLM” — and it is where that care lives: in declared policies, sealed bounds, mode-separated identities, an integer-dot anchor, and a drift record that is published even when one row of it is unflattering.

The demand is not sentimental. Ask first what breaks if this is wrong. When the GX10 prefills, a different machine — CUDA kernels, NVFP4 tensor cores, cuBLAS reduction orders, Blackwell software E2M1 rounding — computes the KV cache that the Mac’s decoder will treat as ground truth for every subsequent token. Not for one token: for all of them. A producer that is slightly wrong does not announce itself with a crash. It hands over a plausible cache, and the Mac then generates fluent, confident text on a foundation nobody checked.

We went in expecting the disagreements to be rounding noise. They were not even the same kind of arithmetic. The two engines do not agree on how to sum: the wizard campaign traced one mismatch to “CUDA’s serial 128-dim attention reduction vs Metal’s 32-lane tree, and F32 vs F16 residual materialization,” and that only surfaced after a layer-0 ladder chase (attn_norm-0 element 4 → K RoPE 256 → attn_out-0 element 4,096) isolated two arithmetic-ABI splits. We kept the trail: [measured-numbers §2 Arc 4; ledger §2b, 2026-08-24].

Nor is this a Muser embarrassment that trying harder would fix. Nobody in the field achieves bitwise CUDA↔Metal equivalence — llama.cpp itself uses tolerance-based backend diffs [docs/disaggregated-prefill-sealing- plan-20260818.md §4]. So “good enough” has to be a contract, and the contract has to be checkable. This chapter is about the three contracts Muser actually runs.


32.1 The trust ladder

Before the details, the shape. The question to carry through the rest of the chapter is not “is the handoff correct” — that is unanswerable as posed — but how much agreement did we buy, and what did each level of agreement cost? Muser’s disaggregated lanes answer it as a three-rung ladder, and every rung names its price and what it rules out (Figure 32.1):

                THE TRUST LADDER (strictest at the top)

 ┌─────────────────────────────────────────────────────────────────────┐
 │ RUNG 3 — BIT-EXACT FULL HANDOFF  (combined kquant lane)             │
 │   exact target tokens  AND  exact FULL logits (max/mean delta = 0)  │
 │   AND exact DFlash tokens/trace                                     │
 │   cost:    a versioned cross-vendor arithmetic ABI — wizard         │
 │            attempts 10–31 chasing one f16 ULP in layer-1 V into     │
 │            51.7 M differing logits (4–7 accelerator-hours)          │
 │   rules out: any logit-level doubt at all; what remains is          │
 │            scheduling and speed                                     │
 ├─────────────────────────────────────────────────────────────────────┤
 │ RUNG 2 — EXACT TOKENS + DECLARED BOUNDED LOGITS  (native/text)      │
 │   greedy tokens bit-exact; full-logit drift must fit a rule that    │
 │   is WRITTEN IN THE FROZEN IDENTITY:  max |Δ| < 11,  mean |Δ| < 1.25│
 │   cost:    "zero drift" wording is prohibited; the envelope must    │
 │            be re-measured per identity; logits are never claimed    │
 │            equal                                                    │
 │   rules out: token divergence, nondeterminism, unbounded or         │
 │            undeclared drift                                         │
 ├─────────────────────────────────────────────────────────────────────┤
 │ RUNG 1 — PARITY-WITHIN-NOISE  (plain decode, both lanes)            │
 │   NVFP4 35.491 tok/s vs adjacent kquant 35.440 — a 0.14% split      │
 │   inside 0.03–0.13% CVs; "never call decode faster"                 │
 │   cost:    no throughput claim may lean on the split                │
 │   rules out: the idea that either lane is the other's inferior      │
 └─────────────────────────────────────────────────────────────────────┘

Figure 32.1: The trust ladder. Rung citations: rung 3 [claims #9; measured-numbers §1l]; rung 2 [scripts/gx10/vllm/native_onboarding_ identity_v1.json:57-71; crates/muser-server/src/node/smoke.rs:451-469]; rung 1 [ledger §P1.3; claims #11]. A lane picks its rung by declaring it at enrollment — it cannot drift between rungs unnoticed, because the rung is part of the identity (§32.3).

The rest of the chapter climbs the ladder from the middle out: rung 2 is the product lane and the most instructive (§32.2–32.3), rung 3 is what the wizard bought with its ABI chase (§32.4), rung 1 is where decode lands (§32.5). Then the apparatus underneath all three: the soft cap’s role in the contract’s units (§32.6), the integer-dot anchor (§32.7), mode-separated cache identities (§32.8), the measured drift record and deep content controls (§32.9–32.10), the reference lock (§32.11), and the reason the Mac never trusts when it can re-check (§32.12).

32.2 Declaring the policy: the rule is written before the run

Start with the question this section answers: when two engines cannot agree exactly, who decides how much disagreement is acceptable — and when do they decide it? The second half is the part with teeth. A rule invented once the outputs are in hand is not a rule; it is a description of what happened, wearing a rule’s clothes. So Muser’s rule is written down first, and written somewhere it cannot be quietly edited later.

The native/text lane’s contract is therefore not a wiki page or a convention; it is a field in the frozen onboarding identity that both peers pin:

// scripts/gx10/vllm/native_onboarding_identity_v1.json:57
"onboarding_qualification": {
  "schema": "muser.native-text-onboarding.v1",
  "prompt_tokens": 2048,
  "output_tokens": 256,
  "repetitions": 3,
  "prompt_sha256": "149ac0d9c37c957823e53c0637b52a38f2ac601089dbda9f98eec4bc5f369030",
  "require_exact_tokens": true,
  "require_deterministic_remote_output": true,
  "full_logits": {
    "mode": "bounded-drift",
    "maximum_absolute": 11.0,
    "maximum_mean_absolute": 1.25
  },
  "basis": "muser-receipt://nvfp4-f4-native-text-p4-accelerator-20260817/20260817T074427Z-44c761545afa47d78f8407734f8d2145.command.log"
}

Deconstruct it. Tokens are exactrequire_exact_tokens: true, no tolerance, no exceptions. The remote output must be deterministic across repetitions — a producer whose own output wobbles cannot be gated at all. And full logits are bounded, not equal: the mode is literally named bounded-drift, with two numbers — a maximum absolute delta (11.0) and a maximum mean absolute delta (1.25) — plus a basis receipt tying the numbers to the measurement they came from. This is what “the special care” means in practice: the tolerance is declared before any qualification run, in a file whose SHA-256 is part of the lane’s identity, not chosen after seeing the outputs.

And the Mac side does not merely read that file — it re-derives it against a sealed constant and refuses to proceed if they disagree:

#![allow(unused)]
fn main() {
// crates/muser-server/src/node/artifacts.rs:188
const NATIVE_SEALED: SealedNativeIdentity = SealedNativeIdentity {
    // … checkpoint, image, adapter, consumer, tokenizer, template,
    // context-policy, RoPE-cache digests …
    target_cache_identity_sha256:
        "a3bbd72fc16116322b5c9dc701f155d35349146b2af3e3b8465732b7df1eabd0",
    prompt_sha256: "149ac0d9c37c957823e53c0637b52a38f2ac601089dbda9f98eec4bc5f369030",
    // …
    maximum_logit_absolute: 11.0,
    maximum_logit_mean_absolute: 1.25,
};

// crates/muser-server/src/node/artifacts.rs:472
        if qualification.schema != "muser.native-text-onboarding.v1"
            || qualification.prompt_tokens != 2_048
            || qualification.output_tokens != 256
            || qualification.repetitions != 3
            || !qualification.require_exact_tokens
            || !qualification.require_deterministic_remote_output
            || qualification.full_logits.mode != "bounded-drift"
            // … finite, positive, and exactly the sealed values …
            || qualification.full_logits.maximum_absolute != NATIVE_SEALED.maximum_logit_absolute
            || qualification.full_logits.maximum_mean_absolute
                != NATIVE_SEALED.maximum_logit_mean_absolute
            || qualification.basis.is_empty()
        {
            return Err("native onboarding qualification rule is invalid".into());
        }
}

(lines 188–210 and 472–488 elided in the middle: the digest pins; see file). Notice what that closes off. An operator who “relaxes” the rule to 12.0 in the JSON now fails onboarding against the compiled constant — the tolerance can only change through a code change that says so, in a commit, in front of a reviewer. Widening a bound stops being a configuration decision and becomes an edit to the product.

Where did the two ceilings come from, then? Not from taste. We measured the lane first and set the rule a hair above what we saw. The fast-vs-exact envelope on the 2,048/256 five-repetition comparator was max/mean 10.884401 / 1.233789 with 100% token agreement in all five runs, and 7.270581 / 1.040619 on the 32-token standard fixture [docs/nvfp4-fast-lane-evidence-20260817.md §Determinism]. Sitting the ceiling that close to the observed envelope is a trap we set for ourselves on purpose: a lane that drifts even slightly worse than the lane we measured does not squeak through, it fails onboarding. It is also why the claims register prohibits the flattering framing — the fixture “may be described as deterministic and token-identical only with its scope; zero drift … remain[s] prohibited” [claims #10]. We are allowed to say the drift is bounded. We are not allowed to say there is none.

32.3 Checking the policy: three ordered handoffs, per sample and per summary

A declared rule is only as good as its enforcement, so the next question is: who reads the rule at run time, and what happens to a run that misses it?

The wizard’s smoke step runs the qualifier three times. What the operator watches scroll past is the contract restated in English — “three ordered 2,048/256 exact-token handoffs with bounded full-logit drift” [crates/muser-server/src/node/smoke.rs:43-52] — driven by the native recipe’s own flags, --onboarding-native --drift-graded --reference-once [crates/muser-server/src/node/smoke.rs:644-650]. The middle flag is the interesting one. Left alone, the qualifier refuses on any logit digest mismatch at all [crates/muser-bench/src/remote.rs:690-694], which is the correct default for a lane claiming equality and useless for a lane claiming a bound. --drift-graded trades refusal for measurement: retain the comparison, compute the deltas, then judge them. And the wizard judges twice — once per sample, once over the summary arrays:

#![allow(unused)]
fn main() {
// crates/muser-server/src/node/smoke.rs:451
        if let Some(rule) = native {
            let maximum = value
                .get("remote_local_logit_max_abs")
                .and_then(serde_json::Value::as_f64)
                .ok_or_else(|| "native sample omits maximum full-logit drift".to_string())?;
            let mean = value
                .get("remote_local_logit_mean_abs")
                .and_then(serde_json::Value::as_f64)
                .ok_or_else(|| "native sample omits mean full-logit drift".to_string())?;
            if !maximum.is_finite()
                || !mean.is_finite()
                || maximum > rule.full_logits.maximum_absolute
                || mean > rule.full_logits.maximum_mean_absolute
            {
                return Err(format!(
                    "native full-logit drift exceeds its identity rule: max={maximum}, mean={mean}"
                ));
            }
        }
}

The summary-level check then re-verifies that all three repetitions agree on token rate exactly 1.0, that every per-repetition maximum and mean sits inside the rule, and that the fast generated-token digest is present [crates/muser-server/src/node/smoke.rs:538-569]. A missing field is a refusal, not a zero (“native sample omits maximum full-logit drift”). Put the same idea the other way round, because it is the one readers hand-wave past: a sample that forgot to report its drift is not a sample with no drift. Silence is not a measurement. Absent evidence is failed evidence — the same fail-closed posture as everywhere else in this book.

All of which is machinery until it meets a real node. The measured result of running exactly this gate live: native/text attempt 9 passed all seven progress labels with three exact-token handoffs under the bounded-logit rule (deltas inside 10.884401 / 1.233788776) at payload rates 6.866 / 6.976 / 6.708 Gbps, and the node reached healthy. The run that proved it is retained: [claims #9; measured-numbers §1l; receipt wizard-validation-20260823/attempt-9-native-live-20260824T051305Z/validation-summary.json].

32.4 Rung 3: what bit-exact costs

One rung up sits the combined kquant lane, and it asks for the whole top box of Figure 32.1: exact target tokens, exact full logits — the digests equal, deltas zero — and exact DFlash tokens and trace. This is the rung we assumed would be a scheduling problem. It was not, and the story of finding that out is the most useful thing in the chapter.

With the transport already trustworthy, we expected the remaining work on the combined lane to be plumbing: line the sequence up, pin the fixtures, watch the digests agree. The digests did not agree, and the wizard spent attempts 10 through 31 finding out why. The trigger was as small as a divergence can be — a single F16 ULP in layer-1 V, the last place anyone wants to go hunting — and by the time it had propagated through the stack it had become 51.7 M differing logits [ledger §2b, 2026-08-24]. That escalation is the point: at the top of the ladder there is no such thing as a small disagreement, because nothing downstream damps it.

What the chase found underneath were the two reduction-order splits named in this chapter’s opening, and no amount of retrying was going to dissolve them. They had to be specified. So the fix was a versioned cross-vendor arithmetic ABI, and we know what it cost: 4–7 accelerator-hours [measured-numbers §1l]. Attempt 31 then passed 7/7 with bit-exact logits and payload rates 9.812 / 8.887 / 8.690 Gbps [claims #9; receipt wizard-validation-20260823/attempt-31-combined-full-20260824T132639Z/validation-summary.json].

The lesson generalizes the integer-exact philosophy of Chapter 7 §7.4.2: when you cannot reproduce another engine’s reduction topology, you either (a) pin an ABI that makes the controllable parts associativity-free — the a16-q8 kernel’s trick — or (b) chase and pin each divergence site until the digests match, which is what the wizard did. What you may not do is (c) declare a tolerance and quietly widen it each time a cell fails. The 4–7 accelerator-hours are the price of (b), and the receipt chain is what makes the spend auditable.

32.5 Rung 1: parity-within-noise, never “faster”

At the bottom of the ladder, the plain decode lanes — where the question is not whether the two lanes agree, but whether either one is entitled to brag. The paired P1.3 cells measured native NVFP4 decode at 35.490711722 tok/s (CV 0.130%) against an adjacent kquant control at 35.439527527 tok/s (CV 0.037%) — a +0.1444% split inside the noise of cells whose CVs bracket it [ledger §P1.3]. The claims register’s standing instruction is three words long: “Never call decode faster” [claims #11]. The trust-relevant content of rung 1 is that neither lane is entitled to be called the other’s quality fallback by speed arguments — the native lane’s differences from kquant are quality-shaped, not speed-shaped, and §32.10 is where they live.

32.6 The soft cap is part of the contract’s units

A tolerance is a number and a unit, and the unit is the part everyone forgets. So before trusting a bound, ask what it is measured in. Chapter 20 §20.7 proved the soft cap is order-preserving (greedy tokens cannot move) but gap-compressing — by hand, a 6.00 raw gap becomes 4.915 capped, a 20.0 gap becomes 4.049. Carry that into this chapter: every number in the bounded-drift rule is a delta of capped logits, because both engines apply 1/√26 then tanh@20 as the last step before these bytes are compared [Ch 20 §20.4, §20.7]. That has two consequences you must hold simultaneously. First, the bounds 11.0/1.25 are units of the contract — recalibrate either engine’s scale-and-cap order and the tolerance is meaningless, which is why the whole transform is pinned in both engines. Second, the cap compresses exactly the large-logit regime where the two lanes’ numerics disagree most, so the measured drift envelope is smaller than an uncapped comparison would produce — the bound is doing quiet work, and “without the cap the same deltas would be much larger and the tolerance would have to be re-derived” [Ch 20 §20.7]. A reader comparing Muser’s 11.0 against some other engine’s uncapped logit tolerance is comparing different units — do not.

32.7 The integer-dot anchor: a producer that exists to be compared against

Every ladder needs a plumb line. “Bounded drift” is a claim of the form no farther than this from something — and the something has to be an engine we can re-run on demand, or the bound is unfalsifiable. Muser’s plumb line is the exact producer mode: the same GX10 node, the same transport, but a producer whose NVFP4 arithmetic is integer-dot deterministic — built to be compared against, not served. The shipped lane matrix gives it its own row:

LanePrefillDecodeSpeculativeIntended use
Native NVFP4Spark tensor-core FP4Mac NVFP4, 35.491 tok/sRejected (fail-closed)Fast product lane
kquant/referenceReference pathkquant, 35.440 tok/s107.9 tok/sSpeculative + reference lock
Exact NVFP4 flagInteger-dot verification producerMac NVFP4Verification onlyDeterministic anchor

[docs/muser-architecture.md, lane matrix]

The mode is selected producer-side only, by the Python environment flag MUSER_NVFP4_EXACT=1. It does not exist anywhere in the Rust tree — a Mac-side reviewer will find nothing to set — and the native benchmark refuses to run with it set, because mixing modes would invalidate the claim being measured:

# scripts/gx10/vllm/benchmark_native_prefill.py:98
    # The benchmark is intentionally stock: importing muser_vllm exact modules
    # or setting MUSER_NVFP4_EXACT would invalidate the native-path claim.
    if os.environ.get("MUSER_NVFP4_EXACT") == "1":
        parser.error("native benchmark refuses MUSER_NVFP4_EXACT=1")
    os.environ["MUSER_NVFP4_EXACT"] = "0"

The resident native daemon likewise pins MUSER_NVFP4_EXACT=0 into the container’s environment [scripts/gx10/vllm/muser_native_prefilld.py:446]. (Chapter 7 §7.7 introduced this split; the point to add here is what the anchor is for.) The exact lane’s value is that its outputs are stable enough to be a reference: the G3 live recheck re-ran the retained strict cell after routing changes and reproduced every retained digest bit-for-bit — generated-token SHA, full-logit digests, payload SHA, all 52 seam digests, KV max deltas 9.625/18.4580078125, logit errors 7.270581/1.040619, 32/32 tokens [docs/nvfp4-fast-lane-evidence-20260817.md §G3]. When you ask “how far has native drifted?”, you are asking it against this lane’s numbers.

32.8 Mode-separated cache identities, and refusing the unknown

An anchor is only an anchor if you can tell it apart from the thing it anchors. Bounded drift is safe only when native KV and exact KV can never be mistaken for each other — a cache entry produced by tensor cores must not be served to a session whose contract says “integer-dot anchor,” because such a session would then be measuring drift against drift and reporting the answer as ground truth. The fast-lane evidence note states the rule in one line: exact and native producers “use mode-separated target-cache identities, so exact and native KV entries cannot alias” [docs/nvfp4-fast-lane-evidence-20260817.md §Product route]. Concretely, the receiver configuration carries a target_cache_identity_sha256 field that is validated as a digest at load [crates/muser-cluster/src/config.rs:41-42, 103-115], and the sealed native identity pins the exact value (a3bbd72f…, [crates/muser-server/src/node/artifacts.rs:206-207]). A cache identity is a property of the recipe, not of the model file alone.

The same fail-closed logic covers recipes Muser has never heard of. Enrollment maps a producer lane to exactly one qualification contract, and an unknown lane dies before any key is minted:

#![allow(unused)]
fn main() {
// crates/muser-server/src/node/registry.rs:39
/// Exact qualification contract selected by the enrolled producer lane.
/// Keeping this exhaustive beside `ProducerKind` means adding a lane without
/// choosing a recipe is a compile error; an unknown serialized lane is
/// refused while loading the registry, before enrollment can mint keys.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum QualificationRecipe {
    KquantTargetPlusDflash,
    NativeText,
}
}

Two recipes exist — KquantTargetPlusDflash (the rung-3 combined lane) and NativeText (the rung-2 bounded-drift lane) [crates/muser-server/src/node/registry.rs:49-76] — and the mapping is total: Llamacpp → KquantTargetPlusDflash, Native → NativeText. Because the compiler forces the match to be exhaustive, a future lane must choose a recipe to compile; because loading refuses unknown serialized values, an old registry file cannot silently re-interpret itself. The architecture document’s summary is the sentence to quote whole: “Exact and native producers derive different target-cache identities; an unknown recipe is refused at enrollment” [docs/muser-architecture.md §Durable and remote KV]. (The native lane additionally cannot enroll a DFlash identity at all [crates/muser-cluster/src/config.rs:128-131] — that refusal is Chapter 33’s opening subject.)

32.9 The measured drift record: what bounded actually means

A bound with no published measurement behind it is empty theater: it tells the reader what we promised, never what actually happened. So here is the drift the rule bounds. The fast-lane evidence note’s determinism section is that record (Figure 32.2), for the standard 2,048-token prompt with 32 greedy tokens [docs/nvfp4-fast-lane-evidence-20260817.md §Determinism]:

QuantityMeasured envelope (fast vs exact)
Greedy token agreement32/32, 100%
First divergent tokennone
Full-logit maximum absolute error7.270581
Full-logit mean absolute error1.040619
KV key maximum absolute delta across 52 layers9.625
KV value maximum absolute delta across 52 layers18.458008
Per-layer KV key mean-absolute range0.015648–0.431876
Per-layer KV value mean-absolute range0.044231–0.387556

Figure 32.2: The standard-fixture drift envelope. The KV deltas are the raw material of everything downstream: 9.625 on keys and 18.458 on values, per layer, feeding attention for every future token — and the greedy tokens still agree 32/32. The 2,048/256 five-repetition comparator extended the same result — 100% token agreement in all five runs, max/mean 10.884401/1.233789 — and immediately disclaims its own scope: “This demonstrates deterministic, coherent output on the standard fixture; it does not prove a general zero-divergence contract” [docs/nvfp4-fast-lane-evidence-20260817.md §Determinism]. And the seam itself is reproducible: repeated A requests, A/B/A queue interleaving, and a full engine restart reproduce the same 52 layer hashes, the same seam SHA-256, and the same payload SHA-256 [docs/nvfp4-fast-lane- evidence-20260817.md §Determinism]. Drift, yes — but deterministic drift, which is the difference between a risk you can bound and one you can only fear.

32.10 Deep content controls: the sensitivity that was published, not hidden

A 2,048-token fixture is a shallow probe of a 131,072-position model. So the honest next question was the uncomfortable one: what is happening further out, where we had not yet looked? The deep ladder went looking, and this is the stretch of the record that contains a row that fails. We walked into it in three stages, and the order we walked them in matters as much as the outcome:

  • E1 calibrated the yardstick first. Before judging native against kquant, the campaign measured an accepted alternate quant (Q6) against kquant on the same rows and set each cell’s disagreement gate from that two-sided 95% Wilson bound plus two points — calibrated gates of 8.796–15.299% across the long-context cells. Native exceeded the band by 1.746–1.996 points at 8k/16k/32k [docs/nvfp4-fast-lane-evidence- 20260817.md §E1]. A tolerance derived from a second real quantization cannot be accused of being tuned for convenience.
  • E2 asked whether length was the culprit, and the answer was no. The natural hypothesis — written down before the run, which is what makes the answer worth anything — was that native drift grows with context: past some length the lane goes bad, and you route around it with a number. E2 held content fixed across three nested documents (rust, python/shell, docs) and varied length. Rust and python passed every disagreement row at every length 2k–32k; only the docs document exceeded its gate at 8k/16k/32k, and the 512-token position profiles kept the sensitive regions localized rather than showing a length transition — so the preregistered replicated-length criterion was false [docs/nvfp4-fast-lane-evidence-20260817.md §E2/E3]. The tidy story we expected was not on offer. What we had instead was a content effect, which is far harder to route around than a length threshold, because content is not a knob on the request.
  • The stage-3 yardstick published the exceedance. At 65,536 tokens on the docs corpus, native-vs-kquant top-token disagreement was 15.134% against a calibrated gate of 13.339% (relative PPL +4.227%) — recorded, published, and explicitly scoped: a content-local sensitivity that “did not replicate cross-document”; persistence at 131,008 could not be tested because the corpus is too short [claims #10; receipts kvpack-ladder-20260820/attempt-5-…-stage3-compact/stage3-e2-quality/].

So what do you do with a row like that? Before publishing it, we tried to make it go away. The route-exhaustion matrix put eight native vLLM runtime variants (chunked prefill, BF16, Triton, batch-invariant CUTLASS, FlashInfer B12X/cuDNN, engine ceiling) against the docs 65,536 row, on the expectation that one of them was the real culprit and the sensitivity was a configuration mistake wearing a numerics costume. All of them failed identically or worse, and no runtime change was promoted [ledger §"native-route exhaustion"; measured-numbers §1i]. There was nothing left to out-configure.

That left two cheap ways out, and the record rejected both. Hiding the row would violate the publish-the-sensitivity rule [measured-numbers §6 rule 9]. Capping context — the “native up to N, kquant beyond N” policy — sounds principled right up until you look at which cells actually fail: a cap would have silently included the failing 8k cell while failing to cover the passing deeper cells, the non-monotonicity that killed exactly that policy back in the D1 routing ladder [docs/nvfp4-fast-lane-evidence-20260817.md §D1]. So no context cap was imposed through the measured 32k range, and the sensitivity was published instead.

32.11 The reference lock: an explicit lane, not an automatic proxy

What is a user who does care about the docs-class sensitivity supposed to do? The answer is deliberately manual: “Users who need a reference lock select the existing kquant lane explicitly” [docs/nvfp4-fast-lane-evidence-20260817.md §E2/E3]; the architecture document repeats it as a first-class property — “the kquant lane is the explicit reference lock for that class” of high-entropy numeric/digest/tabular documentation content [docs/muser-architecture.md, lane-matrix paragraph]. Why explicit? Because no automatic disagreement proxy is claimable — building one “would require a second reference computation” [docs/nvfp4-fast-lane-evidence-20260817.md §E2/E3], i.e., the proxy would itself need the very reference it is pretending to replace. The same logic explains the lane matrix’s “kquant/reference” row: a reference lock is a lane you can route to, whose own parity is maintained against the pinned llama.cpp comparator (the parity ledger is Ch 38’s subject), not a background oracle.

32.12 Why the Mac re-checks rather than trusts

Everything above describes contracts the producer satisfies. The final layer of care is that the Mac never takes even a satisfied contract on faith at qualification time. The remote qualifier’s module doc states the design in four lines — it uses “the same RemoteReceiver as serving” (so a benchmark cannot bypass the machinery), and “every sample performs cold local recomputation and an authenticated remote install, compares 256 greedy tokens plus every full target-logit row, and retains producer/transport phase times needed to prove real overlap” [crates/muser-bench/src/remote.rs:3-8]. The trust direction matters: the remote-installed KV is decoded and compared against a local cold recompute, token by token and logit row by logit row, before a lane is called qualified. The producer is not asked to certify itself; it is checked against a computation the Mac performed on its own.

Serving then inherits the disposition, not the measurement: once enrolled, the lane’s correctness lives in its identity and gates — and the one machinery that must never trust a draft in flight, speculative decoding, is under an absolute rule: “DFlash drafts are always verified by target distributions” [docs/muser-architecture.md §Model and engine], with the remote qualifier enforcing an acceptance floor of DFLASH_ACCEPTANCE_MINIMUM = 0.95 [crates/muser-bench/src/remote.rs:33]. The all-accept diagnostic that proved the native target+DFlash path correct (240/240 drafted tokens, canonical digest retained) is carefully recorded as establishing “correctness of the diagnostic, not competitive speculative throughput” [docs/nvfp4-fast-lane-evidence-20260817.md §G1 tail] — correctness and speed claims are never allowed to borrow each other’s evidence.

32.13 Tradeoffs

Bounded drift vs bit-exactness. Rung 2 gives up logit equality and in exchange gets a lane the tensor cores can actually serve at speed; rung 3 gets equality and pays an arithmetic ABI plus per-divergence-site pinning (4–7 accelerator-hours for the wizard’s fix [measured-numbers §1l]). The measured consequence of choosing rung 2 honestly: a drift envelope of 10.884401/1.233789 published next to the tokens-exact claim, and the prohibition on “zero drift” wording [claims #10]. The measured consequence of rung 3: digests equal, deltas zero, at 9.812/8.887/8.690 Gbps [claims #9] — but only after the ULP chase.

Publishing the sensitivity vs capping the lane. The docs@65,536 exceedance (15.134% vs 13.339%) could have been “fixed” by a context cap; the D1 ladder measured why that fails (non-monotonic, no honest N exists) and E2’s preregistered criterion formally rejected the length-effect interpretation [docs/nvfp4-fast-lane-evidence-20260817.md §D1, §E2/E3]. What publishing costs: one uncomfortable row in the record, carried forever. What it rules out: a user unknowingly serving the sensitive class on the native lane with no way to know [measured-numbers §6 rule 9].

Explicit reference lock vs automatic proxy. An automatic disagreement-triggered fallback was rejected because the proxy needs a second reference computation to be trustworthy [docs/nvfp4-fast-lane-evidence-20260817.md §E2/E3]. The cost of the explicit lock is that routing is a human decision; the benefit is that no silent heuristic ever re-ranks the lanes on unaudited numbers.

Integer-dot anchor vs serving it. The exact producer is qualified enough to reproduce every retained digest bit-for-bit [docs/nvfp4-fast-lane-evidence-20260817.md §G3] and slow enough that an attempted 33,024-token exact-lane scoring pass was stopped during warmup when “the measured approximately 50x compute ratio” showed it would spend the experiment box just warming the graph [docs/nvfp4- fast-lane-evidence-20260817.md §G1 intro]. The anchor is a plumb line, not a product — which is why its matrix row says “Verification only” [docs/muser-architecture.md].

32.14 What comes next

The trust ladder now stands complete: bytes intact (Chapter 31), tokens exact, logits either equal or bounded-by-declared-rule, identities mode-separated, sensitivities published, and a reference lane one switch away. One precision machinery in this system remains unexamined — the one that makes a guess participate in exact inference: speculative decoding, where a five-layer draft proposes and the 52-layer target disposes, and where the same evidence culture had to pass a verdict on its own distributed variant. That is Ch 33: the local win, the fail-closed refusal, and the measured rejection — with receipts.


References

  • [scripts/gx10/vllm/native_onboarding_identity_v1.json:57-71] — the declared bounded-drift rule (11.0 / 1.25), exact-token and determinism requirements, and its basis receipt.
  • [crates/muser-server/src/node/artifacts.rs:188-210, 472-488]NATIVE_SEALED (incl. maximum_logit_absolute: 11.0, maximum_logit_mean_absolute: 1.25, the a3bbd72f… cache identity) and the validation that refuses any other qualification rule.
  • [crates/muser-server/src/node/smoke.rs:43-52, 451-469, 538-569, 644-650] — the recipe’s progress label, per-sample and summary bounded-drift checks, and the --onboarding-native --drift-graded --reference-once wiring.
  • [crates/muser-bench/src/remote.rs:3-8, 32-33, 690-694] — the qualifier’s cold-recompute + 256-token + full-logit-row comparison, LINK_GBPS_MINIMUM/DFLASH_ACCEPTANCE_MINIMUM = 0.95, and the non-graded refusal on logit digest mismatch.
  • [crates/muser-cluster/src/config.rs:41-49, 103-115, 128-131]target_cache_identity_sha256 validation; native mode cannot enroll DFlash geometry.
  • [crates/muser-server/src/node/registry.rs:29-76]ProducerKind/QualificationRecipe, the exhaustive mapping, and the unknown-lane refusal before key minting.
  • [scripts/gx10/vllm/benchmark_native_prefill.py:98-102], [scripts/gx10/vllm/muser_native_prefilld.py:446]MUSER_NVFP4_EXACT is producer-side Python only; the native benchmark refuses it set; the daemon pins =0.
  • [docs/nvfp4-fast-lane-evidence-20260817.md] — §Product route (mode-separated identities, Fallback B), §Determinism (the drift envelope and its disclaimer), §G1/§D1/§E1/§E2-E3 (widened fixtures, yardstick calibration, content controls, routing disposition), §G3 (bit-for-bit anchor recheck).
  • [docs/muser-architecture.md] — the lane matrix (exact-flag row: “Verification only”), “unknown recipe is refused at enrollment,” “DFlash drafts are always verified by target distributions,” the kquant reference-lock paragraph.
  • [claims #9], [claims #10], [claims #11]docs/launch-claims.md: wizard attempts 9/31 with their exactness policies and rates; the docs@65,536 sensitivity scope and prohibited “zero drift” wording; 35.491/35.440 parity scope (“Never call decode faster”).
  • [ledger §P1.3], [ledger §2b 2026-08-24], [ledger §"native-route exhaustion"] — the paired decode cells; the one-ULP wizard chase; the eight-variant exhaustion matrix.
  • [measured-numbers §1i, §1l, §2 Arc 4, §6] — the quality-gate table, wizard validation cells, the arithmetic-ABI arc, and the claim- discipline crib sheet (publish the sensitivity; evidence wins).
  • Ch 7 §7.6–7.7 — Fallback B and the producer-mode preview this chapter builds on.
  • Ch 20 §20.7 — the soft cap’s monotonicity and gap compression; why bounded-logit deltas are capped- logit units.
  • Ch 30, Ch 31 — the authentication and wire machinery underneath; Ch 33 — the verification machinery that never trusts a draft.