Repository navigation
feat(qwen35): prefill recall with the exact scorer on the serial path, recall profile on by default - #963
feat(qwen35): prefill recall with the exact scorer on the serial path, recall profile on by default#963dusterbloom wants to merge 32 commits into
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 31 files
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
View guided diff | Re-trigger cubic
davide221
left a comment
There was a problem hiding this comment.
Code review
Found 2 issues.
Copy to agent
Use the copy button in the top-right corner of the block, then paste into your agent.
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
Repository: Luce-Org/lucebox
Reviewed commit: 4d2313821a060a5de536fd496105e41a6fa0652a
Verify the findings against the current code, run relevant checks for any fixes, and explain any findings you reject.
<file name="server/src/common/kvflash_gpu_mass.h">
<violation number="1" line="239" end_line="241">
permute axes produce [m, NBT, HK, G], but the cpy destination view is shaped (G, HK, NBT, m)
The graph's per-layer output copy must place the mass of plane `hk*G + g`, block `gb`, query `j` into `o_t[l*HQ + hk*G + g, gb, j]`. The source tensor is `mm` reshaped to `[NBT, G, m, HK]` (dim1 index = g, dim2 = query j, dim3 = hk). `ggml_permute(ctx, x, axis0..3)` sets `result->ne[i] = x->ne[axis[i]]`, so `ggml_permute(ctx, ..., 2, 0, 3, 1)` yields shape `[m, NBT, HK, G]` — not the `[G, HK, NBT, m]` the comment claims and the `dst` view on line 240 (`ggml_view_4d(ctx, o_t, G, HK, NBT, m, ...)`) requires. To match `dst`, the axes must be `(1, 3, 0, 2)` (g, hk, gb, j). As written, `ggml_cpy(ctx, mp, dst)` is handed two tensors whose shapes differ and are not even the same permutation order as the strides imply: depending on the vendored ggml's `ggml_cpy`/`ggml_compute_forward_dup` handling of mismatched shapes (the definition to check in ggml.c), this either aborts the process on the first recall scoring call (the default mode-4 profile's GPU path), or copies elements in the wrong order so every chunk score/plane aggregate is garbage and recall picks arbitrary chunks. The unit tests in this PR (test_kvflash_gpu_mass.cpp) exercise exactly this path with G=3, m=8, so if they genuinely pass on the vendored ggml, that port must special-case permuted copies — that behavior in `ggml_compute_forward_dup`/`ggml_cpy` is the fact I could not verify. Fix: change the permute to `ggml_permute(ctx, ggml_reshape_4d(ctx, mm, NBT, G, m, HK), 1, 3, 0, 2)` so the source shape matches the destination view (G, HK, NBT, m).
<suggestion>
ggml_tensor * mp = ggml_permute(ctx, ggml_reshape_4d(ctx, mm, NBT, G, m, HK), 1, 3, 0, 2); // [G, HK, NBT, m]
</suggestion>
</violation>
</file>
<file name="server/src/qwen35/qwen35_backend.cpp">
<violation number="1" line="3165" end_line="3167">
Mode-4 'pool fits' gate ignores the preamble pin reserve, so recall is skipped for prompts that no longer fit the histor
`kvflash_recall_setup` decides whether mode-4 recall runs with `kvflash_pool_fits(end_abs, ctk, reserve, kvflash_pager_.n_pool_blocks())`, which compares the prompt's chunk count plus reserve against the **whole** pool. But with `LUCE_KVFLASH_PREAMBLE_ROWS` active (the shipped recall profile sets it to 1152 rows = 18 chunks of 64, taken out of `--kvflash N`), the pool blocks available for history are `n_pool_blocks - preamble_reserve/ct`, as the same PR's own budget math in `kvflash_recall_tail_chunks` (which subtracts `pin_reserve_chunks`) and the profile test `test_kvflash_recall_profile_leaves_the_pool_at_the_requested_size` (history = pool − reserve) establish.
Concrete break with `--kvflash 16384` and the default profile: a 15,500-token prompt whose last user turn is 400 tokens gives `end_abs/ct = 243` chunks and `reserve = ceil(400/64)+1 = 8`; `243 + 8 = 251 <= 256` so the gate logs "skip, fits" and recall is disabled. In reality 18 blocks hold the pinned system+tools preamble, so only 238 blocks remain: the prefill overflows the history budget and evicts the oldest unpinned chunks mid-prefill with no reselect — exactly the degraded case this feature exists to fix, and the model decodes on a pool that lost the old context the question needed.
Fix: subtract the pin reserve from the block count passed to `kvflash_pool_fits` (e.g. `kvflash_pager_.n_pool_blocks() - kvflash_preamble_reserve_ / ctk`), or add a `pin_reserve_chunks` parameter to the helper, so the gate tests the history budget the rest of the recall path uses.
</violation>
</file>
Issue overview
-
permute axes produce [m, NBT, HK, G], but the cpy destination view is shaped (G, HK, NBT, m) · likely (
server/src/common/kvflash_gpu_mass.h:239-241) -
Mode-4 'pool fits' gate ignores the preamble pin reserve, so recall is skipped for prompts that no longer fit the histor · confirmed (
server/src/qwen35/qwen35_backend.cpp:3165-3167)
Reviewed 4d231382 · 3 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again
CI failed on
|
4d23138 to
2f0b183
Compare
… tensors The attach path pins host staging with cudaMallocHost by default. On the stub-driver CI runner that call fails, so 3 tests in these two fixtures failed in the Linux build of Luce-Org#963. Set set_pinned_host_backing(false) before each attach, as test_kvflash_pager_serde.cpp already does. Local 3090, CUDA_VISIBLE_DEVICES="": 9/12 before, 12/12 after; 12/12 with the GPU.
2f0b183 to
a97fe64
Compare
score_chunks guarded S against the legacy 8-token kLookahead while passing n_lookahead=128 to the drafter; a named constant now feeds both. Review: Luce-Org#963 thread T8
… tensors The attach path pins host staging with cudaMallocHost by default. On the stub-driver CI runner that call fails, so 3 tests in these two fixtures failed in the Linux build of Luce-Org#963. Set set_pinned_host_backing(false) before each attach, as test_kvflash_pager_serde.cpp already does. Local 3090, CUDA_VISIBLE_DEVICES="": 9/12 before, 12/12 after; 12/12 with the GPU.
…t N blocks of SELECT_UNIT chunks The recall budget is 10 x RECALL_BLOCKS chunks (qwen35_backend.cpp in.recall_chunks), picked in SELECT_UNIT-sized units (kvflash_prefill_reselect.h k_units = recall_chunks / unit). The row said N blocks of SELECT_UNIT chunks, i.e. 12 chunks at the defaults instead of 60. Review: Luce-Org#963 thread T5 (cubic)
…inside the pool Production takes the preamble reserve out of --kvflash N (pool stays N). The fixture added it on top (256 + 96) and asserted the feature-off tail, so it encoded the old design. It now uses a 256-block pool and asserts the reduced tail (95 = off - 96); it fails if the tail formula ignores the reserve. Review: Luce-Org#963 thread T7 (cubic)
…or parallelism kvflash_q_cap_all_rows() keys off LUCE_KVFLASH only, so a tp_degree > 1 process reserves the 64-row capture although recall is off there. Threading the resolved plan through create_target_cache_partial (6 callers across layer-split, shard daemon, drafter) and the VRAM estimate exceeds the small-fix cap, so the comment now states the deliberate reservation and its size. Review: Luce-Org#963 thread T10 (cubic)
…or parallelism kvflash_q_cap_all_rows() keys off LUCE_KVFLASH only, so a tp_degree > 1 process reserves the 64-row capture although recall is off there. Threading the resolved plan through create_target_cache_partial (6 callers across layer-split, shard daemon, drafter) and the VRAM estimate exceeds the small-fix cap, so the comment now states the deliberate reservation and its size. Review: Luce-Org#963 thread T10 (cubic)
a97fe64 to
d98512f
Compare
CI failed on
|
score_chunks guarded S against the legacy 8-token kLookahead while passing n_lookahead=128 to the drafter; a named constant now feeds both. Review: Luce-Org#963 thread T8
…e_scorer kvflash_load_drafter() loads the residency drafter once and remembers a failed load; kvflash_ensure_scorer() calls it. Same messages, same order, no behavior change.
…estion queries in the selector With LUCE_KVFLASH_PREFILL_RESELECT=4 the prefill is cut at the start of the last user turn, a first pass over that turn captures its last query, the sealed chunks are scored with the exact attention mass (selector v2), the state is restored to the cut, the pager is reselected by the recall plan and the turn is written again under the new pool. Mode 5 takes the drafter's scores for the same cut in one pass. Under tensor parallelism recall stays off (kvflash_recall_features). Modes 1-3 stay off. Without a KVFlash pool nothing changes.
LUCE_KVFLASH_PREAMBLE_ROWS=R adds pool room (R rounded up to a chunk, clamped by max-ctx) for the pinned system+tools chunks, so the preamble does not eat the history budget, and re-applies the pins after a reset, restore or truncate. 0 turns the pin and the extra VRAM off.
Default-on under --kvflash: selector v2 with the GPU scorer (fixed in the backend, no variable), mode 4, six recalled blocks in 2-chunk units on the recency scoring pool, the 6144-row preamble pin (extra pool VRAM) and 64 user-question queries. The values are named constants in KvflashRecallProfile and are installed as env defaults. A variable already set keeps its value. Other architectures and launches without --kvflash install nothing.
…not 6144 The reserve grows the resident pool and every decode step attends the whole pool. 6144 rows took the pool from 16384 to 22528 and cost 4.7% decode; 1152 (18 chunks of 64) holds the measured 1064-token preamble pin and grows it by 7%.
…he pool no longer grows The reserve took the pool from 16384 to 17664 rows and every decode step attends the whole pool, so the profile silently raised the decode cost the user chose with --kvflash. The reserve (1152 rows) is now held inside N, leaving N - 1152 history rows. A reserve that leaves no history row (>= N) turns the pin off with a log line, like a missing <|im_start|> id.
…evice KV (16 layers, 4 kv heads, hd256, q8_0)
…per (layer, K/V) instead of one per kv head
…and not written since (dirty flag, d2h_skipped stat)
…unzeroed; a recall overwrites it, a fresh chunk zeroes it on handout, reselect flushes the rest
…ce at the end, not per hand-out, with residency unchanged (red: KvFlashStats has no fence_points yet)
reselect() recalls its page-ins as one batch on the page stream and waits once at the end; on HIP every hand-out was a host wait (one per page-in, ~0.7 ms/event over 191k events in the K3 night). Residency and bytes unchanged. The recent-pool and pick lines log the paging time on their own line.
Review findings #2-#7, #11, #15 of the G2 landing review. - Eviction hook: chunks at or past the committed position rank as pinned, so a drafter ledger that covers the whole prompt cannot evict the batch being allocated (RECALL_BLOCKS=0 lost 3 of 8 chunks). - Recall gate (kvflash_recall_gate): mode 4 and 5 skip when no sealed chunk precedes the turn, when chunks(R) + the turn's reserve already fit the pool (the turn was counted twice), and when the pool cannot hold pins + recall + reserve; those requests run as a recency pool. - GPU scorer: a request with nothing sealed or a free-VRAM refusal no longer latches kvflash_gpu_failed_ for the process. - Victim tier defaults to 2x the pool in mode 5 only, and the auto pool budget accounts for its device bytes. - Profile no longer exports PFLASH_SELECT_QUERY_PARSER (it changed PFlash compression's query), the retired PFLASH_SELECT_PARAGRAPH_JOIN or the unread LUCE_KVFLASH_SELECT_QUERIES; the mode-5 gate drops the parser. - Docs: --prefill-drafter, RECALL_BLOCKS=0 wording, victim tier row. Tests: test_kvflash_prefill_reselect (hook, gate, latch, victim), test_launch_policy, test_kvflash_pflash_map.
Review findings #9 and #10 of the G2 landing review. - victim_fifo_ gained an entry per page-out and was popped only when the free list ran empty; a freed slot is handed out again, so dead entries looked live. 2000 page-out/row-write cycles left 2000 entries; now the queue is swept (newest entry per chunk kept) past twice the tier. - kvflash_pooled_ubatch clamps to the unpinned capacity: with 1152 rows pinned in a 2048 pool, LUCE_PREFILL_UBATCH=1024 no longer puts a 16-chunk batch into 14 free blocks. Tests: test_kvflash_pager_victim_tier (fifo growth), test_kvflash_ubatch.
… tensors The attach path pins host staging with cudaMallocHost by default. On the stub-driver CI runner that call fails, so 3 tests in these two fixtures failed in the Linux build of Luce-Org#963. Set set_pinned_host_backing(false) before each attach, as test_kvflash_pager_serde.cpp already does. Local 3090, CUDA_VISIBLE_DEVICES="": 9/12 before, 12/12 after; 12/12 with the GPU.
…ageable staging on CPU tensors Same cause as the pager recall and pin tests: attach pins host staging with cudaMallocHost, which fails without a CUDA device (the stub-driver CI runner). CUDA_VISIBLE_DEVICES="" on the 3090: 3 of these tests failed before, all pass after; with the GPU they pass both ways.
|
Thanks for the fixes. I ran
CI is green on this head and it merges cleanly with main. |
…changed A turn that continues the last request under the same question picks from the last exact scores when they cover every recall candidate, and scores before the pass (stored question queries + the previous request's tail queries) when a candidate has none. Neither rewinds or resets the pool to recency. A new question, another conversation or a rewritten scored chunk takes the unchanged two-pass path. LUCE_KVFLASH_M4_SINGLE=0 forces the two-pass path on every turn. Old M4 on 14 LIVECODE sessions (2026-11-10, lucebox4): 810 reselects, scorer 512 s, 750779 tokens prefilled twice, prefill 0.504x of 1.0 recency.
…t N blocks of SELECT_UNIT chunks The recall budget is 10 x RECALL_BLOCKS chunks (qwen35_backend.cpp in.recall_chunks), picked in SELECT_UNIT-sized units (kvflash_prefill_reselect.h k_units = recall_chunks / unit). The row said N blocks of SELECT_UNIT chunks, i.e. 12 chunks at the defaults instead of 60. Review: Luce-Org#963 thread T5 (cubic)
…inside the pool Production takes the preamble reserve out of --kvflash N (pool stays N). The fixture added it on top (256 + 96) and asserted the feature-off tail, so it encoded the old design. It now uses a 256-block pool and asserts the reduced tail (95 = off - 96); it fails if the tail formula ignores the reserve. Review: Luce-Org#963 thread T7 (cubic)
…or parallelism kvflash_q_cap_all_rows() keys off LUCE_KVFLASH only, so a tp_degree > 1 process reserves the 64-row capture although recall is off there. Threading the resolved plan through create_target_cache_partial (6 callers across layer-split, shard daemon, drafter) and the VRAM estimate exceeds the small-fix cap, so the comment now states the deliberate reservation and its size. Review: Luce-Org#963 thread T10 (cubic)
Defect: the m4 tail capture was gated on !rw_done, so on a rewound turn kvflash_q_tail_ kept pass 1's queries (old pool) and pass 2, which recomputes [R, end) under the new pool, captured nothing. The stale tail was stored in kvflash_m4_last_.q_tail and scored by the next turn's Prescore. Fix: capture in both passes (pass 1's tail is still what the rewind scores with), and when kvflash_rewind_turn returns rc > 0 drop the tail rows at positions >= R (kvflash_q_tail_trim) so pass 2 refills them. The append and trim logic are pure helpers in kvflash_query_capture.h. Tests: test_kvflash_uq.cpp q_tail_* cases (trim at R, ragged tail, pass 1 below R followed by pass 2 rows). Red: the helpers did not exist (compile failure, red-item2-compile.log); green: 1377 passed, 0 failed. Not run on hardware.
Defect: q_cap_all is allocated in every KVFlash process, and the graph copied into it whenever in.q_capture was set. The qk policy keeps capture_qk on for decode and restore steps, so every decode step copied Q into a tensor nothing reads there. Under tensor parallelism that tensor has no split (qwen35_tensor_parallel.cpp classifies only q_cap) while the Q it copies from is split by head; the view-of-view copy into q_cap_all is expected to trip the split_dim_view_src assert in ggml-backend-meta.cpp on the first decode (derived from reading, not run on hardware). Fix: separate graph input QwenGraphInputs::q_capture_all, set from the pure kvflash_capture_q_all(recall_on, uq_on, tp_degree, capture_qk). Only the prefill windows that feed kvflash_read_captured_queries (recall mode 4 tail, UQ window) pass it; decode and restore steps and every other caller keep the default false. The rewind-only capture (rw_capture) reads q_cap, so it does not set it. TP zeroes recall and UQ in the feature plan, so it never writes. Tests: test_kvflash_exact_mass.cpp q_cap_all_is_written_only_for_recall_or_uq_prefill. Red: helper missing (compile failure, red-item1-compile.log); green: 1378 passed, 0 failed.
…he startup line Defect: --help still said --kvflash-policy defaults to drafter, but the recall profile makes it qk on qwen35 with --kvflash. The startup line chose "mode 4" from kvflash_mode5_requested() alone, while apply_env_defaults keeps an already-set LUCE_KVFLASH_PREFILL_RESELECT, so the line could say mode 4 while 0, 4 or 5 (without its GGUFs) ran; on other architectures the profile is not applied at all but the line still said mode 4. Fix: kvflash_recall_banner(arch, mode5, get) prints the resolved mode (the operator's env value when set, else the profile's), says when the environment set it, and says the profile is not applied off qwen35. Help text states the real default. Tests: test_launch_policy.cpp test_kvflash_banner_prints_the_resolved_reselect_mode (test_feature_gate target). Red: helper missing (compile failure, red-item4b-compile.log); green: test_feature_gate 5 passed, 0 failed (green-item4-gate.log).
Defect: kvflash_read_captured_queries copied the whole q_cap_all tensor (64 rows x every plane, about 24 MiB on the 27B) with one ggml_backend_tensor_get per captured batch, then used only n_q rows per plane. The tensor is [hd, T, n_head, n_fa]: the n_q rows of one plane are contiguous but planes are T*hd floats apart, so a plain prefix read is not enough. Fix: one ggml_backend_tensor_get_2d reading n_q*hd floats from each plane (kvflash_q_cap_all_read gives the arguments), then kvflash_regroup_queries turns [plane][n_q][hd] into the scorer's [t][plane][hd]. Output is identical to the old loop; bytes read drop from planes*T*hd*4 to planes*n_q*hd*4. Tests: test_kvflash_exact_mass.cpp captured_queries_are_read_as_n_q_rows_per_plane_and_regrouped (n_q 1, 5, 64; element-wise against the tensor, byte count asserted). Red: helpers missing (compile failure, red-item4c-compile.log); green: 1379 passed, 0 failed. The real tensor_get_2d path was not run on a GPU.
…ayed Mode-4 recall rewinds to R, the start of the last user turn, and skipped the recall when nothing was sealed before R (a cold single turn starts its user turn in chunk 0) or when the turn was longer than the pool can replay (PoolTooSmall). That turn then ran LRU only, and a fact the pool dropped was gone at decode: one cold 81,430-token turn, pool 16384, answered key_123 wrong. kvflash_max_turn_tokens is the longest turn the recall gate replays (its PoolTooSmall bound). kvflash_mode4_plan takes it as an optional cap and rewinds a longer turn to that tail, so the recall scores and pages in everything before it. The serial call site (kvflash_recall_setup) passes the cap; the gate is unchanged and a turn within the cap keeps its R. A capped R that lands inside the question window (LUCE_KVFLASH_UQ_QUERIES) moves to the window end (kvflash_uq_snap_rewind): a replay from inside it would leave the stored question rows from the old pool. Tests: test_kvflash_prefill_reselect, the cap equals the gate's bound and the capped plan on the needle's shape rewinds to the tail; test_kvflash_uq, the snap.
Mode 5 (the PFlash drafter's scores as the recall scorer) is dropped; mode 4 is the only recall mode. kvflash_reselect_mode returns 4 or 0, the mode-5 branches of the recall setup and the drafter scorer glue in the Qwen35 backend are gone, and so is the device victim tier that only mode 5 used (kvflash_victim_blocks_default, kvflash_pool_density_factor and their test). The ENVIRONMENT.md rows and the pflash-map gate are back to the mode-4 text.
/props gains kvflash {enabled, pool_tokens, mode}. server_main fills it after the --kvflash, --kvflash-policy
and --kvflash-tau flags have set the environment the backend reads, so env-only KVFlash counts too.
pool_tokens is LUCE_KVFLASH (0 means auto, sized at load). mode is the backend's own kvflash_reselect_mode
over LUCE_KVFLASH_PREFILL_RESELECT, LUCE_KVFLASH_POLICY and LUCE_KVFLASH_RESELECT_SCORER:
"prefill-reselect=4" while recall runs, empty on another architecture, with --kvflash-policy lru or with a
scorer other than exact.
a21ae48 to
4f2de18
Compare
test_server_unit.cpp: main's Luce-Org#1031 pruned the model-card props tests next to the three /props.kvflash tests; take main's version and keep the three tests.
Builds on #934 (merged). Base: main d972beb. Review range: d972beb..4f2de18, 31 commits, 32 files, +2668 / -60. The merge commit
2a01709b0brings in main1d949fd88. Its only conflict wastest_server_unit.cpp, where #1031 pruned the model-card/propstests next to the three/props.kvflashtests. It takes main's version of that file and keeps the three tests, and it adds no code of its own.This head replaces the earlier head
a21ae4883(51 commits, +3955 / -190) with three changes:LUCE_KVFLASH_M5_RESCORE_TOKENS), the device victim tier (LUCE_KVFLASH_VICTIM_BLOCKS), and the drafter-side PFlash changes only mode 5 needed (1024-token session resume, the device strict-scorer reduce,PFLASH_TW_HOST_REDUCE). Mode 5 failed this PR's gate (P2 19/40 < 22), and on the 2026-11-10 needle replay it kept the needle resident in 0/10 agent sessions against 10/10 for mode 4. PFlash without--kvflashis no longer touched by this PR./props.kvflash {enabled, pool_tokens, mode}, see "/props.kvflash".Mode 4, the only recall mode and the one every result below measures, behaves the same as before. On 3 recorded sessions (165 requests, lucebox6 R9700), the earlier head and this head give the same text, tool calls, cached tokens and recall picks on every request, at the same session prefill (640 vs 640 tok/s) and decode (30.1 vs 30.1 tok/s); see "This head against the earlier head". The 14-session gate and the other engine results below ran on the earlier head.
The landing gate passed on the earlier head against both baselines, on 14 recorded coding sessions: mode 4 (the
--kvflashdefault) against K1 (recency pool with this PR's preamble pin) and against K0 (KVFlash 1.0 as shipped without a drafter: recency pool, no pin). P1 31/40 against 0/40 (K1) and 1/40 (K0); P2 26/40 against 24/40 and 19/40; session prefill 0.860x and 0.859x (bar 0.85x). Before the mode-4 cost fix it failed this gate (prefill 0.504x, P2 one probe short).Why
With
--kvflash Non main and no Qwen3.5-0.8B drafter next to the model, the KV cache is a recency pool (policy=lru) of 64-token chunks that keeps only the newest chunks. With a drafter present main picks chunks by drafter scores. Every K arm below ran with no drafter (server log:policy=lru (recency-only: no Qwen3.5-0.8B drafter found ...)). A coding agent that asks about a file it read 60K tokens ago gets a pool without that file. On 14 recorded coding sessions, the far-back recall probe (P1) scored 0/40 on a recency pool with the preamble pin (K1, below).This PR makes the single-request path bring back the old chunks the new question needs, before it writes the question into the pool. It is the first caller of the #934 scorers. Nothing concurrent is in it: no arena and no slot pagers.
The decode-mask commit
3047a9ff2is required by recall: a rewind leaves rows above the decode position in the pool.What it does
Turn by turn, on a coding session with a 16K pool (
--kvflash 16384, mode 4):parse_configignore the second section?" The prompt is the old 60K plus about 400 new tokens.LUCE_KVFLASH_RECALL_BLOCKS) are picked, ties to the newer block. The pinned preamble is never a candidate.Steps 4 to 6 cost one extra forward over the last turn plus one scorer call. The mode-4 cost fix (below) skips them for tool-loop turns under an unchanged question.
Files:
kvflash_prefill_reselect.h: the mode gate (0 and 4; the retired modes 1, 2, 3 and 5 are off), the turn cut,kvflash_recall_pick,kvflash_recall_plan(the budget), the preamble pin helpers, andkvflash_recall_gate. The gate runs a request as a plain recency pool when no sealed chunk precedes the turn, when the chunks up to the cut plus the turn's reserve already fit the pool, or when the pool cannot hold pins + recall + reserve.kvflash_m4_reuse.h(last commit): the decision for the single-pass turn, as pure functions.--kvflashinstalls it; the startup line saysrecall profile: mode 4 (exact scorer), or names aLUCE_KVFLASH_PREFILL_RESELECTthe environment set.LUCE_KVFLASH_UQ_QUERIES).LUCE_KVFLASH_PREAMBLE_ROWS, 1152 rows = 18 chunks): the system+tools chunks are pinned, and re-pinned after reset, restore and truncate. The reserve comes out of--kvflash N, so the pool and the decode cost stay at N and the history budget is N - 1152.reselect_recent(); strided page copies and block zeroing (one 2D call per layer and K/V); a chunk recalled from host and not written since skips its D2H copy; page-outs leave the freed block unzeroed; one fence per reselect; victim FIFO compaction; pooled ubatch clamped to the unpinned capacity.TargetCache::q_cap_allholds the lastkQueryCaptureTokens = 64queries of a batch. It exists only in a process with aLUCE_KVFLASHpool, and a graph writes it only on prefill windows that recall mode 4 or UQ read (kvflash_capture_q_all); decode and restore steps never write it. The reader fetches only the captured rows. After a mode-4 rewind the query tail is trimmed at the rewind point and re-captured by the second pass.kvflash_recall_features(tp_degree, ask)turns recall and UQ off at TP degree > 1, with the line[kvflash] recall disabled: tensor parallelism has no split for the per-token query capture. Non-qwen35 targets get no recall profile fromkvflash_recall_env. No tensor-parallel run was made.kvflash_upload_mask(cur_pos)bounds the slot mask tocur_pos + 1and rebuilds it every decode step (commita261eb59e). It hides rows above the decode position (rejected drafts, an earlier turn's tokens after a rewind) and changes AR numerics for KVFlash decode. It applies only when a KVFlash pool is active.With
--kvflashoff the KVFlash paths are not entered: noq_cap_all, no recall branch, no profile env. PFlash is unchanged.Mode-4 cost fix (
160a786b2)Cause, measured on the 882-request replay of the 14 sessions on the earlier build
787e9f89b(lucebox4, R9700): 810 requests reselected. Mode 4 paid a scorer call (512 s total, mean 632 ms, of which 241 s host-to-device copies) and a rewind that re-prefilled 750,779 tokens (51% of all 1,470,012 new tokens) on each. Prefill was 3609 s against K1's 1821 s, a gap of 1788 s; 1776 s of that gap was on the 810 reselect requests. Skipping the rewind when the picks equal the resident set saves nothing: across 700 consecutive reselect pairs under one question, 0 had identical picks (mean overlap 0.55).A request now pays the scorer and the rewind only when its latest human question is new. Tool-loop turns under the same question are the bulk of the 810.
LUCE_KVFLASH_M4_SINGLE=0forces the two-pass path on every turn.On same-question turns the picks can differ from the two-pass picks, because the tail queries can be up to one scoring old (at most one tail window). On the gate replay this cost no recall: P1 31/40 and P2 26/40 with 637 of 882 requests on reuse, against 26/40 and 21/40 for the two-pass mode 4 on the earlier head (other binary, see below).
Measured on the gate (below): 637 requests reuse, 80 prescore, 93 take the two-pass path, 72 skip (pool fits or too small); scorer total 106 s (was 512 s), re-prefill 63,332 tokens (was 750,779); session prefill 0.860x of K1. The simulator estimate before the run was 640 / up to 82 / 88 and 0.86x to 0.90x.
Tests
test_kvflash_m4_reusehas 11 cases, includingan_unscored_chunk_left_of_the_window_never_reusesandreuse_pick_ignores_whatever_the_window_chunks_would_score. The test file and the code are in the same commit; the test failed to build before the header existed (kvflash_m4_reuse.h: No such file or directory). Other test commits come before the code they cover; those that failed against their parent say so in the subject (red: ...).All runs below: RTX 3090 (sm_86, WSL), head
50020e4c5,ctest -j1 --timeout 120. Baseline: maindd85248d3on the same 3090 (1515 tests, 7 failed;cuda3090/main-ctest.log). Maind972beba3was not run as a full ctest.test_server_unit KvflashM4Reusetest_server_unit Kvflashctestd972beba3is pending. None is a KVFlash test.ctestwithCUDA_VISIBLE_DEVICES=""2f0b18313: Linux build, Windows MSVC + CUDA, Windows ROCm HIP; GPU tests on DGX GB10 (sm_121), R9700 (gfx1201), Strix Halo (gfx1151)a21ae4883(run 43282854502): Linux build, Windows MSVC + CUDA, Windows ROCm HIP, DGX GB10 and Strix Halo pass. The R9700 job failed: the runner got a shutdown signal mid-build (no test ran) and needs a re-run.Engine run
Arms. K1 =
LUCE_KVFLASH_POLICY=lruwith this PR's preamble pin on (its server log showspreamble pin end=). K1 is not KVFlash 1.0 as shipped. K0 =LUCE_KVFLASH_POLICY=lru LUCE_KVFLASH_PREAMBLE_ROWS=0(pin off, the 1.0 pool), run on this head (below). M4 = mode 4 (--kvflashdefault).This head against the earlier head (identity)
lucebox6 R9700 (gfx1201), 2026-11-18, Qwen3.8-27B UD-IQ4_XS (
/opt/models, the gate's model). HIP builds (gfx1151;gfx1201) of the earlier heada21ae4883(luce_servermd5 78bfef46) and this head4f2de18da(md5 269cdd9a). Flags on every arm, no env:--kvflash 16384 --max-ctx 139264 --cache-type-k q8_0 --cache-type-v q8_0 --prefix-cache-slots 8 --prefix-cache-max-mib 40960 --disk-prefix-cache off --no-kv-cache --target-device hip:0. 3 of the 14 recorded sessions (165 requests, P1 and P2 probes included), temperature 0, seed 1, thinking off. Arms ran in the order old, lean, lean, old, one server at a time, page cache dropped before each.a21ae4883)a21ae4883)Same output = same text (sha256) and the same tool-call names and arguments; prompt and cached tokens are also equal on 165/165. The 657
[kvflash]selection lines (153 recall picks of 30 blocks) are identical across all four arms once timings are masked and the earlier head's(0 tier hits)suffix is dropped: the device victim tier never hit in mode 4 (186 lines, all 0). Every arm logsrecall profile: mode 4. old1 shared the GPUs with a unit-test run for about 30 s near its start./propson this head:"kvflash": {"enabled": true, "pool_tokens": 16384, "mode": "prefill-reselect=4"}.HIP unit tests, same builds,
ctest -R "[Kk]vflash|[Ll]aunch|[Pp]rops|[Ff]eature.?[Gg]ate": this head 284/284 on gfx1201 (R9700) and 284/284 on gfx1151 (Strix Halo); the earlier head 299/299 on both (it still has the mode-5, victim-tier and probe tests, and not the 3/propstests). Every commit of this head builds (CPU, CUDA toolchain).Landing gate on the earlier head (failing)
lucebox4 R9700 (PCIe 3.0 x2), 2026-11-10, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), 14 recorded sessions, 40 P1 and 40 P2 probes, 882 requests per arm. Binary
luce_servermd5 a2f83fbd built from787e9f89b(pre-rebase). It differs from2f0b18313only in the--kvflashdefault-mode selection (launch_profiles.h,server_main.cpp); the K1 and M4 arms have no drafter and are unaffected.Parity:
--kvflash 16384,--max-ctx 139264, q8_0 K and V, prefix cache on (40960 MiB, 8 slots, disk off), thinking on, T 1.0, top_p 0.95, top_k 20, effort medium, probe max_tokens 16384, same sessions on every arm. Caching state: M4'sprefix_lenhistogram differs from K1's on 50 of 882 requests (total 4,106,452 against 4,106,602 tokens). No power limit was set or recorded. Session prefill = total new tokens / total prefill seconds. Wall = sum of client turn seconds.Mode 4: P1 passes (>= K1 + 3), P2 fails (21 < 22), prefill fails (0.504x < 0.85x). This table is the cost the fix above targets; it is not a result for this head. P1 is 0/40 on K1 and 26/40 on mode 4 on the same box and binary. K0 did not run on that head; it ran on this one (below).
Re-gate on this head (passing)
lucebox4 R9700 (PCIe 3.0 x2), 2026-11-11, head
50020e4c5, oneluce_server(md5 ecbe4373) for both arms, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), the same 14 sessions (882 requests, 40 P1 and 40 P2 probes per arm). Server flags on both arms:--max-ctx 139264 --prefix-cache-max-mib 40960 --prefix-cache-slots 8 --disk-prefix-cache off --no-kv-cache --cache-type-k q8_0 --cache-type-v q8_0 --kvflash 16384. Runner: thinking on, effort medium, T 1.0, top_p 0.95, top_k 20, probe max_tokens 16384. Arms differ only in env: K1LUCE_KVFLASH_POLICY=lru; K0LUCE_KVFLASH_POLICY=lru LUCE_KVFLASH_PREAMBLE_ROWS=0; M4 none. The K1 and M4 server logs show the preamble pin (851 lines each), K0's none. Caching state: prompt tokens 62,012,284 on every arm, cached prefix 60,542,275 (K0, K1) vs 60,542,272 (M4), equal on 881 of 882 requests against each. No power limit set or recorded. Gates were fixed before any run (m4cost_gate.sh).Gate lines, M4 vs K1: prefill >= 0.85x PASS (0.860); P1 >= K1 + 3 PASS (31 vs 0); P2 >= K1 - 2 PASS (26 vs 24). M4 vs K0: prefill >= 0.85x PASS (0.859); P1 >= K0 + 3 PASS (31 vs 1); P2 >= K0 - 2 PASS (26 vs 19). Wall is not engine speed here: it includes the probes' answers, and K0's probes generated 63,116 output tokens against 37,166 (K1) and 39,113 (M4). Mode-4 paths: reused 637, prescore 80, full 93, skip-fits 21, skip-pool 20, none 31; scorer 106.3 s, paging 28.0 s, re-prefill 63,332 tokens.
Screen before the gate (3 of the sessions, 151 requests, same box): this head vs its parent
a90f8f1de(md5 33daf153): session prefill 737 vs 443 tok/s (0.883x vs 0.530x of K1's 835), scorer 23.5 s vs 96.0 s, P1 9/9 on both. WithLUCE_KVFLASH_M4_SINGLE=0the head's picks equal the parent's on 25 of 25 requests (the two-pass path is unchanged).Versus main (same box, serial, speed and text)
lucebox6 R9700 (gfx1201), ROCm 7.2.4, 2026-11-11, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), q8_0 K/V,
--max-ctx 131072 --prefix-cache-slots 0 --no-kv-cache --kvflash 16384, temperature 0, no draft model in either arm. Serial engine; 3 conversations, turn 1 21585-26509 and turn 2 21758-26682 prompt tokens,prefix_len=0on both sides (full reprocess, no caching). Two server starts per pin, interleaved; n = 6 per pin and turn. maind972beba3(luce_servermd5 f2f9ae3f) against50020e4c5(md5 b38275f8; it differs from this head only inENVIRONMENT.md, a comment and one unit test).The text differs from main on all 6 requests, and each side repeats its own text on both starts: main logs
policy=drafter (attaches on first reselect)and, with no drafter, keeps the newest chunks; this PR logsrecall profile: mode 4/policy=qkand picks chunks by score. This run has no recall probes; recall against KVFlash 1.0 is the K0 row of the gate above.Versus full KV (no KVFlash)
These compare against no KVFlash, not against KVFlash 1.0, and none was measured on this head. They also ran with the preamble reserve outside
--kvflash Non earlier binaries, so the pool size differs from this head (see the LongMemEval-S line).E20 (20 live coding-agent tasks), lucebox4, 2026-11-10,
luce_servermd5 6ae4a114 (commite964d21a7; not the binary of the gate table above, which is a2f83fbd /787e9f89b;--kvflashthere installs mode 4). Server flags:--max-ctx 135168 --prefix-cache-max-mib 12288 --prefix-cache-slots 4 --disk-prefix-cache off --cache-type-k q8_0 --cache-type-v q8_0 --no-kv-cache,--kvflash 16384on the K arms. Runner: thinking on, effort medium, T 1.0, top_p 0.95, top_k 20, seed 1, max_tokens 16384, same 20 tasks on every arm.Full KV is at ceiling on this set (it fails the pre-registered 25-85% band), so the set measures no regression against full KV and speed, not recall. It cannot show a recall edge over recency: mode 4 and K1L tie. Mode 4 session prefill is 0.80x of K1L and wall +7%.
EXECODE A6 (2026-10-27, lucebox6 R9700, binary md5 0ad28447,
--max-ctx 135168, q8_0 K/V, 20 paired tasks, pre-registered): 19/20 against 20/20 for full KV, difference -0.05, bootstrap CI [-0.15, 0.0], exact McNemar p = 1.000, parity PASS (frozen margin d <= 2). Medians: decode 19.2 to 27.6 tok/s, request wall 23.0 to 12.9 s, total wall 7,062.5 to 5,135.6 s, first-request prefill (>50K new tokens) 566.2 to 749.6 tok/s.LongMemEval-S (binary md5 cec754b6, thinking off, T 0, seed 1, max_tokens 512, judge gpt-4o-2024-08-06, n = 500, one request with the whole history):
--kvflash 16384against full context is a measured loss, -8.0 points [-11.4, -4.6] with official labels, -8.8 [-12.0, -5.8] with audited labels (pre-registered verdict: accuracy loss). That binary reserved 6144 extra preamble rows on top of the pool; this head keeps 1152 inside N. Neither A6 nor LongMemEval-S bears on this head's pool size.Long cold turn (
724119d03, wasa21ae4883)With the
--kvflashprofile, a single cold turn longer than the pool skipped the recall: its user turn starts in chunk 0, so nothing is sealed before R, and the turn is longer than the recall gate'sPoolTooSmallbound. The turn then ran LRU only, and a fact the pool dropped from the middle of the prompt was gone at decode.kvflash_max_turn_tokensis that bound.kvflash_mode4_plantakes it as an optional cap and rewinds a longer turn to its tail, so the recall scores and pages in everything before it. The gate is unchanged and a turn within the cap keeps its R. A capped R can land inside the latest question's 64-query window, where a replay would recompute the window's tail while the stored question rows keep the old pool's values;kvflash_uq_snap_rewindmoves such an R to the window end.Serial needle on the RTX 3090: Qwen3.8-27B IQ4_XS, DFlash2 drafter,
--max-ctx 131072,--kvflash 16384, q8_0 K and V, temperature 0, the shipped--kvflashprofile with nothing else set, one cold 81,430-token turn of 3,500 records ("record i: the code for key_i is ..."), question at the end, cached tokens 0, one run per head.d98512ff0prefill reselect mode=4: skip, no sealed chunk before R (LRU)a21ae4883prefill reselect mode=4: R4=70230 (end 81430, re-prefill 11200 tokens)The 13 s difference is the scoring and the 11,200-token replay. One key, one run per head.
Second shape, the three-turn serial smoke (
--kvflash 16384, same model and flags; turn 1 is a cold 50,408-token request with two user messages, the last one longer than the cap; one run per head):d98512ff043ebed574a21ae4883R4=39208, re-prefill 11200)What ran on the commits after
d98512ff0(RTX 3090):test_server_uniton CPU: 1382 passed, 0 failed, 17 skipped ona21ae4883(1374/0/17 ond98512ff0).test_server_uniton the GPU: 1390/0/6 on43ebed574(1385/0/6 ond98512ff0).43ebed574andd98512ff0.Not run on them: ctest, the landing gate, ROCm, tensor parallelism.
/props.kvflash (was #999)
/propsgainskvflash {enabled, pool_tokens, mode}, so a client can see whether recall is on without reading the server log. server_main fills it after the--kvflash*flags have set the environment the backend reads, so env-onlyLUCE_KVFLASHcounts too.pool_tokensisLUCE_KVFLASH(0 =auto, sized at load).modecomes from the backend's ownkvflash_reselect_modeoverLUCE_KVFLASH_PREFILL_RESELECT,LUCE_KVFLASH_POLICYandLUCE_KVFLASH_RESELECT_SCORER:prefill-reselect=4while recall runs, empty on another architecture, with--kvflash-policy lruor with a scorer other thanexact. It does not see tensor parallelism, which turns recall off later at load. #999's nestedrecall {protocol, active, mode}object and the--helpmarker are not carried: no client reads them.Env knobs
--kvflashon a qwen35 target installs mode 4:LUCE_KVFLASH_POLICY=qk,RECALL_BLOCKS=6,SCORE_POOL_RECENT=1,SELECT_UNIT=2,PREAMBLE_ROWS=1152,PREFILL_RESELECT=4,RESELECT_SCORER=exact,UQ_QUERIES=64. A variable that is already set keeps its value. The values are the named constants ofKvflashRecallProfileinlaunch_profiles.h. Other architectures install nothing.test_launch_policy.cpppins the full (name, value) set of both profiles.New knobs documented in
server/docs/ENVIRONMENT.md: the recall knobs listed there.LUCE_KVFLASH_M4_SINGLE(0forces the two-pass path) is read inqwen35_backend.cppand is not yet inENVIRONMENT.md.Limits
Not run on this head (they ran on the earlier head, whose mode 4 this head matches request for request on the 3-session replay):
dd85248d3, notd972beba3; the alone rerun ofq4k_rowsis pending (one alone attempt ofcommitted_tokens_match_ar_samplingis logged with rc=77 and no pass or fail count, so it is not counted). No CPU-only ctest baseline.Commit map (earlier head -> this head)
a261eb59e3047a9ff250020e4c5160a786b22a6e64c53b6d1bea00LUCE_KVFLASH_RECALL_BLOCKSis a 10 x N chunk budget9fc7a36a43a8a77357d98512ff005c4c3a31q_cap_allreservation under tensor parallelismd13a9185799cede4b689a1c9ffadbde568eaq_cap_allonly when recall or UQ is on68e5485a17d435c8e743ebed574fba9ef4a2q_cap_alla21ae4883724119d03efec5616a4f2de18da/props.kvflashreports whether recall is on