Skip to content

feat(qwen35): prefill recall with the exact scorer on the serial path, recall profile on by default - #963

Open
dusterbloom wants to merge 32 commits into
Luce-Org:mainfrom
dusterbloom:kvflash/prefill-recall
Open

dusterbloom wants to merge 32 commits into
Luce-Org:mainfrom
dusterbloom:kvflash/prefill-recall

Conversation

@dusterbloom

@dusterbloom dusterbloom commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #934 (merged). Base: main d972beb. Review range: d972beb..4f2de18, 31 commits, 32 files, +2668 / -60. The merge commit 2a01709b0 brings in main 1d949fd88. Its only conflict was test_server_unit.cpp, where #1031 pruned the model-card /props tests next to the three /props.kvflash tests. It takes main's version of that file and keeps the three tests, and it adds no code of its own.

This head replaces the earlier head a21ae4883 (51 commits, +3955 / -190) with three changes:

  • Mode 5 is out. Gone: the PFlash drafter as recall scorer, its rescore cut (LUCE_KVFLASH_M5_RESCORE_TOKENS), the device victim tier (LUCE_KVFLASH_VICTIM_BLOCKS), and the drafter-side PFlash changes only mode 5 needed (1024-token session resume, the device strict-scorer reduce, PFLASH_TW_HOST_REDUCE). Mode 5 failed this PR's gate (P2 19/40 < 22), and on the 2026-11-10 needle replay it kept the needle resident in 0/10 agent sessions against 10/10 for mode 4. PFlash without --kvflash is no longer touched by this PR.
  • feat(server): /props.kvflash advertises the recall capability (stacked on #963) #999 is folded in (last commit): /props.kvflash {enabled, pool_tokens, mode}, see "/props.kvflash".
  • The commits were replayed, so the SHAs changed. The map is at the end.

Mode 4, the only recall mode and the one every result below measures, behaves the same as before. On 3 recorded sessions (165 requests, lucebox6 R9700), the earlier head and this head give the same text, tool calls, cached tokens and recall picks on every request, at the same session prefill (640 vs 640 tok/s) and decode (30.1 vs 30.1 tok/s); see "This head against the earlier head". The 14-session gate and the other engine results below ran on the earlier head.

The landing gate passed on the earlier head against both baselines, on 14 recorded coding sessions: mode 4 (the --kvflash default) against K1 (recency pool with this PR's preamble pin) and against K0 (KVFlash 1.0 as shipped without a drafter: recency pool, no pin). P1 31/40 against 0/40 (K1) and 1/40 (K0); P2 26/40 against 24/40 and 19/40; session prefill 0.860x and 0.859x (bar 0.85x). Before the mode-4 cost fix it failed this gate (prefill 0.504x, P2 one probe short).

Why

With --kvflash N on main and no Qwen3.5-0.8B drafter next to the model, the KV cache is a recency pool (policy=lru) of 64-token chunks that keeps only the newest chunks. With a drafter present main picks chunks by drafter scores. Every K arm below ran with no drafter (server log: policy=lru (recency-only: no Qwen3.5-0.8B drafter found ...)). A coding agent that asks about a file it read 60K tokens ago gets a pool without that file. On 14 recorded coding sessions, the far-back recall probe (P1) scored 0/40 on a recency pool with the preamble pin (K1, below).

This PR makes the single-request path bring back the old chunks the new question needs, before it writes the question into the pool. It is the first caller of the #934 scorers. Nothing concurrent is in it: no arena and no slot pagers.

The decode-mask commit 3047a9ff2 is required by recall: a rewind leaves rows above the decode position in the pool.

What it does

Turn by turn, on a coding session with a 16K pool (--kvflash 16384, mode 4):

  1. Turns 1 to 20: the agent reads files and runs tests. The prompt grows to 60K tokens. The pool holds the pinned system+tools preamble (up to 1152 rows), the sink, and the newest chunks. Old file reads are evicted to host memory.
  2. Turn 21 arrives: "why does parse_config ignore the second section?" The prompt is the old 60K plus about 400 new tokens.
  3. The prefill is cut at the start of that last user turn. Everything before the cut is written as before.
  4. First pass: the last user turn runs on its own. The query of its final token is kept, with the last 64 queries of the prompt.
  5. Recall: those queries, plus the latest question's queries from the store, are scored against every sealed chunk with the exact attention mass (GPU scorer, AVX2 fallback). The best 30 blocks of 2 chunks (60 chunks, the budget 10 x LUCE_KVFLASH_RECALL_BLOCKS) are picked, ties to the newer block. The pinned preamble is never a candidate.
  6. The state is restored to the cut, the pager is truncated there and reselected: preamble, sink, the picked blocks, then the newest chunks. The picked chunks are paged in from host.
  7. The last user turn is written again under the new pool, and decoding starts.

Steps 4 to 6 cost one extra forward over the last turn plus one scorer call. The mode-4 cost fix (below) skips them for tool-loop turns under an unchanged question.

Files:

  • kvflash_prefill_reselect.h: the mode gate (0 and 4; the retired modes 1, 2, 3 and 5 are off), the turn cut, kvflash_recall_pick, kvflash_recall_plan (the budget), the preamble pin helpers, and kvflash_recall_gate. The gate runs a request as a plain recency pool when no sealed chunk precedes the turn, when the chunks up to the cut plus the turn's reserve already fit the pool, or when the pool cannot hold pins + recall + reserve.
  • kvflash_m4_reuse.h (last commit): the decision for the single-pass turn, as pure functions.
  • Mode 4 (exact scorer, rewind). --kvflash installs it; the startup line says recall profile: mode 4 (exact scorer), or names a LUCE_KVFLASH_PREFILL_RESELECT the environment set.
  • User-question queries (UQ): the latest human turn's last 64 query rows are kept in a per-session LRU and appended to the scorer's queries (LUCE_KVFLASH_UQ_QUERIES).
  • Preamble pin (LUCE_KVFLASH_PREAMBLE_ROWS, 1152 rows = 18 chunks): the system+tools chunks are pinned, and re-pinned after reset, restore and truncate. The reserve comes out of --kvflash N, so the pool and the decode cost stay at N and the history budget is N - 1152.
  • Pager: reselect_recent(); strided page copies and block zeroing (one 2D call per layer and K/V); a chunk recalled from host and not written since skips its D2H copy; page-outs leave the freed block unzeroed; one fence per reselect; victim FIFO compaction; pooled ubatch clamped to the unpinned capacity.
  • Graph: TargetCache::q_cap_all holds the last kQueryCaptureTokens = 64 queries of a batch. It exists only in a process with a LUCE_KVFLASH pool, and a graph writes it only on prefill windows that recall mode 4 or UQ read (kvflash_capture_q_all); decode and restore steps never write it. The reader fetches only the captured rows. After a mode-4 rewind the query tail is trimmed at the rewind point and re-captured by the second pass.
  • Tensor parallelism: kvflash_recall_features(tp_degree, ask) turns recall and UQ off at TP degree > 1, with the line [kvflash] recall disabled: tensor parallelism has no split for the per-token query capture. Non-qwen35 targets get no recall profile from kvflash_recall_env. No tensor-parallel run was made.
  • Decode mask: kvflash_upload_mask(cur_pos) bounds the slot mask to cur_pos + 1 and rebuilds it every decode step (commit a261eb59e). It hides rows above the decode position (rejected drafts, an earlier turn's tokens after a rewind) and changes AR numerics for KVFlash decode. It applies only when a KVFlash pool is active.

With --kvflash off the KVFlash paths are not entered: no q_cap_all, no recall branch, no profile env. PFlash is unchanged.

Mode-4 cost fix (160a786b2)

Cause, measured on the 882-request replay of the 14 sessions on the earlier build 787e9f89b (lucebox4, R9700): 810 requests reselected. Mode 4 paid a scorer call (512 s total, mean 632 ms, of which 241 s host-to-device copies) and a rewind that re-prefilled 750,779 tokens (51% of all 1,470,012 new tokens) on each. Prefill was 3609 s against K1's 1821 s, a gap of 1788 s; 1776 s of that gap was on the 810 reselect requests. Skipping the rewind when the picks equal the resident set saves nothing: across 700 consecutive reselect pairs under one question, 0 had identical picks (mean overlap 0.55).

A request now pays the scorer and the rewind only when its latest human question is new. Tool-loop turns under the same question are the bulk of the 810.

  • Reuse: every recall candidate (chunk left of the tail window) has an exact score from the last scoring and no scored chunk was rewritten. The turn picks again from those scores. No scorer, no rewind, no recency reset.
  • Prescore: a candidate has no score. The exact scorer runs before the pass with the question's stored queries plus the tail queries the previous request left, then picks. No rewind.
  • A new question, another conversation, or a rewritten scored chunk takes the unchanged two-pass path. LUCE_KVFLASH_M4_SINGLE=0 forces the two-pass path on every turn.
  • No candidate is ranked by a score the scorer did not compute for it: an unscored candidate forces a scoring.

On same-question turns the picks can differ from the two-pass picks, because the tail queries can be up to one scoring old (at most one tail window). On the gate replay this cost no recall: P1 31/40 and P2 26/40 with 637 of 882 requests on reuse, against 26/40 and 21/40 for the two-pass mode 4 on the earlier head (other binary, see below).

Measured on the gate (below): 637 requests reuse, 80 prescore, 93 take the two-pass path, 72 skip (pool fits or too small); scorer total 106 s (was 512 s), re-prefill 63,332 tokens (was 750,779); session prefill 0.860x of K1. The simulator estimate before the run was 640 / up to 82 / 88 and 0.86x to 0.90x.

Tests

test_kvflash_m4_reuse has 11 cases, including an_unscored_chunk_left_of_the_window_never_reuses and reuse_pick_ignores_whatever_the_window_chunks_would_score. The test file and the code are in the same commit; the test failed to build before the header existed (kvflash_m4_reuse.h: No such file or directory). Other test commits come before the code they cover; those that failed against their parent say so in the subject (red: ...).

All runs below: RTX 3090 (sm_86, WSL), head 50020e4c5, ctest -j1 --timeout 120. Baseline: main dd85248d3 on the same 3090 (1515 tests, 7 failed; cuda3090/main-ctest.log). Main d972beba3 was not run as a full ctest.

Check Result
test_server_unit KvflashM4Reuse 11 passed, 0 failed
test_server_unit Kvflash 280 passed, 0 failed
full ctest 7 failed of 1707. 6 of the 7 also fail on the main baseline: SampledVerifyFixture.constrained_tokens_match_constrained_ar_sampling, SampledVerifyFixture.committed_tokens_match_ar_sampling and SampledVerifyFixture.redrawing_a_drawn_row_is_caught (all three time out at 120 s on main too), test_ds41_mmid_width_invariance, qwen4exp_prefix_cache, server_unit_entrypoint. Only QnSplitFusionsFixture.q4k_rows (timeout at 120 s, box shared) is outside the baseline; its rerun alone on this head and on d972beba3 is pending. None is a KVFlash test.
full ctest with CUDA_VISIBLE_DEVICES="" 18 failed of 1707 (GPU tests without a GPU). No CPU-only baseline exists.
CI on the earlier head 2f0b18313: Linux build, Windows MSVC + CUDA, Windows ROCm HIP; GPU tests on DGX GB10 (sm_121), R9700 (gfx1201), Strix Halo (gfx1151) pass. On this head a21ae4883 (run 43282854502): Linux build, Windows MSVC + CUDA, Windows ROCm HIP, DGX GB10 and Strix Halo pass. The R9700 job failed: the runner got a shutdown signal mid-build (no test ran) and needs a re-run.

Engine run

Arms. K1 = LUCE_KVFLASH_POLICY=lru with this PR's preamble pin on (its server log shows preamble pin end=). K1 is not KVFlash 1.0 as shipped. K0 = LUCE_KVFLASH_POLICY=lru LUCE_KVFLASH_PREAMBLE_ROWS=0 (pin off, the 1.0 pool), run on this head (below). M4 = mode 4 (--kvflash default).

This head against the earlier head (identity)

lucebox6 R9700 (gfx1201), 2026-11-18, Qwen3.8-27B UD-IQ4_XS (/opt/models, the gate's model). HIP builds (gfx1151;gfx1201) of the earlier head a21ae4883 (luce_server md5 78bfef46) and this head 4f2de18da (md5 269cdd9a). Flags on every arm, no env: --kvflash 16384 --max-ctx 139264 --cache-type-k q8_0 --cache-type-v q8_0 --prefix-cache-slots 8 --prefix-cache-max-mib 40960 --disk-prefix-cache off --no-kv-cache --target-device hip:0. 3 of the 14 recorded sessions (165 requests, P1 and P2 probes included), temperature 0, seed 1, thinking off. Arms ran in the order old, lean, lean, old, one server at a time, page cache dropped before each.

Arm Same output as old1 Session prefill Decode med / mean / best / min (n 10, decode > 2 s) Wall
old1 (a21ae4883) - 641 tok/s 30.05 / 30.04 / 30.20 / 29.90 632.8 s
lean1 (this head) 165/165 640 tok/s 30.10 / 30.06 / 30.20 / 29.90 633.7 s
lean2 (this head) 165/165 639 tok/s 30.10 / 30.05 / 30.20 / 29.80 634.1 s
old2 (a21ae4883) 165/165 640 tok/s 30.10 / 30.06 / 30.20 / 29.90 633.6 s

Same output = same text (sha256) and the same tool-call names and arguments; prompt and cached tokens are also equal on 165/165. The 657 [kvflash] selection lines (153 recall picks of 30 blocks) are identical across all four arms once timings are masked and the earlier head's (0 tier hits) suffix is dropped: the device victim tier never hit in mode 4 (186 lines, all 0). Every arm logs recall profile: mode 4. old1 shared the GPUs with a unit-test run for about 30 s near its start. /props on this head: "kvflash": {"enabled": true, "pool_tokens": 16384, "mode": "prefill-reselect=4"}.

HIP unit tests, same builds, ctest -R "[Kk]vflash|[Ll]aunch|[Pp]rops|[Ff]eature.?[Gg]ate": this head 284/284 on gfx1201 (R9700) and 284/284 on gfx1151 (Strix Halo); the earlier head 299/299 on both (it still has the mode-5, victim-tier and probe tests, and not the 3 /props tests). Every commit of this head builds (CPU, CUDA toolchain).

Landing gate on the earlier head (failing)

lucebox4 R9700 (PCIe 3.0 x2), 2026-11-10, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), 14 recorded sessions, 40 P1 and 40 P2 probes, 882 requests per arm. Binary luce_server md5 a2f83fbd built from 787e9f89b (pre-rebase). It differs from 2f0b18313 only in the --kvflash default-mode selection (launch_profiles.h, server_main.cpp); the K1 and M4 arms have no drafter and are unaffected.

Parity: --kvflash 16384, --max-ctx 139264, q8_0 K and V, prefix cache on (40960 MiB, 8 slots, disk off), thinking on, T 1.0, top_p 0.95, top_k 20, effort medium, probe max_tokens 16384, same sessions on every arm. Caching state: M4's prefix_len histogram differs from K1's on 50 of 882 requests (total 4,106,452 against 4,106,602 tokens). No power limit was set or recorded. Session prefill = total new tokens / total prefill seconds. Wall = sum of client turn seconds.

Arm P1 P2 Session prefill Decode median (out >= 64) Wall
K1 (lru + pin) 0/40 24/40 807 tok/s 29.0 tok/s 3312 s
Mode 4 26/40 21/40 407 tok/s (0.504x) 28.8 tok/s 5791 s

Mode 4: P1 passes (>= K1 + 3), P2 fails (21 < 22), prefill fails (0.504x < 0.85x). This table is the cost the fix above targets; it is not a result for this head. P1 is 0/40 on K1 and 26/40 on mode 4 on the same box and binary. K0 did not run on that head; it ran on this one (below).

Re-gate on this head (passing)

lucebox4 R9700 (PCIe 3.0 x2), 2026-11-11, head 50020e4c5, one luce_server (md5 ecbe4373) for both arms, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), the same 14 sessions (882 requests, 40 P1 and 40 P2 probes per arm). Server flags on both arms: --max-ctx 139264 --prefix-cache-max-mib 40960 --prefix-cache-slots 8 --disk-prefix-cache off --no-kv-cache --cache-type-k q8_0 --cache-type-v q8_0 --kvflash 16384. Runner: thinking on, effort medium, T 1.0, top_p 0.95, top_k 20, probe max_tokens 16384. Arms differ only in env: K1 LUCE_KVFLASH_POLICY=lru; K0 LUCE_KVFLASH_POLICY=lru LUCE_KVFLASH_PREAMBLE_ROWS=0; M4 none. The K1 and M4 server logs show the preamble pin (851 lines each), K0's none. Caching state: prompt tokens 62,012,284 on every arm, cached prefix 60,542,275 (K0, K1) vs 60,542,272 (M4), equal on 881 of 882 requests against each. No power limit set or recorded. Gates were fixed before any run (m4cost_gate.sh).

Arm P1 P2 Session prefill (all) Per-session prefill med / mean / best / min Decode, out >= 64: med / mean / best / min Wall
K0 (1.0: lru, no pin) 1/40 19/40 811 tok/s (1812 s) 818 / 813 / 879 / 715 29.1 / 29.2 / 29.5 / 29.0 (n 79) 4241 s
K1 (lru + pin) 0/40 24/40 811 tok/s (1813 s) 818 / 813 / 876 / 715 29.2 / 29.2 / 29.6 / 29.0 (n 80) 3347 s
Mode 4 31/40 26/40 697 tok/s (2109 s), 0.860x 718 / 704 / 790 / 589 29.0 / 29.0 / 29.4 / 28.7 (n 71) 3718 s (1.11x)

Gate lines, M4 vs K1: prefill >= 0.85x PASS (0.860); P1 >= K1 + 3 PASS (31 vs 0); P2 >= K1 - 2 PASS (26 vs 24). M4 vs K0: prefill >= 0.85x PASS (0.859); P1 >= K0 + 3 PASS (31 vs 1); P2 >= K0 - 2 PASS (26 vs 19). Wall is not engine speed here: it includes the probes' answers, and K0's probes generated 63,116 output tokens against 37,166 (K1) and 39,113 (M4). Mode-4 paths: reused 637, prescore 80, full 93, skip-fits 21, skip-pool 20, none 31; scorer 106.3 s, paging 28.0 s, re-prefill 63,332 tokens.

Screen before the gate (3 of the sessions, 151 requests, same box): this head vs its parent a90f8f1de (md5 33daf153): session prefill 737 vs 443 tok/s (0.883x vs 0.530x of K1's 835), scorer 23.5 s vs 96.0 s, P1 9/9 on both. With LUCE_KVFLASH_M4_SINGLE=0 the head's picks equal the parent's on 25 of 25 requests (the two-pass path is unchanged).

Versus main (same box, serial, speed and text)

lucebox6 R9700 (gfx1201), ROCm 7.2.4, 2026-11-11, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), q8_0 K/V, --max-ctx 131072 --prefix-cache-slots 0 --no-kv-cache --kvflash 16384, temperature 0, no draft model in either arm. Serial engine; 3 conversations, turn 1 21585-26509 and turn 2 21758-26682 prompt tokens, prefix_len=0 on both sides (full reprocess, no caching). Two server starts per pin, interleaved; n = 6 per pin and turn. main d972beba3 (luce_server md5 f2f9ae3f) against 50020e4c5 (md5 b38275f8; it differs from this head only in ENVIRONMENT.md, a comment and one unit test).

Pin Turn Prefill tok/s med (mean, best, worst) Decode tok/s med Wall s med
main 1 901.1 (900.4, 903.6, 897.0) 29.5 30.0
this PR 1 880.3 (881.7, 886.2, 878.6), -2.3% 29.3 30.7
main 2 899.2 (897.7, 899.6, 894.1) 29.5 30.3
this PR 2 861.0 (861.0, 861.2, 860.6), -4.2% 29.3 31.5

The text differs from main on all 6 requests, and each side repeats its own text on both starts: main logs policy=drafter (attaches on first reselect) and, with no drafter, keeps the newest chunks; this PR logs recall profile: mode 4 / policy=qk and picks chunks by score. This run has no recall probes; recall against KVFlash 1.0 is the K0 row of the gate above.

Versus full KV (no KVFlash)

These compare against no KVFlash, not against KVFlash 1.0, and none was measured on this head. They also ran with the preamble reserve outside --kvflash N on earlier binaries, so the pool size differs from this head (see the LongMemEval-S line).

E20 (20 live coding-agent tasks), lucebox4, 2026-11-10, luce_server md5 6ae4a114 (commit e964d21a7; not the binary of the gate table above, which is a2f83fbd / 787e9f89b; --kvflash there installs mode 4). Server flags: --max-ctx 135168 --prefix-cache-max-mib 12288 --prefix-cache-slots 4 --disk-prefix-cache off --cache-type-k q8_0 --cache-type-v q8_0 --no-kv-cache, --kvflash 16384 on the K arms. Runner: thinking on, effort medium, T 1.0, top_p 0.95, top_k 20, seed 1, max_tokens 16384, same 20 tasks on every arm.

Arm Pass Sum client wall Session prefill Median decode
Full KV 20/20 7002 s 494 tok/s 19.1 tok/s
Mode 4 19/20 (p02) 4572 s 690 tok/s 28.9 tok/s
K1L (lru + pin) 19/20 (p02) 4261 s 859 tok/s 29.1 tok/s

Full KV is at ceiling on this set (it fails the pre-registered 25-85% band), so the set measures no regression against full KV and speed, not recall. It cannot show a recall edge over recency: mode 4 and K1L tie. Mode 4 session prefill is 0.80x of K1L and wall +7%.

EXECODE A6 (2026-10-27, lucebox6 R9700, binary md5 0ad28447, --max-ctx 135168, q8_0 K/V, 20 paired tasks, pre-registered): 19/20 against 20/20 for full KV, difference -0.05, bootstrap CI [-0.15, 0.0], exact McNemar p = 1.000, parity PASS (frozen margin d <= 2). Medians: decode 19.2 to 27.6 tok/s, request wall 23.0 to 12.9 s, total wall 7,062.5 to 5,135.6 s, first-request prefill (>50K new tokens) 566.2 to 749.6 tok/s.

LongMemEval-S (binary md5 cec754b6, thinking off, T 0, seed 1, max_tokens 512, judge gpt-4o-2024-08-06, n = 500, one request with the whole history): --kvflash 16384 against full context is a measured loss, -8.0 points [-11.4, -4.6] with official labels, -8.8 [-12.0, -5.8] with audited labels (pre-registered verdict: accuracy loss). That binary reserved 6144 extra preamble rows on top of the pool; this head keeps 1152 inside N. Neither A6 nor LongMemEval-S bears on this head's pool size.

Long cold turn (724119d03, was a21ae4883)

With the --kvflash profile, a single cold turn longer than the pool skipped the recall: its user turn starts in chunk 0, so nothing is sealed before R, and the turn is longer than the recall gate's PoolTooSmall bound. The turn then ran LRU only, and a fact the pool dropped from the middle of the prompt was gone at decode. kvflash_max_turn_tokens is that bound. kvflash_mode4_plan takes it as an optional cap and rewinds a longer turn to its tail, so the recall scores and pages in everything before it. The gate is unchanged and a turn within the cap keeps its R. A capped R can land inside the latest question's 64-query window, where a replay would recompute the window's tail while the stored question rows keep the old pool's values; kvflash_uq_snap_rewind moves such an R to the window end.

Serial needle on the RTX 3090: Qwen3.8-27B IQ4_XS, DFlash2 drafter, --max-ctx 131072, --kvflash 16384, q8_0 K and V, temperature 0, the shipped --kvflash profile with nothing else set, one cold 81,430-token turn of 3,500 records ("record i: the code for key_i is ..."), question at the end, cached tokens 0, one run per head.

head server log key_123 (want 74010) prefill
d98512ff0 prefill reselect mode=4: skip, no sealed chunk before R (LRU) 10098, wrong 74.4 s
a21ae4883 prefill reselect mode=4: R4=70230 (end 81430, re-prefill 11200 tokens) 74010, right 87.4 s

The 13 s difference is the scoring and the 11,200-token replay. One key, one run per head.

Second shape, the three-turn serial smoke (--kvflash 16384, same model and flags; turn 1 is a cold 50,408-token request with two user messages, the last one longer than the cap; one run per head):

head turn 1 (want 78146) turn 2 (want 17216) turn 3 (want 75631) turn 1 prefill
d98512ff0 74084, wrong right right 48.3 s
43ebed574 74084, wrong right right 51.6 s
a21ae4883 78146, right (R4=39208, re-prefill 11200) right right 56.8 s

What ran on the commits after d98512ff0 (RTX 3090):

  • test_server_unit on CPU: 1382 passed, 0 failed, 17 skipped on a21ae4883 (1374/0/17 on d98512ff0).
  • test_server_unit on the GPU: 1390/0/6 on 43ebed574 (1385/0/6 on d98512ff0).
  • The three-turn smoke above: the same answers and the same 30 picks on 43ebed574 and d98512ff0.

Not run on them: ctest, the landing gate, ROCm, tensor parallelism.

/props.kvflash (was #999)

/props gains kvflash {enabled, pool_tokens, mode}, so a client can see whether recall is on without reading the server log. server_main fills it after the --kvflash* flags have set the environment the backend reads, so env-only LUCE_KVFLASH counts too. pool_tokens is LUCE_KVFLASH (0 = auto, sized at load). mode comes from the backend's own kvflash_reselect_mode over LUCE_KVFLASH_PREFILL_RESELECT, LUCE_KVFLASH_POLICY and LUCE_KVFLASH_RESELECT_SCORER: prefill-reselect=4 while recall runs, empty on another architecture, with --kvflash-policy lru or with a scorer other than exact. It does not see tensor parallelism, which turns recall off later at load. #999's nested recall {protocol, active, mode} object and the --help marker are not carried: no client reads them.

Env knobs

--kvflash on a qwen35 target installs mode 4: LUCE_KVFLASH_POLICY=qk, RECALL_BLOCKS=6, SCORE_POOL_RECENT=1, SELECT_UNIT=2, PREAMBLE_ROWS=1152, PREFILL_RESELECT=4, RESELECT_SCORER=exact, UQ_QUERIES=64. A variable that is already set keeps its value. The values are the named constants of KvflashRecallProfile in launch_profiles.h. Other architectures install nothing. test_launch_policy.cpp pins the full (name, value) set of both profiles.

New knobs documented in server/docs/ENVIRONMENT.md: the recall knobs listed there. LUCE_KVFLASH_M4_SINGLE (0 forces the two-pass path) is read in qwen35_backend.cpp and is not yet in ENVIRONMENT.md.

Limits

Not run on this head (they ran on the earlier head, whose mode 4 this head matches request for request on the 3-session replay):

  • The 14-session landing gate.
  • E20, EXECODE and LongMemEval-S on this head.
  • Any engine run of the cost fix on CUDA. The RTX 3090 results are unit tests and ctest only; the engine numbers are R9700 (HIP).
  • HIP unit tests and Strix (gfx1151) / DGX GB10 results (Strix and DGX GB10 pass on this head's CI; the R9700 job is to be re-run; no local run).
  • A main-vs-branch run with recall probes; the serial run against main measures speed and text only.
  • Tensor parallelism.
  • The ctest baseline is main dd85248d3, not d972beba3; the alone rerun of q4k_rows is pending (one alone attempt of committed_tokens_match_ar_sampling is logged with rc=77 and no pass or fail count, so it is not counted). No CPU-only ctest baseline.
  • A power limit is not recorded for any engine run.

Commit map (earlier head -> this head)

Earlier This head Subject
a261eb59e 3047a9ff2 the KVFlash decode mask hides rows above the decode position
50020e4c5 160a786b2 mode 4 runs a turn once while the human question is unchanged
2a6e64c53 b6d1bea00 LUCE_KVFLASH_RECALL_BLOCKS is a 10 x N chunk budget
9fc7a36a4 3a8a77357 the pinned-preamble recall fixture carves the reserve inside the pool
d98512ff0 05c4c3a31 the per-process q_cap_all reservation under tensor parallelism
d13a91857 99cede4b6 recapture the turn's tail queries after a rewind
89a1c9ffa dbde568ea write q_cap_all only when recall or UQ is on
68e5485a1 7d435c8e7 kvflash help default and the resolved reselect mode in the startup line
43ebed574 fba9ef4a2 read only the captured rows of q_cap_all
a21ae4883 724119d03 recall a turn's tail when the whole turn cannot be replayed
(new) efec5616a remove mode 5
(#999) 4f2de18da /props.kvflash reports whether recall is on

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 31 files

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

View guided diff | Re-trigger cubic

Comment thread server/src/common/parallel_for.h Resolved
Comment thread server/docs/ENVIRONMENT.md Resolved
Comment thread server/docs/ENVIRONMENT.md Resolved
Comment thread server/src/common/kvflash_gpu_mass.h Resolved
Comment thread server/src/common/kvflash_pool_flag.h Resolved
@dusterbloom
dusterbloom marked this pull request as draft November 7, 2026 13:28

@davide221 davide221 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review

Found 2 issues.

Copy to agent

Use the copy button in the top-right corner of the block, then paste into your agent.

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.

Repository: Luce-Org/lucebox
Reviewed commit: 4d2313821a060a5de536fd496105e41a6fa0652a
Verify the findings against the current code, run relevant checks for any fixes, and explain any findings you reject.

<file name="server/src/common/kvflash_gpu_mass.h">

<violation number="1" line="239" end_line="241">
permute axes produce [m, NBT, HK, G], but the cpy destination view is shaped (G, HK, NBT, m)

The graph's per-layer output copy must place the mass of plane `hk*G + g`, block `gb`, query `j` into `o_t[l*HQ + hk*G + g, gb, j]`. The source tensor is `mm` reshaped to `[NBT, G, m, HK]` (dim1 index = g, dim2 = query j, dim3 = hk). `ggml_permute(ctx, x, axis0..3)` sets `result-&gt;ne[i] = x-&gt;ne[axis[i]]`, so `ggml_permute(ctx, ..., 2, 0, 3, 1)` yields shape `[m, NBT, HK, G]` — not the `[G, HK, NBT, m]` the comment claims and the `dst` view on line 240 (`ggml_view_4d(ctx, o_t, G, HK, NBT, m, ...)`) requires. To match `dst`, the axes must be `(1, 3, 0, 2)` (g, hk, gb, j). As written, `ggml_cpy(ctx, mp, dst)` is handed two tensors whose shapes differ and are not even the same permutation order as the strides imply: depending on the vendored ggml's `ggml_cpy`/`ggml_compute_forward_dup` handling of mismatched shapes (the definition to check in ggml.c), this either aborts the process on the first recall scoring call (the default mode-4 profile's GPU path), or copies elements in the wrong order so every chunk score/plane aggregate is garbage and recall picks arbitrary chunks. The unit tests in this PR (test_kvflash_gpu_mass.cpp) exercise exactly this path with G=3, m=8, so if they genuinely pass on the vendored ggml, that port must special-case permuted copies — that behavior in `ggml_compute_forward_dup`/`ggml_cpy` is the fact I could not verify. Fix: change the permute to `ggml_permute(ctx, ggml_reshape_4d(ctx, mm, NBT, G, m, HK), 1, 3, 0, 2)` so the source shape matches the destination view (G, HK, NBT, m).

<suggestion>
                ggml_tensor * mp = ggml_permute(ctx, ggml_reshape_4d(ctx, mm, NBT, G, m, HK), 1, 3, 0, 2);   // [G, HK, NBT, m]
</suggestion>
</violation>

</file>

<file name="server/src/qwen35/qwen35_backend.cpp">

<violation number="1" line="3165" end_line="3167">
Mode-4 'pool fits' gate ignores the preamble pin reserve, so recall is skipped for prompts that no longer fit the histor

`kvflash_recall_setup` decides whether mode-4 recall runs with `kvflash_pool_fits(end_abs, ctk, reserve, kvflash_pager_.n_pool_blocks())`, which compares the prompt's chunk count plus reserve against the **whole** pool. But with `LUCE_KVFLASH_PREAMBLE_ROWS` active (the shipped recall profile sets it to 1152 rows = 18 chunks of 64, taken out of `--kvflash N`), the pool blocks available for history are `n_pool_blocks - preamble_reserve/ct`, as the same PR's own budget math in `kvflash_recall_tail_chunks` (which subtracts `pin_reserve_chunks`) and the profile test `test_kvflash_recall_profile_leaves_the_pool_at_the_requested_size` (history = pool − reserve) establish.

Concrete break with `--kvflash 16384` and the default profile: a 15,500-token prompt whose last user turn is 400 tokens gives `end_abs/ct = 243` chunks and `reserve = ceil(400/64)+1 = 8`; `243 + 8 = 251 &lt;= 256` so the gate logs "skip, fits" and recall is disabled. In reality 18 blocks hold the pinned system+tools preamble, so only 238 blocks remain: the prefill overflows the history budget and evicts the oldest unpinned chunks mid-prefill with no reselect — exactly the degraded case this feature exists to fix, and the model decodes on a pool that lost the old context the question needed.

Fix: subtract the pin reserve from the block count passed to `kvflash_pool_fits` (e.g. `kvflash_pager_.n_pool_blocks() - kvflash_preamble_reserve_ / ctk`), or add a `pin_reserve_chunks` parameter to the helper, so the gate tests the history budget the rest of the recall path uses.
</violation>

</file>
Issue overview
  1. permute axes produce [m, NBT, HK, G], but the cpy destination view is shaped (G, HK, NBT, m) · likely (server/src/common/kvflash_gpu_mass.h:239-241)

  2. Mode-4 'pool fits' gate ignores the preamble pin reserve, so recall is skipped for prompts that no longer fit the histor · confirmed (server/src/qwen35/qwen35_backend.cpp:3165-3167)

Reviewed 4d231382 · 3 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again

Comment thread server/src/common/kvflash_gpu_mass.h Resolved
Comment thread server/src/qwen35/qwen35_backend.cpp Resolved
@davide221

Copy link
Copy Markdown
Contributor

CI failed on 4d231382

New KVFlash pager tests fail on CI: page-out needs pinned host memory the stub CUDA driver cannot provide

Three of the pull request's new unit tests fail because they never switch the pager to pageable host memory. On the CI runner the CUDA driver is a stub, so the pinned host allocation inside page-out fails, eviction stops working, and the pool fills up exactly where the tests expect it to recycle blocks. The fault is in the new tests' setup, not in the pager or the recall logic under test.

How to fix: Call set_pinned_host_backing(false) on the pagers in the two new test files so page-out works under CI's stub CUDA driver.

  1. In server/test/unit/test_kvflash_pager_recall.cpp, add set_pinned_host_backing(false) right after each attach(...) of the fresh, stale and second-test p pagers.
  2. In server/test/unit/test_kvflash_pin_preamble.cpp, add the same call after must(p.attach(cfg, h.k, h.v)); in pin_survives_the_dense_reset_path.
  3. Rebuild and run ctest -R 'KvflashPagerRecallFixture|KvflashPinPreambleFixture' --output-on-failure (or push and let CI rerun the Build job).

failing unit tests (stub CUDA host backing) · high confidence · run

Details and evidence

Failed: Build (cmake + uv sync --extra megakernel) › Build and run C++ CPU unit tests

Job "Build (cmake + uv sync --extra megakernel)", step "Build and run C++ CPU unit tests": 3 of 1436 tests fail — KvflashPagerRecallFixture.recent_pool_after_truncate_equals_a_fresh_lru_pager, KvflashPagerRecallFixture.reselect_recent_on_a_fitting_prefix_is_a_noop, and KvflashPinPreambleFixture.pin_survives_the_dense_reset_path.

Root cause: the two new test files added by this PR build their own pager harness and never call KvFlashPager::set_pinned_host_backing(false). On the CI runner the CUDA driver is a stub, so when a test first forces an eviction, page_out → alloc_host_buffer_ → cudaMallocHost fails, page_out returns false, ensure_free_block() finds no victim, and slot_for/alloc_span fail. The log shows the exact chain for the pin test:

[kvflash] cudaMallocHost failed: CUDA driver is a stub library
[kvflash] no pool slot at pos 512 (pool 512 exhausted)
Test: KvflashPinPreambleFixture::pin_survives_the_dense_reset_path
    Fail: pin helper precondition failed

Pos 512 is chunk 8 — the first allocation that must evict on the 8-block test pool (server/test/unit/test_kvflash_pin_preamble.cpp:147, alloc_span(4*64, 8*64) with 4 chunks already resident). The two recall tests die at the same point inside leave_stale_pick (server/test/unit/test_kvflash_pager_recall.cpp:58, alloc_span(0, 14*kChunk)), throwing "pager recall helper precondition failed" from must() (line 50).

The pass/fail pattern confirms it: every failing test triggers a page-out; every new test that stays within the pool (slot_mask_hides_rows_above_the_decode_position, pin_preamble_truncate_below_then_repin, all the pure-helper cases) passes. The pager itself documents the knob for exactly this situation: "Pass false to use pageable memory instead ... works where pinned allocation is unavailable (hosted CI with a stub driver, CPU-backend tests)" (server/src/common/kvflash_pager.h:201-206, setter at :206, failure path at :1199-1207). The pre-existing pager tests pass on the same runner, so the production code is fine.

Other things to try

  • Reuse the shared helper server/test/unit/kvflash_pager_test_host.h that the older pager tests build on, instead of hand-rolled harnesses, so this setup cannot be missed again.
  • Consider making alloc_host_buffer_ fall back to pageable malloc when cudaMallocHost fails, so the pager degrades instead of refusing to evict; that is a behavior change and should be a separate PR.
  • Reproduce in the CI container with ctest -R 'KvflashPagerRecallFixture|KvflashPinPreambleFixture' --output-on-failure.
Patch (2 files)

Not applied anywhere. To try it, save the diff as fix.patch and run git apply fix.patch.

diff --git a/server/test/unit/test_kvflash_pager_recall.cpp b/server/test/unit/test_kvflash_pager_recall.cpp
index ff362058..2794a003 100644
--- a/server/test/unit/test_kvflash_pager_recall.cpp
+++ b/server/test/unit/test_kvflash_pager_recall.cpp
@@ -70,12 +70,14 @@ TEST_CASE(KvflashPagerRecallFixture, recent_pool_after_truncate_equals_a_fresh_l
     Harness F;
     KvFlashPager fresh;
     REQUIRE(fresh.attach(cfg(), F.k, F.v));
+    fresh.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     REQUIRE(fresh.alloc_span(0, C));
     const std::vector<bool> want = resident_set(fresh, 10);
 
     Harness A;
     KvFlashPager stale;
     REQUIRE(stale.attach(cfg(), A.k, A.v));
+    stale.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     leave_stale_pick(stale);
     stale.truncate(C);
     CHECK(stale.is_resident(2));                       // without the recency step the stale pick survives
@@ -91,6 +93,7 @@ TEST_CASE(KvflashPagerRecallFixture, reselect_recent_on_a_fitting_prefix_is_a_no
     Harness A;
     KvFlashPager p;
     REQUIRE(p.attach(cfg(), A.k, A.v));
+    p.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     leave_stale_pick(p);
     p.truncate(6 * kChunk);                            // 6 chunks <= 8 blocks
     const std::vector<bool> before = resident_set(p, 6);
diff --git a/server/test/unit/test_kvflash_pin_preamble.cpp b/server/test/unit/test_kvflash_pin_preamble.cpp
index 728388ae..350dcdd4 100644
--- a/server/test/unit/test_kvflash_pin_preamble.cpp
+++ b/server/test/unit/test_kvflash_pin_preamble.cpp
@@ -138,6 +138,7 @@ TEST_CASE(KvflashPinPreambleFixture, pin_survives_the_dense_reset_path) {
     cfg.pool_tokens = 512; cfg.chunk_tokens = 64; cfg.sink_chunks = 1; cfg.tail_window_chunks = 4;
     KvFlashPager p;
     must(p.attach(cfg, h.k, h.v));
+    p.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     must(p.alloc_span(0, 3 * 64));
     kvflash_pin_preamble(p, 300, 64);
     p.reset();
Prompt for a coding agent
A GitHub Actions run failed in Luce-Org/lucebox. Please fix it.

- Pull request #963 (branch `kvflash/prefill-recall`), commit 4d2313821a060a5de536fd496105e41a6fa0652a
- Workflow: CI (https://github.com/Luce-Org/lucebox/actions/runs/42376524808)
- Failed: Build (cmake + uv sync --extra megakernel) › Build and run C++ CPU unit tests

Diagnosis:
Job "Build (cmake + uv sync --extra megakernel)", step "Build and run C++ CPU unit tests": 3 of 1436 tests fail — `KvflashPagerRecallFixture.recent_pool_after_truncate_equals_a_fresh_lru_pager`, `KvflashPagerRecallFixture.reselect_recent_on_a_fitting_prefix_is_a_noop`, and `KvflashPinPreambleFixture.pin_survives_the_dense_reset_path`.

Root cause: the two new test files added by this PR build their own pager harness and never call `KvFlashPager::set_pinned_host_backing(false)`. On the CI runner the CUDA driver is a stub, so when a test first forces an eviction, `page_out` → `alloc_host_buffer_` → `cudaMallocHost` fails, `page_out` returns false, `ensure_free_block()` finds no victim, and `slot_for`/`alloc_span` fail. The log shows the exact chain for the pin test:

```
[kvflash] cudaMallocHost failed: CUDA driver is a stub library
[kvflash] no pool slot at pos 512 (pool 512 exhausted)
Test: KvflashPinPreambleFixture::pin_survives_the_dense_reset_path
    Fail: pin helper precondition failed
```

Pos 512 is chunk 8 — the first allocation that must evict on the 8-block test pool (server/test/unit/test_kvflash_pin_preamble.cpp:147, `alloc_span(4*64, 8*64)` with 4 chunks already resident). The two recall tests die at the same point inside `leave_stale_pick` (server/test/unit/test_kvflash_pager_recall.cpp:58, `alloc_span(0, 14*kChunk)`), throwing "pager recall helper precondition failed" from `must()` (line 50).

The pass/fail pattern confirms it: every failing test triggers a page-out; every new test that stays within the pool (`slot_mask_hides_rows_above_the_decode_position`, `pin_preamble_truncate_below_then_repin`, all the pure-helper cases) passes. The pager itself documents the knob for exactly this situation: "Pass false to use pageable memory instead ... works where pinned allocation is unavailable (hosted CI with a stub driver, CPU-backend tests)" (server/src/common/kvflash_pager.h:201-206, setter at :206, failure path at :1199-1207). The pre-existing pager tests pass on the same runner, so the production code is fine.

Proposed fix: Call set_pinned_host_backing(false) on the pagers in the two new test files so page-out works under CI's stub CUDA driver.
1. In `server/test/unit/test_kvflash_pager_recall.cpp`, add `set_pinned_host_backing(false)` right after each `attach(...)` of the `fresh`, `stale` and second-test `p` pagers.
2. In `server/test/unit/test_kvflash_pin_preamble.cpp`, add the same call after `must(p.attach(cfg, h.k, h.v));` in `pin_survives_the_dense_reset_path`.
3. Rebuild and run `ctest -R 'KvflashPagerRecallFixture|KvflashPinPreambleFixture' --output-on-failure` (or push and let CI rerun the Build job).

Suggested patch (a starting point):
```diff
diff --git a/server/test/unit/test_kvflash_pager_recall.cpp b/server/test/unit/test_kvflash_pager_recall.cpp
index ff362058..2794a003 100644
--- a/server/test/unit/test_kvflash_pager_recall.cpp
+++ b/server/test/unit/test_kvflash_pager_recall.cpp
@@ -70,12 +70,14 @@ TEST_CASE(KvflashPagerRecallFixture, recent_pool_after_truncate_equals_a_fresh_l
     Harness F;
     KvFlashPager fresh;
     REQUIRE(fresh.attach(cfg(), F.k, F.v));
+    fresh.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     REQUIRE(fresh.alloc_span(0, C));
     const std::vector<bool> want = resident_set(fresh, 10);
 
     Harness A;
     KvFlashPager stale;
     REQUIRE(stale.attach(cfg(), A.k, A.v));
+    stale.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     leave_stale_pick(stale);
     stale.truncate(C);
     CHECK(stale.is_resident(2));                       // without the recency step the stale pick survives
@@ -91,6 +93,7 @@ TEST_CASE(KvflashPagerRecallFixture, reselect_recent_on_a_fitting_prefix_is_a_no
     Harness A;
     KvFlashPager p;
     REQUIRE(p.attach(cfg(), A.k, A.v));
+    p.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     leave_stale_pick(p);
     p.truncate(6 * kChunk);                            // 6 chunks <= 8 blocks
     const std::vector<bool> before = resident_set(p, 6);
diff --git a/server/test/unit/test_kvflash_pin_preamble.cpp b/server/test/unit/test_kvflash_pin_preamble.cpp
index 728388ae..350dcdd4 100644
--- a/server/test/unit/test_kvflash_pin_preamble.cpp
+++ b/server/test/unit/test_kvflash_pin_preamble.cpp
@@ -138,6 +138,7 @@ TEST_CASE(KvflashPinPreambleFixture, pin_survives_the_dense_reset_path) {
     cfg.pool_tokens = 512; cfg.chunk_tokens = 64; cfg.sink_chunks = 1; cfg.tail_window_chunks = 4;
     KvFlashPager p;
     must(p.attach(cfg, h.k, h.v));
+    p.set_pinned_host_backing(false);   // CI's stub CUDA driver cannot cudaMallocHost
     must(p.alloc_span(0, 3 * 64));
     kvflash_pin_preamble(p, 300, 64);
     p.reset();
```

Check out that commit, confirm the diagnosis, make the fix (adjust it if the diagnosis is wrong), and re-run the failing step locally before pushing.

Reply with /ci-fix and what to change to get a revised diagnosis.

@dusterbloom
dusterbloom force-pushed the kvflash/prefill-recall branch from 4d23138 to 2f0b183 Compare November 11, 2026 07:57
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
… tensors

The attach path pins host staging with cudaMallocHost by default. On the
stub-driver CI runner that call fails, so 3 tests in these two fixtures
failed in the Linux build of Luce-Org#963. Set set_pinned_host_backing(false)
before each attach, as test_kvflash_pager_serde.cpp already does.

Local 3090, CUDA_VISIBLE_DEVICES="": 9/12 before, 12/12 after; 12/12 with the GPU.
@dusterbloom
dusterbloom force-pushed the kvflash/prefill-recall branch from 2f0b183 to a97fe64 Compare November 11, 2026 15:20
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
score_chunks guarded S against the legacy 8-token kLookahead while passing
n_lookahead=128 to the drafter; a named constant now feeds both.

Review: Luce-Org#963 thread T8
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
… tensors

The attach path pins host staging with cudaMallocHost by default. On the
stub-driver CI runner that call fails, so 3 tests in these two fixtures
failed in the Linux build of Luce-Org#963. Set set_pinned_host_backing(false)
before each attach, as test_kvflash_pager_serde.cpp already does.

Local 3090, CUDA_VISIBLE_DEVICES="": 9/12 before, 12/12 after; 12/12 with the GPU.
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
…t N blocks of SELECT_UNIT chunks

The recall budget is 10 x RECALL_BLOCKS chunks (qwen35_backend.cpp in.recall_chunks), picked in
SELECT_UNIT-sized units (kvflash_prefill_reselect.h k_units = recall_chunks / unit). The row said
N blocks of SELECT_UNIT chunks, i.e. 12 chunks at the defaults instead of 60.

Review: Luce-Org#963 thread T5 (cubic)
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
…inside the pool

Production takes the preamble reserve out of --kvflash N (pool stays N). The fixture added it on top
(256 + 96) and asserted the feature-off tail, so it encoded the old design. It now uses a 256-block
pool and asserts the reduced tail (95 = off - 96); it fails if the tail formula ignores the reserve.

Review: Luce-Org#963 thread T7 (cubic)
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
…or parallelism

kvflash_q_cap_all_rows() keys off LUCE_KVFLASH only, so a tp_degree > 1 process reserves the
64-row capture although recall is off there. Threading the resolved plan through
create_target_cache_partial (6 callers across layer-split, shard daemon, drafter) and the VRAM estimate
exceeds the small-fix cap, so the comment now states the deliberate reservation and its size.

Review: Luce-Org#963 thread T10 (cubic)
dusterbloom added a commit to dusterbloom/lucebox-hub that referenced this pull request Nov 11, 2026
…or parallelism

kvflash_q_cap_all_rows() keys off LUCE_KVFLASH only, so a tp_degree > 1 process reserves the
64-row capture although recall is off there. Threading the resolved plan through
create_target_cache_partial (6 callers across layer-split, shard daemon, drafter) and the VRAM estimate
exceeds the small-fix cap, so the comment now states the deliberate reservation and its size.

Review: Luce-Org#963 thread T10 (cubic)
@dusterbloom
dusterbloom force-pushed the kvflash/prefill-recall branch from a97fe64 to d98512f Compare November 11, 2026 15:58
@davide221

Copy link
Copy Markdown
Contributor

CI failed on d98512ff

Build failed before compiling: the runner could not resolve Ubuntu's apt mirror to install CUDA packages

The build job failed inside the CUDA toolkit setup step, before any code was compiled. The runner's DNS could not resolve the Ubuntu package mirror, so apt could not download build-essential and the step exited with an error. This is a transient runner/network problem; the pull request's changes are not at fault.

How to fix: Re-run the failed job; no code change is needed.

  1. Open the failed workflow run in GitHub Actions.
  2. Click 'Re-run failed jobs' (or gh run rerun <run-id> --failed).
  3. The CUDA toolkit step should succeed once the runner can resolve the Ubuntu apt mirrors again.

dns failure fetching apt packages on runner · high confidence · run

Details and evidence

Failed: Build (cmake + uv sync --extra megakernel) › Run Jimver/cuda-toolkit@3d45d15

Job Build (cmake + uv sync --extra megakernel), step Jimver/cuda-toolkit (.github/workflows/ci.yml:56). The action adds NVIDIA's apt repo and runs apt-get -y install cuda-nvcc-12-8 cuda-cudart-dev-12-8 cuda-thrust-12-8 cuda-driver-dev-12-8 libcublas-dev-12-8, which pulls in build-essential from the Ubuntu archive. Every fetch from the Ubuntu mirrors failed with Temporary failure resolving 'azure.archive.ubuntu.com' (the azure mirror, archive.ubuntu.com and security.ubuntu.com all resolve via that name here), ending with:

E: Failed to fetch https://security.ubuntu.com/ubuntu/pool/main/b/build-essential/build-essential_12.10ubuntu1_amd64.deb  Temporary failure resolving 'azure.archive.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
##[error]Error: The process '/usr/bin/sudo' failed with exit code 100

The NVIDIA repo (developer.download.nvidia.com) resolved and downloaded fine, so this was a DNS/network outage specific to the Ubuntu mirrors on that hosted runner, not a package or version problem. Nothing in the diff touches the workflow or the CUDA setup; the failure happened before cmake ran.

Other things to try

  • If this recurs, harden the job: run sudo apt-get update and pre-install build-essential in a separate step with a retry, so the CUDA action does not depend on the Ubuntu mirrors being reachable in the same window.
  • Alternatively switch the apt mirror to a non-azure one (e.g. sudo sed -i 's|azure.archive.ubuntu.com|archive.ubuntu.com|g' /etc/apt/sources.list.d/*.list) before the CUDA step if the azure mirror proves flaky on ubuntu-24.04 runners.
Prompt for a coding agent
A GitHub Actions run failed in Luce-Org/lucebox. Please fix it.

- Pull request #963 (branch `kvflash/prefill-recall`), commit d98512ff0e6c92f2172645a7908cc5bde36e2f4b
- Workflow: CI (https://github.com/Luce-Org/lucebox/actions/runs/43053633989)
- Failed: Build (cmake + uv sync --extra megakernel) › Run Jimver/cuda-toolkit@3d45d157f327c09c04b50ee6ccdea2d9d017ec76

Diagnosis:
Job `Build (cmake + uv sync --extra megakernel)`, step `Jimver/cuda-toolkit` (.github/workflows/ci.yml:56). The action adds NVIDIA's apt repo and runs `apt-get -y install cuda-nvcc-12-8 cuda-cudart-dev-12-8 cuda-thrust-12-8 cuda-driver-dev-12-8 libcublas-dev-12-8`, which pulls in `build-essential` from the Ubuntu archive. Every fetch from the Ubuntu mirrors failed with `Temporary failure resolving 'azure.archive.ubuntu.com'` (the azure mirror, archive.ubuntu.com and security.ubuntu.com all resolve via that name here), ending with:

```
E: Failed to fetch https://security.ubuntu.com/ubuntu/pool/main/b/build-essential/build-essential_12.10ubuntu1_amd64.deb  Temporary failure resolving 'azure.archive.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
##[error]Error: The process '/usr/bin/sudo' failed with exit code 100
```

The NVIDIA repo (developer.download.nvidia.com) resolved and downloaded fine, so this was a DNS/network outage specific to the Ubuntu mirrors on that hosted runner, not a package or version problem. Nothing in the diff touches the workflow or the CUDA setup; the failure happened before cmake ran.

Proposed fix: Re-run the failed job; no code change is needed.
1. Open the failed workflow run in GitHub Actions.
2. Click 'Re-run failed jobs' (or `gh run rerun <run-id> --failed`).
3. The CUDA toolkit step should succeed once the runner can resolve the Ubuntu apt mirrors again.

Check out that commit, confirm the diagnosis, make the fix (adjust it if the diagnosis is wrong), and re-run the failing step locally before pushing.

Reply with /ci-fix and what to change to get a revised diagnosis.

…e_scorer

kvflash_load_drafter() loads the residency drafter once and remembers a failed load;
kvflash_ensure_scorer() calls it. Same messages, same order, no behavior change.
…estion queries in the selector

With LUCE_KVFLASH_PREFILL_RESELECT=4 the prefill is cut at the start of the last user turn, a
first pass over that turn captures its last query, the sealed chunks are scored with the exact
attention mass (selector v2), the state is restored to the cut, the pager is reselected by the
recall plan and the turn is written again under the new pool. Mode 5 takes the drafter's scores
for the same cut in one pass. Under tensor parallelism recall stays off (kvflash_recall_features).
Modes 1-3 stay off. Without a KVFlash pool nothing changes.
LUCE_KVFLASH_PREAMBLE_ROWS=R adds pool room (R rounded up to a chunk, clamped by max-ctx) for the
pinned system+tools chunks, so the preamble does not eat the history budget, and re-applies the pins
after a reset, restore or truncate. 0 turns the pin and the extra VRAM off.
Default-on under --kvflash: selector v2 with the GPU scorer (fixed in the backend, no variable),
mode 4, six recalled blocks in 2-chunk units on the recency scoring pool, the 6144-row preamble
pin (extra pool VRAM) and 64 user-question queries. The values are named constants in
KvflashRecallProfile and are installed as env defaults. A variable already set keeps its value.
Other architectures and launches without --kvflash install nothing.
…not 6144

The reserve grows the resident pool and every decode step attends the whole pool. 6144 rows took the pool from 16384 to 22528 and cost 4.7% decode; 1152 (18 chunks of 64) holds the measured 1064-token preamble pin and grows it by 7%.
…he pool no longer grows

The reserve took the pool from 16384 to 17664 rows and every decode step attends the whole pool, so the profile silently raised the decode cost the user chose with --kvflash. The reserve (1152 rows) is now held inside N, leaving N - 1152 history rows. A reserve that leaves no history row (>= N) turns the pin off with a log line, like a missing <|im_start|> id.
…evice KV (16 layers, 4 kv heads, hd256, q8_0)
…and not written since (dirty flag, d2h_skipped stat)
…unzeroed; a recall overwrites it, a fresh chunk zeroes it on handout, reselect flushes the rest
…ce at the end, not per hand-out, with residency unchanged (red: KvFlashStats has no fence_points yet)
reselect() recalls its page-ins as one batch on the page stream and waits
once at the end; on HIP every hand-out was a host wait (one per page-in,
~0.7 ms/event over 191k events in the K3 night). Residency and bytes
unchanged. The recent-pool and pick lines log the paging time on their own
line.
Review findings #2-#7, #11, #15 of the G2 landing review.

- Eviction hook: chunks at or past the committed position rank as pinned,
  so a drafter ledger that covers the whole prompt cannot evict the
  batch being allocated (RECALL_BLOCKS=0 lost 3 of 8 chunks).
- Recall gate (kvflash_recall_gate): mode 4 and 5 skip when no sealed
  chunk precedes the turn, when chunks(R) + the turn's reserve already
  fit the pool (the turn was counted twice), and when the pool cannot
  hold pins + recall + reserve; those requests run as a recency pool.
- GPU scorer: a request with nothing sealed or a free-VRAM refusal no
  longer latches kvflash_gpu_failed_ for the process.
- Victim tier defaults to 2x the pool in mode 5 only, and the auto pool
  budget accounts for its device bytes.
- Profile no longer exports PFLASH_SELECT_QUERY_PARSER (it changed PFlash
  compression's query), the retired PFLASH_SELECT_PARAGRAPH_JOIN or the
  unread LUCE_KVFLASH_SELECT_QUERIES; the mode-5 gate drops the parser.
- Docs: --prefill-drafter, RECALL_BLOCKS=0 wording, victim tier row.

Tests: test_kvflash_prefill_reselect (hook, gate, latch, victim),
test_launch_policy, test_kvflash_pflash_map.
Review findings #9 and #10 of the G2 landing review.

- victim_fifo_ gained an entry per page-out and was popped only when the
  free list ran empty; a freed slot is handed out again, so dead entries
  looked live. 2000 page-out/row-write cycles left 2000 entries; now the
  queue is swept (newest entry per chunk kept) past twice the tier.
- kvflash_pooled_ubatch clamps to the unpinned capacity: with 1152 rows
  pinned in a 2048 pool, LUCE_PREFILL_UBATCH=1024 no longer puts a
  16-chunk batch into 14 free blocks.

Tests: test_kvflash_pager_victim_tier (fifo growth), test_kvflash_ubatch.
… tensors

The attach path pins host staging with cudaMallocHost by default. On the
stub-driver CI runner that call fails, so 3 tests in these two fixtures
failed in the Linux build of Luce-Org#963. Set set_pinned_host_backing(false)
before each attach, as test_kvflash_pager_serde.cpp already does.

Local 3090, CUDA_VISIBLE_DEVICES="": 9/12 before, 12/12 after; 12/12 with the GPU.
…ageable staging on CPU tensors

Same cause as the pager recall and pin tests: attach pins host staging with
cudaMallocHost, which fails without a CUDA device (the stub-driver CI runner).
CUDA_VISIBLE_DEVICES="" on the 3090: 3 of these tests failed before, all pass
after; with the GPU they pass both ways.
@davide221

Copy link
Copy Markdown
Contributor

Thanks for the fixes. I ran a21ae488 merged onto current main on an R9700 + Strix Halo box (Qwen3.8-27B IQ4_XS on the R9700, no drafter):

  • The full unit suite passes on both GPUs (1412 tests).
  • Without --kvflash, the output is identical to main on 4 fixed prompts at temperature 0.
  • An 89.5K-token chat with 8 hidden codes at different depths, then 8 question turns, --kvflash 16384: it finds 8 of 8. Main with --kvflash finds 1 of 8, and main with the full context finds 8 of 8.
  • Cost against main with --kvflash: prefill 113.6 s vs 99.2 s (0.87x, in line with your 0.86x), a question turn 2.56 s vs 1.33 s, peak VRAM 15.4 GB vs 14.0 GB.

CI is green on this head and it merges cleanly with main.

…changed

A turn that continues the last request under the same question picks from the
last exact scores when they cover every recall candidate, and scores before the
pass (stored question queries + the previous request's tail queries) when a
candidate has none. Neither rewinds or resets the pool to recency. A new
question, another conversation or a rewritten scored chunk takes the unchanged
two-pass path. LUCE_KVFLASH_M4_SINGLE=0 forces the two-pass path on every turn.

Old M4 on 14 LIVECODE sessions (2026-11-10, lucebox4): 810 reselects, scorer
512 s, 750779 tokens prefilled twice, prefill 0.504x of 1.0 recency.
…t N blocks of SELECT_UNIT chunks

The recall budget is 10 x RECALL_BLOCKS chunks (qwen35_backend.cpp in.recall_chunks), picked in
SELECT_UNIT-sized units (kvflash_prefill_reselect.h k_units = recall_chunks / unit). The row said
N blocks of SELECT_UNIT chunks, i.e. 12 chunks at the defaults instead of 60.

Review: Luce-Org#963 thread T5 (cubic)
…inside the pool

Production takes the preamble reserve out of --kvflash N (pool stays N). The fixture added it on top
(256 + 96) and asserted the feature-off tail, so it encoded the old design. It now uses a 256-block
pool and asserts the reduced tail (95 = off - 96); it fails if the tail formula ignores the reserve.

Review: Luce-Org#963 thread T7 (cubic)
…or parallelism

kvflash_q_cap_all_rows() keys off LUCE_KVFLASH only, so a tp_degree > 1 process reserves the
64-row capture although recall is off there. Threading the resolved plan through
create_target_cache_partial (6 callers across layer-split, shard daemon, drafter) and the VRAM estimate
exceeds the small-fix cap, so the comment now states the deliberate reservation and its size.

Review: Luce-Org#963 thread T10 (cubic)
Defect: the m4 tail capture was gated on !rw_done, so on a rewound turn
kvflash_q_tail_ kept pass 1's queries (old pool) and pass 2, which
recomputes [R, end) under the new pool, captured nothing. The stale tail was
stored in kvflash_m4_last_.q_tail and scored by the next turn's Prescore.

Fix: capture in both passes (pass 1's tail is still what the rewind scores
with), and when kvflash_rewind_turn returns rc > 0 drop the tail rows at
positions >= R (kvflash_q_tail_trim) so pass 2 refills them. The append and
trim logic are pure helpers in kvflash_query_capture.h.

Tests: test_kvflash_uq.cpp q_tail_* cases (trim at R, ragged tail, pass 1
below R followed by pass 2 rows). Red: the helpers did not exist
(compile failure, red-item2-compile.log); green: 1377 passed, 0 failed.
Not run on hardware.
Defect: q_cap_all is allocated in every KVFlash process, and the graph
copied into it whenever in.q_capture was set. The qk policy keeps capture_qk
on for decode and restore steps, so every decode step copied Q into a tensor
nothing reads there. Under tensor parallelism that tensor has no split
(qwen35_tensor_parallel.cpp classifies only q_cap) while the Q it copies from
is split by head; the view-of-view copy into q_cap_all is expected to trip
the split_dim_view_src assert in ggml-backend-meta.cpp on the first decode
(derived from reading, not run on hardware).

Fix: separate graph input QwenGraphInputs::q_capture_all, set from the pure
kvflash_capture_q_all(recall_on, uq_on, tp_degree, capture_qk). Only the
prefill windows that feed kvflash_read_captured_queries (recall mode 4 tail,
UQ window) pass it; decode and restore steps and every other caller keep the
default false. The rewind-only capture (rw_capture) reads q_cap, so it does
not set it. TP zeroes recall and UQ in the feature plan, so it never writes.

Tests: test_kvflash_exact_mass.cpp q_cap_all_is_written_only_for_recall_or_uq_prefill.
Red: helper missing (compile failure, red-item1-compile.log); green: 1378
passed, 0 failed.
…he startup line

Defect: --help still said --kvflash-policy defaults to drafter, but the
recall profile makes it qk on qwen35 with --kvflash. The startup line chose
"mode 4" from kvflash_mode5_requested() alone, while apply_env_defaults keeps
an already-set LUCE_KVFLASH_PREFILL_RESELECT, so the line could say mode 4
while 0, 4 or 5 (without its GGUFs) ran; on other architectures the profile
is not applied at all but the line still said mode 4.

Fix: kvflash_recall_banner(arch, mode5, get) prints the resolved mode (the
operator's env value when set, else the profile's), says when the
environment set it, and says the profile is not applied off qwen35. Help
text states the real default.

Tests: test_launch_policy.cpp test_kvflash_banner_prints_the_resolved_reselect_mode
(test_feature_gate target). Red: helper missing (compile failure,
red-item4b-compile.log); green: test_feature_gate 5 passed, 0 failed
(green-item4-gate.log).
Defect: kvflash_read_captured_queries copied the whole q_cap_all tensor
(64 rows x every plane, about 24 MiB on the 27B) with one
ggml_backend_tensor_get per captured batch, then used only n_q rows per
plane. The tensor is [hd, T, n_head, n_fa]: the n_q rows of one plane are
contiguous but planes are T*hd floats apart, so a plain prefix read is not
enough.

Fix: one ggml_backend_tensor_get_2d reading n_q*hd floats from each plane
(kvflash_q_cap_all_read gives the arguments), then kvflash_regroup_queries
turns [plane][n_q][hd] into the scorer's [t][plane][hd]. Output is identical
to the old loop; bytes read drop from planes*T*hd*4 to planes*n_q*hd*4.

Tests: test_kvflash_exact_mass.cpp
captured_queries_are_read_as_n_q_rows_per_plane_and_regrouped (n_q 1, 5, 64;
element-wise against the tensor, byte count asserted). Red: helpers missing
(compile failure, red-item4c-compile.log); green: 1379 passed, 0 failed.
The real tensor_get_2d path was not run on a GPU.
…ayed

Mode-4 recall rewinds to R, the start of the last user turn, and skipped the
recall when nothing was sealed before R (a cold single turn starts its user
turn in chunk 0) or when the turn was longer than the pool can replay
(PoolTooSmall). That turn then ran LRU only, and a fact the pool dropped was
gone at decode: one cold 81,430-token turn, pool 16384, answered key_123 wrong.

kvflash_max_turn_tokens is the longest turn the recall gate replays (its
PoolTooSmall bound). kvflash_mode4_plan takes it as an optional cap and
rewinds a longer turn to that tail, so the recall scores and pages in
everything before it. The serial call site (kvflash_recall_setup) passes the
cap; the gate is unchanged and a turn within the cap keeps its R.

A capped R that lands inside the question window (LUCE_KVFLASH_UQ_QUERIES) moves
to the window end (kvflash_uq_snap_rewind): a replay from inside it would leave
the stored question rows from the old pool.

Tests: test_kvflash_prefill_reselect, the cap equals the gate's bound and the
capped plan on the needle's shape rewinds to the tail; test_kvflash_uq, the snap.
Mode 5 (the PFlash drafter's scores as the recall scorer) is dropped; mode 4 is the only recall mode.
kvflash_reselect_mode returns 4 or 0, the mode-5 branches of the recall setup and the drafter scorer
glue in the Qwen35 backend are gone, and so is the device victim tier that only mode 5 used
(kvflash_victim_blocks_default, kvflash_pool_density_factor and their test). The ENVIRONMENT.md rows
and the pflash-map gate are back to the mode-4 text.
/props gains kvflash {enabled, pool_tokens, mode}. server_main fills it after the --kvflash, --kvflash-policy
and --kvflash-tau flags have set the environment the backend reads, so env-only KVFlash counts too.
pool_tokens is LUCE_KVFLASH (0 means auto, sized at load). mode is the backend's own kvflash_reselect_mode
over LUCE_KVFLASH_PREFILL_RESELECT, LUCE_KVFLASH_POLICY and LUCE_KVFLASH_RESELECT_SCORER:
"prefill-reselect=4" while recall runs, empty on another architecture, with --kvflash-policy lru or with a
scorer other than exact.
@dusterbloom
dusterbloom force-pushed the kvflash/prefill-recall branch from a21ae48 to 4f2de18 Compare November 18, 2026 17:17
test_server_unit.cpp: main's Luce-Org#1031 pruned the model-card props tests next to the three /props.kvflash tests;
take main's version and keep the three tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants