Repository navigation
Agent race: granite-switch SR against stock vLLM LoRA, agents switching adapters - #145
Closed
AlonMalach wants to merge 20 commits into
Closed
AlonMalach wants to merge 20 commits into
AlonMalach wants to merge 20 commits into
Conversation
AlonMalach
requested review from
antonpibm,
aviv1ron1,
freunda and
yairallouche
as code owners
October 13, 2026 10:58
AlonMalach
marked this pull request as draft
October 14, 2026 07:48
A race page in the style of the ALORA-vs-LoRA race (docs/race_live.html),
for the adapter benchmark's switching cell: per lane, 16 agents, at most
8 at a time, each with a prompt of its own, generating 512 tokens and
switching adapter every 32 over 64 synthetic adapters, with tool calls
and round trips as real sleeps.
gs-sr-vllm granite-switch SR (the switch benchmark's shadow-residual
arm): one generation until a tool call, one adapter per task
lora-vllm stock vLLM serving the LoRA adapters (its native-lora arm):
a new request, and a re-prefill, at every switch
benchmarks/adapter_eval/race/:
- lane.py runs a lane: the switch benchmark's agent loop unchanged,
through slots (an agent starts when one finishes, not in waves), and
through a recording engine that notes each request's start, first
token, tokens and end and what the engine re-prefilled; the gaps
between requests are named from the agent's schedule. A warm-up run
first, the prefix cache reset before each run.
- run_race.py runs both lanes on one GPU through the benchmark's script
mode, with its workload settings, synthetic adapters, composer and
checkpoint check, and prints the race into the log as gzipped base64.
- build_page.py embeds a recorded race into docs/agent_race.html, and
refuses one carrying an absolute path.
The page has no race yet; tests/unit/test_agent_race.py runs the race end
to end on CPU with the real generator and agent loop and a fake engine.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
The race recorded on one A100 (vLLM 0.26.0): 16 agents per lane, 8 at a time, 512 tokens each, a switch every 32 over 64 synthetic adapters. gs-sr-vllm all agents done in 28.4 s, 66,974 prompt tokens re-prefilled lora-vllm all agents done in 91.8 s, 1,033,024 re-prefilled SR finished 3.2x sooner. Per agent, 12-16 s against 32-59 s, in line with the switching block's warm p95 on the same commit (15.9 s and 56.7 s). The SR checkpoint passed verify_composed.py, each lane ran after an untimed warm-up, and the KV demand stayed at 12% of capacity or less. The page test now empties the page it reads, rather than expect it empty. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The page's prefix hit counted every request, including each agent's first, which prefills the agent's own prompt and so can never hit: SR read 51%, though its later requests hit 99.9%. Its "re-prefilled" was likewise mostly that first prefill (66,974 of SR's tokens), the same in both lanes. An agent's first request is now its first prefill, counted apart; the page shows the re-prefill (prompt tokens computed again after it) and the context hit (the share of the context sent again that the cache held), computed from each request's recorded counts, so the race already recorded reads right without a new run: gs-sr-vllm re-prefilled 95 tokens, context hit 99.9% lora-vllm re-prefilled 966,161 tokens, context hit 9.2% The lane's metrics and report carry the same split for later runs. Signed-off-by: AlonMalach <alonmalach@gmail.com>
SR's adapter is now shown switching on the same schedule as stock vLLM's, as many times, instead of the one adapter the run keeps: the switch itself is not implemented, and on the GPU SR costs the same with or without one, since its K/V come from the base model and its generation runs on through the switch points. The recorder follows each SR agent's segments as its tokens stream and records a switch at each segment's end; stock vLLM's are recorded where it re-issues its request. Both lanes count their switches and their requests, and each lane reports the adapters the engine actually ran (one for SR). The race now runs 64 agents per lane through the same 8 slots (run_race.py --agents, default 64), and the page lays out a large fleet in two compact sub-columns per lane, with a 20x speed and a 5x default for a long race. The page still holds the 16-agent race; the next run replaces it. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…ches The page now holds one recorded race per number of agents per lane, chosen above its controls (the link keeps the choice): build_page.py adds a run under its number, replacing that number's race and keeping the others, and reads a page holding a single race as that number's. The 16-agent race, recorded before the recorder kept switches, has them now (build_page --add-switches): its workload regenerated by the same generator and seed, refused unless it matches the recorded agents and stock vLLM's adapters, and the switches placed at the recorded token marks, where the recorder now puts them (a test strips a new recording to the old form and gets it back). Times and counts are as recorded: gs-sr-vllm 256 switches over 32 requests (one adapter run) lora-vllm 256 switches over 256 requests Signed-off-by: AlonMalach <alonmalach@gmail.com>
The race with 64 agents per lane through the same 8 slots, recorded on
one A100 (vLLM 0.26.0), next to the 16-agent race on the page:
gs-sr-vllm all agents done in 114.4 s; 1,024 switches over 128
requests; 380 prompt tokens re-prefilled, context hit 99.9%
lora-vllm all agents done in 365.0 s; 1,024 switches over 1,024
requests; 3,822,836 re-prefilled, context hit 10.2%
SR finished 3.2x sooner, as with 16 agents; per agent 11-18 s against
28-66 s. The SR checkpoint passed verify_composed.py, each lane ran after
an untimed warm-up of the whole fleet, and the KV demand stayed at 22% of
capacity or less.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
run_race.py takes the agents at a time (--concurrency) and the tokens each agent generates (--decode), or a mix of lengths (--decode-mix): the generator runs once per length for an equal share of the agents, each with a seed of its own, and the agents are shuffled together with a fixed seed and renumbered, so the lengths interleave and both lanes read the same agents, a rerun the same workload. The page keys each race by its configuration (agents per lane, at a time, tokens per agent; the two recorded races keep theirs), labels its buttons from the race's metadata, sizes each agent's bar and label by its own length, and plays a long race faster by default. Signed-off-by: AlonMalach <alonmalach@gmail.com>
64 agents per lane, 16 at a time, generating 128, 256, 512 or 1,024 tokens (16 agents each, the same agents in both lanes), recorded on one A100 at b7e5421: gs-sr-vllm all agents done in 89.4 s; 960 switches over 128 requests; 380 prompt tokens re-prefilled, context hit 99.9% lora-vllm all agents done in 335.9 s; 960 switches over 960 requests; 3,796,552 re-prefilled, context hit 6.9% SR finished 3.8x sooner. By length, stock vLLM's median agent took 2.7x SR's at 128 tokens and 4.4x at 1,024: more switches, more re-prefill. The SR checkpoint passed verify_composed.py; KV demand stayed at 35% of capacity or less. Signed-off-by: AlonMalach <alonmalach@gmail.com>
64 agents per lane, 16 at a time, each generating 1,024 tokens, recorded on one A100 at b7e5421: gs-sr-vllm all agents done in 129.1 s; 2,048 switches over 256 requests; 1,140 prompt tokens re-prefilled, context hit 99.9% lora-vllm all agents done in 747.7 s; 2,048 switches over 2,048 requests; 8,971,268 re-prefilled, context hit 3.6% SR finished 5.8x sooner, its widest margin of the four races: each agent switches twice as often as at 512 tokens, and stock vLLM re-prefills a longer context at each switch. Per agent, SR took 27-39 s and stock vLLM 138-237 s. The SR checkpoint passed verify_composed.py; KV demand stayed at 37% of capacity or less. AGENT_RACE.md: the switch count, the first prefill's size and the context hit now cover all four races, not only the 512-token ones. Signed-off-by: AlonMalach <alonmalach@gmail.com>
docs/agent_race.html is 5.4 MB: its four recorded races are embedded so it opens straight from disk, as the ALORA-vs-LoRA race does. It is excluded from check-added-large-files next to uv.lock, the other file that is intentionally large; everything else is still held to 500 KB. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The page opened on a dense paragraph. It now opens with the experiment in
six short lines (question, agents, same work, real waits, what is
measured, setup) and a diagram, drawn in the race bars' own colors:
- one agent's ReAct loop, as the switching workload runs it: its prompt,
32-token think steps switching adapter at each (the adapters of a
recorded agent), a tool call after every 256 tokens, done;
- what each lane does at a switch: stock vLLM ends the request, waits a
round trip and prefills the whole context again; SR runs one request
until the tool call, its adapter changing in place (SR's switch is not
implemented yet, which a footnote keeps saying).
Hovering a block in the diagram, or a metric's label in the panels,
explains it; before, only two metrics were explained. The diagram keeps a
readable size and scrolls sideways on a narrow window. The race data is
unchanged.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
docs/media/agent_race_diagram.svg is the diagram the page opens with, as a file of its own, so the PR description (and any README) can show it. It carries the page's background, font and text styles, which inline it takes from the page. It is generated by the same tool as the page's copy and draws the same marks. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…labels
A fifth race, run on the paper's ReAct agents: the switch benchmark's agent
study generator (benchmarks/gen_agent_configs.py, copied unchanged from
staging 5cf9aef), built by race/react.py. Each agent walks a ReAct cycle of
typed steps at the generator's own lengths and switch rates: think (median
120 tokens, 35% switch, 1 + Poisson(2) segments), prepare_call (10, 95%), act
(a tool call), observe (35, 80%) and reflect (90, 70%), over its three cycles,
choosing adapters novelty-first over 64. Three settings are the race's, so
it reads against the other four: 64 agents of 128 to 1,024 tokens (16 each),
the switching workload's prompts, and a round trip within one datacenter.
Its agents switch about every 35 tokens, as the paper's do. run_race.py takes
--agents-kind react; the page keys it 64-16-mix-react. The job is still to
run.
The page:
- a prefill now shows what it computed and what the prefix cache held,
e.g. "prefill 3 tok · 2.6k cached": SR's request after a tool call
resends its context but recomputes almost nothing, which the old label
("prefill 2.6k tok") hid. From each request's recorded cached count, so
the four recorded races show it too.
- a generation names its step: "think (generation) ad26", or "generation"
for the switching block's agents; the tool call is "act (tool call)".
- the round trip at a switch is "adapter switching (rtt)", explained on the
page and in a diagram: stock vLLM sets a LoRA per request, so at each
switch its tokens go back to the agent, which sends a new request; 2-10 ms
within a datacenter, drawn per switch and slept for real.
- two diagrams, chosen with the race: the switching block's agent, or a
recorded ReAct agent's first cycle at scale. Both are generated by
docs/generation/generate_race_diagram.py, which also writes them as
images of their own (docs/media/agent_race_diagram*.svg).
The four recorded races' data is unchanged. 20 race tests pass, 5 of them
new: the ReAct agents are the generator's (steps, cycles, lengths and switch
rates), both lanes count the same switches and name the steps, and the race
has its own key.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
The paper's ReAct agents (think, prepare_call, act, observe, reflect), 64 per lane, 16 at a time, generating 128, 256, 512 or 1,024 tokens (16 each), within one datacenter, recorded on one A100 at b7e5421: gs-sr-vllm all agents done in 91.6 s; 866 switches over 215 requests; 1,351 prompt tokens re-prefilled, context hit 99.8% lora-vllm all agents done in 310.0 s; 866 switches over 1,009 requests; 3,393,152 re-prefilled, context hit 16.2% SR finished 3.4x sooner. By length, stock vLLM's median agent took 1.9x SR's at 128 tokens and 3.4x at 1,024. The pod's agents are those react.py builds locally (lengths, prompts, cycles and planned switches match). The SR checkpoint passed verify_composed.py; KV demand stayed at 35% of capacity or less. The page shows a step run on the base model as "base", which stock vLLM requests with no LoRA; the Requests help says stock vLLM also starts one at every ReAct step. AGENT_RACE.md: the context hit's range takes in this race. Signed-off-by: AlonMalach <alonmalach@gmail.com>
On top of feature/adapter-benchmark, whose harness grew to 101 KB, the race's code (race/, 22 KB) took every job's harness payload to 123 KB, over the 120 KB a single environment string can carry: render_job failed for all jobs, and 8 render tests with it. The harness payload now leaves race/ out. A script job whose script is in race/ (the race's own, run_race.py) carries it in the switch benchmark's payload instead, with the scripts the race runs on, where there is room: harness, every job 107 KB of 120 KB switch, other jobs 78 KB switch, a race job 100 KB The pod unpacks both payloads into one folder, so the race's code lands where it always has. A test checks that a race job carries it and any other job does not. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The ReAct race's diagrams now open with the agent as a state machine, LangGraph-style: its steps as nodes, START to END. The LLM nodes (think, prepare_call, observe, reflect) show their median tokens and how often they switch adapter; act is the tool node. The edges are the transitions the generator's three cycles take: think -> prepare_call -> act -> observe prepare_call -> prepare_call another call before acting (shortadapters) observe -> prepare_call act again (toolheavy10) observe, act, prepare_call -> reflect reflect -> think the next cycle any step -> END when the agent's 128-1,024 tokens run out with the three cycles written out under it. It shows with the ReAct race only, above the recorded agent's steps at scale; the other races keep their own diagram. Generated by generate_race_diagram.py machine, which also writes it as docs/media/agent_race_machine.svg. Each diagram's arrowhead now has an id of its own, as the page holds three. The races are unchanged. Signed-off-by: AlonMalach <alonmalach@gmail.com>
AlonMalach
force-pushed
the
feature/sr-lora-agent-race
branch
from
October 19, 2026 07:09
df598fb to
872c2b1
Compare
run_race.py takes --cycles (with --agents-kind react): the generator's cycles its agents walk, e.g. classic5. react.build draws each agent's cycle from those; given none, from all three, so the recorded ReAct race's agents are unchanged (same workload, byte for byte). A race records its cycles in its workload. The state machine is redrawn for classic5 alone, the agents the next ReAct race runs: think -> prepare_call -> act -> observe -> reflect, and back to think for the next cycle; END when the agent's tokens are done. The step diagram draws a recorded classic5 agent's first cycle (agent 37). Tests: classic5 agents walk only its steps, in its order, with the same lengths and prompts as before; an unknown cycle is refused; the race records its cycles and keeps the ReAct race's place on the page. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The ReAct race is now run on the classic5 cycle alone (think, prepare_call, act, observe, reflect, and around again): 64 agents per lane, 16 at a time, 128 to 1,024 tokens (16 each), within one datacenter, recorded on one A100 at b7e5421: gs-sr-vllm all agents done in 90.4 s; 743 switches over 170 requests; 957 prompt tokens re-prefilled, context hit 99.8% lora-vllm all agents done in 282.4 s; 743 switches over 892 requests; 3,054,156 re-prefilled, context hit 16.6% SR finished 3.1x sooner; by length, stock vLLM's median agent took 2.2x SR's at 128 tokens and 3.2x at 1,024. It replaces the race on all three cycles, under the same key. The pod's agents are those react.py builds locally. The SR checkpoint passed verify_composed.py; KV demand stayed at 35% of capacity or less. The page now opens on the demo only: a sentence on what races, the outcome ("granite-switch SR finished 3.1x sooner", from the chosen race), and the race, starting on the ReAct race. Everything else is behind buttons: How it works the experiment, the state machine, the step diagram, rtt The models each lane's setup: base model, 64 adapters, rank 32 (alpha 64), the modules they adapt Metrics the panels More races the switching block's four races The models panel's facts come from the run's own log and the base model's config. AGENT_RACE.md describes the layout and the classic5 race. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…per lane
The page's demo, sharpened:
- The chosen race's settings in a card under the picker, one labeled line
each: agent architecture (ReAct and its cycle, or fixed steps), # agents,
concurrency, tokens per agent, adapters, network, and the run (model,
GPU, vLLM, commit, date). The picker names the races shortly instead:
"ReAct", or "Fixed steps" with their sizes, behind More races.
- Three tiles under the headline, from the race's own counts: prompt
tokens computed again (957 vs 3.05M), requests (170 vs 892), and the
context already cached (99.8% vs 16.6%), each with why.
- A track per lane above the agents, filling as its agents finish: SR's
is full at 1:30, stock vLLM's at 4:42.
- Adapter switching (rtt) in purple (#9575cd), on the page and in the
diagrams. Its cyan was 9.5 apart from generation's blue (OKLab x100),
hard to tell apart; purple is 20.4 apart, and every pair of the phase
colors at least 19.
The How it works setup line is gone: the card's Run row says it. The races
are unchanged.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
The race page, its diagrams and their generator now live on main
(feature/agent-race-page): a page needs nothing of this benchmark, while the
code that runs and records races runs on its harness, so the code stays
here and the page goes:
- removed: docs/agent_race.html, docs/media/agent_race_*.svg,
docs/generation/generate_race_diagram.py;
- .pre-commit-config.yaml back to this branch's own: the 500 KB size
exemption was for the page alone;
- build_page.py writes into a page given by --page, now required: a main
checkout's docs/agent_race.html. Its test builds a small page of its own;
- docs/AGENT_RACE.md is about running and recording races; what the page
shows is in main's copy.
All 464 unit tests that run without vLLM pass; job bundles 107 KB and
101 KB of 120 KB.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
Collaborator
Author
|
Closed: only PRs into main from now on. The race code landed on feature/adapter-benchmark as one commit, f0aa2c7, and the page is in PR 156 (into main). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The code that runs and records the agent race: agents that switch adapters as they work, on one GPU, served by granite-switch's Shadow Residual (
gs-sr-vllm) and by stock vLLM with the same adapters as LoRAs (lora-vllm). The race page itself, its diagrams and their generator are on main, in #156. This PR holds the code, because it runs on this branch's harness: the switch benchmark's scripts, the composer step, the switching settings and the jobs.gs-sr-vllmlora-vllmFiles
benchmarks/adapter_eval/race/lane.pyrun_agent, unchanged), run through slots (the next agent starts as one finishes) and a recording engine (each request's start, first token, tokens, end, and vLLM's cached prompt tokens)benchmarks/adapter_eval/race/run_race.pyverify_composed.py), both lanes on one GPU one after the other, each after an untimed warm-up; prints the race into the job's logbenchmarks/adapter_eval/race/react.pybenchmarks/gen_agent_configs.py, copied unchanged)benchmarks/adapter_eval/race/build_page.py--page: a main checkout'sdocs/agent_race.html. Refuses a race that carries an absolute pathbenchmarks/adapter_eval/vela/render_job.pydocs/AGENT_RACE.mdThe races recorded with it
All on the page in #156. Granite 4.1 3B at b7e5421, one A100-80GB, vLLM 0.26.0:
Each is one
submit.sh script <commit> benchmarks/adapter_eval/race/run_race.py --extra-args "..."job;docs/AGENT_RACE.mdhas every race's options.Job size
Every job ships the benchmark's code inside itself, within a 120 KB limit. The race's code goes only with the race's own jobs, in the second bundle each job carries, where there is room:
Testing
tests/unit/test_agent_race.py(22 tests) runs the race end to end on CPU, with the real workload generators and agent loop and a fake engine: slots, the recorded phases and prefill accounting, equal switch counts on both lanes, the race options and the mixed workload, races kept per configuration in a page, the ReAct agents (the generator's steps, cycles, lengths and switch rates; agents walking only the cycles asked for).tests/unit/test_adapter_benchmark.py: a race job carries the race's code, and any other job does not.feature/adapter-benchmarkat 021927c; GitHub merges it cleanly onto the branch's current head.