Skip to content

benchmarking: add the nighthawk-egress adaptive runner and test type - #3708

Draft
Yuan Gao (ygao-g) wants to merge 15 commits into
agent-substrate:mainfrom
ygao-g:nighthawk-egress-b2-runner
Draft

benchmarking: add the nighthawk-egress adaptive runner and test type#3708
Yuan Gao (ygao-g) wants to merge 15 commits into
agent-substrate:mainfrom
ygao-g:nighthawk-egress-b2-runner

Conversation

@ygao-g

@ygao-g Yuan Gao (ygao-g) commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The egress gateway has a manual Nighthawk setup, but no benchmark that searches for its capacity. The new benchmark finds the highest request rate that one gateway replica holds within a 25 ms latency SLO. It does this for each egress mode, at the CPU count that the suite sets. The load starts inside an actor, so each request takes the same path as the traffic of a real actor.

A runner drives one adaptive Nighthawk run and reads the gateway counters before and after it. It writes capacity.json with the result and the counter checks. The benchmark automation gets a nighthawk-egress test type and three suites: egress_gatewaycap_http_1cpu, egress_gatewaycap_mitm_1cpu and egress_gatewaycap_passthrough_1cpu. Before each suite, the test type sets the gateway to one replica and to the CPU count of the suite. run-dev.sh runs one suite from a working tree.

The runner uploads the artifacts and then exits 1 in these cases:

  • The run gives no valid result.
  • A gateway pod changes or restarts during the run, or the second counter read fails.
  • The counters show load outside the filter chain of the mode.

Effects outside the egress benchmark:

  • The scheduled benchmark job runs the three new suites.
  • A test type can now deploy and delete its own workloads in place of the shared workloads. The locust and nighthawk-ingress test types keep their behavior.
  • The runner reaches the load actor through the ingress router as the traffic-driver invoker. The automation adds the SPIFFE id of the invoker to the ingress allow list only when it deploys Substrate for an egress suite.
  • The egress policy of the benchmark names the Nighthawk server host in place of *. A kubectl-ate test reads that file, so its expected value changes.
  • The run-dev.sh script of nighthawk-ingress now continues after a failed kubectl or gcloud read. Before, one failed read ended the script without a verdict.

Not handled yet:

  • A 2 vCPU gateway with one actor. Envoy pins each of the actor's 3 to 5 tunnel connections to one worker thread.
  • The microVM runtime. The test type accepts only the gVisor sandbox class.

The three suites ran once through the orchestrator on GKE, on 4 x c3-standard-22 nodes, and mitm ran twice through run-dev.sh. All five runs completed, and every gateway check was true. Through the orchestrator, slo_max_rps was 8,240 for http, 9,732 for mitm and 43,184 for passthrough. Through run-dev.sh, it was 8,431 and 9,307 for mitm. The second run-dev.sh call patched a gateway that already had the right shape, and the gateway did not restart.

Part of #3624

AI use: AI-assisted for the diff, the split and the test runs; I reviewed the diff.

  • Tests pass
  • Docs updated

@ygao-g
benchmarking/nighthawk_common: move the Nighthawk output parser out o…
833a874 
…f nighthawk-ingress

Move nighthawk-ingress/output.py to nighthawk_common unchanged so
nighthawk-egress can reuse it. The ingress Dockerfile copies the package
into the image and the ingress tests put benchmarking/ on sys.path.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk_common: move the adaptive session spec builder…
7deee2e 
… out of nighthawk-ingress

Move nighthawk-ingress/spec.py to nighthawk_common unchanged and point its importers at the new path. actors.py imported it without using it.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-ingress: keep the router-specific spec parts o…
5d125f1 
…ut of nighthawk_common

build_session_spec in nighthawk_common now takes a finished traffic template. The router SNI, the certificate paths and build_spec_dict, which assembles that template, move back to nighthawk-ingress/spec.py.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g Yuan Gao (ygao-g) self-assigned this Nov 6, 2026
This was referenced Nov 6, 2026
Draft
Open
Closed
Closed
Open
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-b2-runner branch 2 times, most recently from 384d7d2 to 62dcc31 Compare November 6, 2026 17:51
This was referenced Nov 6, 2026
Open
Open
Open
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-b2-runner branch 6 times, most recently from b88f3c5 to 6f0b791 Compare November 7, 2026 17:03
@ygao-g
benchmarking/nighthawk_common: compute stage rates from exact duration
af0a3db 
Nighthawk divides achieved-rps and send-rate by execution_duration
truncated to whole seconds, so a stage that overran its nominal length
by a fraction of a second read high, by up to 1/N for an N-second
stage. stage_rows() now computes both from the exact duration and keeps
Nighthawk's values under the *_cpp keys. Stage verdicts come from
Nighthawk's threshold scores and do not change. The nighthawk-ingress
README notes how far runs recorded before this change read high.

Each stage row also records the queue_to_connect and sequencer.blocking
statistics and the latency pstdev, so a client-side stall can be told
apart from gateway latency. The tests for the new behavior live in
benchmarking/nighthawk_common/tests; the ingress suite is unchanged.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk_common: export the helpers the egress parser uses
9e2e162 
Rename _builtin_key, _global_result and _counters to builtin_key,
global_result and counters, because nighthawk-egress/output.py imports
them in the next commit. capacity_summary() takes actors as an optional
argument and emits it only when given, since the egress benchmark has no
actor fleet. build_session_spec() treats a tail_latency_slo_ms of 0 as
disabled, as the ingress README documents.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-b2-runner branch 3 times, most recently from 918b944 to 804be78 Compare November 10, 2026 19:52
@ygao-g Yuan Gao (ygao-g) changed the title benchmarking: add the nighthawk-egress adaptive runner and CI suites benchmarking: add the nighthawk-egress adaptive runner and test type Nov 10, 2026
@ygao-g

Yuan Gao (ygao-g) commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Stack: #3628, #3711, #3707 and #3627, then this PR. #3627 can merge in any order with the other three. Review the last five commits. The diff of this PR alone is at https://github.com/ygao-g/substrate/pull/87/files

Suggested review order:

  1. The runner, its image, the manifests and test_runner.py.
  2. The test type, the orchestrator changes, tests.yaml and run-dev.sh.
  3. The two READMEs.
  4. The fourth commit: the scrape on both IP families, the kept artifacts and the exit rule on the two gateway checks.
  5. The last commit: the nighthawk-ingress run-dev.sh gets the guards that the egress script has.

hack/verify/benchmarking-python.sh passes at the head and at each of the three commits before it. At the head: nighthawk-egress 10 passed and 33 subtests passed, automation 70 passed and 26 subtests passed, nighthawk-ingress 12 passed, nighthawk_common 2 passed. hack/verify/shellcheck.sh passes at the head.

The GKE runs in the description are from 2026-11-10. After the runs, one suite was removed from tests.yaml, with its test assertion and two README lines, and the nighthawk-ingress run-dev.sh got its fix. The code that the three suites use did not change. The fix in the nighthawk-ingress script has no live run yet.

@ygao-g
benchmarking/nighthawk-egress: add adaptive load spec and gateway cou…
47b28b5 
…nter parser

Define the egress-specific failure predicates and Nighthawk template
for the adaptive load search, targeting the atenet-egress gateways.
Add the prometheus output parser for the egress metrics.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
hack/verify: run the nighthawk-egress python tests
2f1e219 
benchmarking/nighthawk-egress/tests covers spec.py and output.py. Add
the suite to benchmarking-python.sh so it runs with the other
benchmarking suites.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking: share build_kubectl_ate between the deploy scripts
19bbe18 
benchmarking/workloads/deploy.sh built kubectl-ate inline, and the
nighthawk-egress deploy script was about to carry a copy. Move the
function into benchmarking/lib/kubectl-ate.sh for both to source.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-egress: size actor and server for gateway satu…
dbd85c7 
…ration

With a 2 vCPU actor sandbox and no requests on the worker pod or
nighthawk-server, nothing kept the load generator and the origin server
from saturating before atenet-egress, so a run could measure them
instead of the gateway. Give the sandbox 16 vCPUs, request 12 vCPUs for
the worker pod and 6 for nighthawk-server, run the server with
--concurrency 8, and move the example nighthawk_client run from one
worker to sixteen.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-egress: build kubectl-ate in the deploy script
f6baa38 
deploy-traffic-driver.sh required kubectl-ate on PATH, so a fresh
environment needed a manual go install step first. Source
build_kubectl_ate from benchmarking/lib/kubectl-ate.sh instead, and say
so in the README prerequisites in place of the install step.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-egress: add the adaptive capacity runner
a191af0 
runner.py drives one adaptive Nighthawk session through
traffic-driver-invoker and scrapes atenet-egress Prometheus counters
before and after it, through a new headless atenet-egress-metrics
Service that deploy.sh applies. traffic-driver-invoker gets its own
ServiceAccount so the router can allow it by name.

egress-policy.yaml names nighthawk-server with and without the
cluster.local suffix, because the actor resolv.conf sets ndots:5 and
atunnel sees the suffixed name.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/automation: add nighthawk-egress test type and run-dev.sh
5e90f2b 
A suite selects the mode and the CPU count. Before each suite, the
pre-test hook of the type sets the egress gateway to one replica and
to that CPU count. The hook then deploys the workloads of the
benchmark: the Nighthawk server, the traffic-driver invoker and the
traffic-driver actor.

The type needs two changes in the orchestrator, which runs every test
type through one loop. That loop deployed the shared workloads for
every test and deleted only those. The egress type does not use them,
so a type can now skip them and delete its own workloads. The runner
reaches the traffic-driver actor through atenet-router as the invoker.
The orchestrator adds that SPIFFE id to the ingress allow list only
when it deploys Substrate for an egress suite. The locust and
nighthawk-ingress types keep their behavior.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-egress: document the adaptive runner and suites
5afbda9 
Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-egress: scrape both IP families, keep artifacts
e56bd42 
Make the metrics scrape resolve both IPv4 and IPv6 and prefer IPv4.
Keep the Nighthawk artifacts when the gateway pod IPs change or the
post-run scrape fails.

Exit 1 when the gateway counters show that the load did not take the
filter chain of the suite's mode. mode_exclusivity_ok is false when a
filter chain of another mode recorded traffic, for example passthrough
connections in a mitm suite. ext_proc_invariant_ok is false when the
gateway did not make one ext-proc request for each tunnel and for each
inspected HTTP stream. In both cases the rate does not measure the
mode that the suite names. Until now capacity.json reported the two
checks and the runner still exited 0. The runner uploads the artifacts
before it exits.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
benchmarking/nighthawk-ingress: guard the Job status reads in run-dev.sh
3a59119 
A failed kubectl status read ended run-dev.sh without a verdict while
the runner Job still ran. A failed gcloud listing ended the script
after the Job completed, before the verdict and the Job cleanup. The
log follower outlived the script in both cases. The script now
continues after a failed read, prints a line when it finds no
capacity.json, and stops the log follower on every exit. The egress
run-dev.sh already works this way.

Signed-off-by: Yuan Gao <ypgao@google.com>
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-b2-runner branch from 68523ba to 3a59119 Compare November 11, 2026 22:42
@ridhoq Ridwan Hoq (ridhoq) mentioned this pull request Nov 11, 2026
Open
@arun-elr Arunkumar Eli (arun-elr) mentioned this pull request Nov 12, 2026
Draft
2 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@ygao-g Yuan Gao (ygao-g)

Labels

area/benchmarking area/network

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ygao-g