Skip to content

benchmarking/nighthawk-egress: size the rig so atenet-egress is the bottleneck under test - #3627

Open
Yuan Gao (ygao-g) wants to merge 4 commits into
agent-substrate:mainfrom
ygao-g:nighthawk-egress-1-sizing
Open

Yuan Gao (ygao-g) wants to merge 4 commits into
agent-substrate:mainfrom
ygao-g:nighthawk-egress-1-sizing

Conversation

@ygao-g

@ygao-g Yuan Gao (ygao-g) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #3624: size the rig so atenet-egress is the bottleneck under test.

Before this PR, the benchmark ran at a fixed request rate. The later PRs of #3624 measure the highest rate that atenet-egress holds, with the gateway as the system under test. So the load generator, which runs inside an actor, and the origin server should get more CPU and memory.

Measured on a GKE cluster with 4 c3-standard-22 nodes and envoy of atenet-egress at 1 vCPU. In each of the modes http, mitm and passthrough, envoy reached its CPU limit. The worker pod of the load generator peaked at 12 of its 18 vCPUs, and nighthawk-server at 2.1 of its 8. The run used the search of those later PRs.

AI use: AI-assisted for the diff, the split and the test runs; I reviewed the diff.

  • Tests pass
  • Docs updated

@ygao-g

Yuan Gao (ygao-g) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

Reviewer notes (Stage 1 stack guide)

Stack position: Independent of #3628; both sit on main. #3629 and #3630 carry this PR's three commits after #3628's.

Commit-by-commit checks (8c06a5bc3):

  1. 05070e6c3: build_kubectl_ate moves from benchmarking/workloads/deploy.sh into benchmarking/lib/kubectl-ate.sh unchanged; both deploy scripts source it.
  2. 6ce5bd931: Check resource alignment across traffic-driver-template.yaml.tmpl (16000m / 8 GiB actor), traffic-driver.yaml.tmpl (12-18 vCPU worker pod), and nighthawk-server.yaml (--concurrency 8, 6-8 vCPU). atelet registers the pod CPU limit from the downward API (internal/ateworker/register/register.go:117), and checkRoom (cmd/ateapi/internal/scheduling/scheduling.go:321-337) admits at most one 16000m actor onto an 18000m worker pod (18000 - 16000 < 16000).
  3. 8c06a5bc3: Check that deploy-traffic-driver.sh sources the helper and builds kubectl-ate before its first kubectl ate call, in deploy and in delete.

Behavior impact:

  • Default CPU and memory requests target multi-node clusters (c3-standard-22) and no longer fit a single-node kind cluster without lowering requests first.

@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-1-sizing branch from d847d8c to 8ac15c2 Compare November 5, 2026 19:01
@ygao-g
Yuan Gao (ygao-g) marked this pull request as draft November 5, 2026 19:10
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-1-sizing branch 2 times, most recently from a70ba52 to e98a359 Compare November 5, 2026 21:19
@ygao-g
Yuan Gao (ygao-g) marked this pull request as ready for review November 5, 2026 21:28
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-1-sizing branch 2 times, most recently from 61962aa to 507ece3 Compare November 5, 2026 22:14
@ygao-g
Yuan Gao (ygao-g) marked this pull request as draft November 5, 2026 22:34
@ygao-g
Yuan Gao (ygao-g) force-pushed the nighthawk-egress-1-sizing branch 2 times, most recently from 61962aa to 8c06a5b Compare November 5, 2026 22:53
@ygao-g
Yuan Gao (ygao-g) marked this pull request as ready for review November 6, 2026 00:12
Move the function into benchmarking/lib/kubectl-ate.sh for reuse.

Signed-off-by: Yuan Gao <ypgao@google.com>
…ration

Give the actor sandbox, its worker pod and nighthawk-server
enough CPU and memory that the gateway saturates first.

Signed-off-by: Yuan Gao <ypgao@google.com>
deploy-traffic-driver.sh builds kubectl-ate from the tree into a
temporary directory.

Signed-off-by: Yuan Gao <ypgao@google.com>
With no context set, "${args[@]}" is an unbound variable under the macOS
system bash, and the EXIT trap of build_kubectl_ate made the script exit
0 after that error. The array now expands only when it has elements, as
in hack/run-e2e.sh.

Signed-off-by: Yuan Gao <ypgao@google.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants