Repository navigation
Support S3 and Azure destinations for benchmark results - #3814
Draft
Arunkumar Eli (arun-elr) wants to merge 1 commit into
Draft
Arunkumar Eli (arun-elr) wants to merge 1 commit into
Arunkumar Eli (arun-elr) wants to merge 1 commit into
Conversation
Use obstore for runner uploads and result retrieval while preserving the existing run layout. Keep provider credentials in the environment and propagate upload failures. Signed-off-by: Arunkumar Eli <elrarun@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #3811 for the runners currently on main.
Use one obstore module for benchmark artifact uploads, listing, and reads. Locust and Nighthawk ingress accept S3, GCS, Azure, or local destinations and keep the existing run layout. The development script and retrieval examples use the same module; runner images no longer install google-cloud-storage.
The evaluation reader in #3552 and the egress runner in #3708 can adopt this module before they merge. They are not copied into this branch. Azure URLs use
az://container/prefix, with the account configured separately through obstore's environment settings.Validation: local/in-memory and Azure URL tests; 104 Locust tests and 14 Nighthawk tests, including upload-failure exit paths; a RustFS S3 upload/list/read check and missing-bucket failure; Locust Python dependency image build.
hack/verify-all.shpasses, including Python tests, licenses, code generation, and shell checks. No live GCS/Azure or deployed benchmark run yet.AI use: Codex assisted with implementation, tests, and independent review.