feat(upload): split profile archive uploads into concurrent parts

[lvaroqui]

Upload profile archives of 64 MiB and more as S3 multipart uploads, with parts sent concurrently.

S3 rejects a single upload request above 5 GiB, so large walltime and memory profile archives could not be uploaded (walltime folders above 5 GiB were gzipped on disk to try to fit). Even below that limit, a single connection to S3 only reaches about 20-25 MiB/s on GitHub-hosted runners, so a 1 GiB archive took close to a minute to upload.

How it works

• Part layout: archives of 64 MiB and more are split in about two parts per concurrent upload, each between 16 MiB and 256 MiB, so a slow part doesn't leave the other upload slots idle at the end. The md5 of every part is computed in the same pass over the archive as the archive md5, and sent in the upload metadata (bumped to version 12) as profileMultipart (size, partSize, partMd5s).
• Upload: the upload endpoint answers such a request with multipartUploadUrls (partUrls, completeUrl) instead of uploadUrl. Each part URL is presigned with the part's Content-MD5, so S3 checks every part. Parts are uploaded 8 at a time, each with its own retries, then the ETags are sent in part order to the completion URL. S3 can report a failed completion with a 200 status, so the response body is also checked for an error.
• Both archive kinds: on-disk (walltime, memory) archives are streamed from disk, and in-memory gzip (simulation) archives are sliced without copying.
• S3 specifics (request headers, ETag, completion body) live in the new upload::s3 module.
• CODSPEED_UPLOAD_CONCURRENCY overrides the number of concurrent part uploads.

Archives below 64 MiB keep the single upload request, and their metadata is unchanged apart from the version.

Other changes

• Walltime profile folders above 5 GiB are no longer gzipped on disk, and the maximum archive size goes from 5 GiB to 15 GiB.
• Archives are hashed while streaming on the blocking thread pool, instead of being read whole into memory on the async runtime.
• Single-request uploads of in-memory archives now go through the same retry loop as on-disk ones.

Measurements

Concurrency sweep on a 6 GiB walltime archive (25 parts of 256 MiB), which led to the default of 8:

Concurrency	ubuntu-latest	CodSpeed macro runner (Ryzen 9950X)
1	240.3s · 25.6 MiB/s	59.5s · 103.2 MiB/s
2	83.4s · 73.7 MiB/s	33.9s · 181.4 MiB/s
4	57.2s · 107.4 MiB/s	21.3s · 289.0 MiB/s
8	45.2s · 135.8 MiB/s	8.2s · 746.5 MiB/s
16	48.7s · 126.1 MiB/s	10.3s · 596.5 MiB/s

Smaller archives with the final part layout, concurrency 1 (close to the previous single request) vs 8:

Archive	Runner	Concurrency 1	Concurrency 8
256 MiB	ubuntu-latest	11.6s	2.8s (4.1×)
1 GiB	ubuntu-latest	55.1s	9.0s (6.1×)
256 MiB	macro runner	2.7s	1.3s (2.1×)
1 GiB	macro runner	10.1s	1.6s (6.3×)

The backend support for profileMultipart is not released yet, so the multipart path cannot be verified end to end against production for now.

Closes COD-3700

[codspeed]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 31 untouched benchmarks
⏩ 6 skipped benchmarks¹

---

_{Comparing cod-3700-multipart-upload (148e6dd) with main (8c84fcc)}

Footnotes

1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 5/5

[High risk] Changes how profile archives are uploaded to cloud storage.

The PR appears safe to merge; the remaining new concern is non-blocking error reporting.

Findings

1.  Failed reads hide the reason ▶

Fix with agent prompt

### Issue 1
src/upload/uploader.rs:242-246
If reading a failed upload response times out or fails, this fallback replaces the read error with an empty string. The reported error then says `Reason:` with no explanation, making it harder to distinguish a stalled S3 response from one that had no body.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR adds concurrent multipart uploads for large profile archives, including per-part hashes, streamed upload bodies, completion handling, and tests. Since the previous review, it bounds failed-response body reads and distinguishes transient from permanent completion errors.

_{Reviews (4) · Last reviewed commit: "feat(upload): split profile archive uplo..." · Reviewed by Greptile}

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Start a greploop in Claude Code and it will work through the open comments and keep going until this PR reviews clean.

[adriencaccia]

Seen together, let's remove the single part upload path and always use multipart, albeit with a single part for archives under 64MiB.