Skip to content

perf(ci): use ARM runners for builds and selected tests - #26106

Merged
nicohrubec merged 10 commits into
developfrom
feat/benchmark-arm-ci
Nov 6, 2026
Merged

nicohrubec merged 10 commits into
developfrom
feat/benchmark-arm-ci

Conversation

@nicohrubec

@nicohrubec nicohrubec commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Today our CI uses x86 runners for all our job, this PR moves a selection of our CI runners to the free ARM64 runners to reduce CI time. The jobs that are moved over were determined by doing a few benchmark runs with more details in the tables below.

x86 baseline is our CI as it runs today. The ARM run then uses ARM runners as much as possible with the caveat that we cannot move node/bun/deno integration tests to ARM atm because they run SQL Server container tests that do not work on ARM (potentially something we can look into next if we go with this). Running on ARM improves many jobs a lot but is worse for browser and E2E tests, partially because of missing cache hits. That's how I ended up with the "hybrid" configuration that basically takes best of both worlds.

Configuration Required CI time* vs x86
x86 baseline 10m42s —
ARM trial 9m47s 55s faster (8.6%)
Hybrid 9m31s 71s faster (11.1%)

In the table below you can see in more detail how the job timings changed between the configurations for the latest test run:

Job / group x86 ARM trial
Required CI completion* 10m42s 9m47s
Build 1m58s 1m54s
Build bundles 2m52s 2m21s
Build tarballs 3m59s 2m56s
Lint 2m19s 1m40s
Size check 2m22s 1m29s
Node unit tests 4m28s 3m14s
Bun unit tests ~56s 47s
Deno unit tests 1m18s 1m02s
Bundler integration 1m44s 1m41s
Browser unit tests 2m44s 2m52s
Browser Playwright 2m40s 3m47s
Browser loader 1m22s 1m48s
E2E 1m07s 1m35s
Node integration† 4m54s 4m46s
Bun integration† 4m31s 4m48s
Deno integration† 4m15s 4m39s

* Excludes initial queueing and optional jobs. Each run had one required-test failure; ARM browser caches were cold.
† These integration jobs stayed on x86 in both runs.

Co-Authored-By: OpenAI Codex <codex@openai.com>
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

size-limit report 📦

Compared: ba36175 (PR test merge using develop at a2dbec0)

Against: eadbabb (develop) · base run #65073

⚠️ Baseline mismatch: 3 develop commit(s) in the PR build are missing from the baseline; 3 develop commit(s) in the baseline are missing from the PR build. Their size changes also affect this comparison. Compare commits. Rebase your PR onto the latest develop to compare against an up-to-date baseline.

Path Size % Change Change
@sentry/browser 30.4 kB - -
@sentry/browser - with treeshaking flags 28.54 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 28.43 kB - -
@sentry/browser (incl. Tracing) 52.73 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 52.73 kB -0.01% -1 B 🔽
@sentry/browser (incl. Tracing, Profiling) 55.7 kB - -
@sentry/browser (incl. Tracing, Replay) 93.03 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 81.63 kB -0.01% -1 B 🔽
@sentry/browser (incl. Tracing, Replay with Canvas) 97.78 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 110.77 kB - -
@sentry/browser (incl. Feedback) 47.98 kB - -
@sentry/browser (incl. sendFeedback) 35.5 kB - -
@sentry/browser (incl. FeedbackAsync) 40.61 kB - -
@sentry/browser (incl. Metrics) 31.44 kB - -
@sentry/browser (incl. Logs) 31.73 kB - -
@sentry/browser (incl. Metrics & Logs) 32.45 kB - -
@sentry/react 32.29 kB - -
@sentry/react (incl. Tracing) 55.1 kB -0.01% -1 B 🔽
@sentry/vue 38.67 kB - -
@sentry/vue (incl. Tracing) 55.75 kB -0.01% -1 B 🔽
@sentry/svelte 30.43 kB - -
@sentry/remix (Remix 3 client bundle) 57.82 kB - -
CDN Bundle 32.19 kB -0.01% -1 B 🔽
CDN Bundle (incl. Tracing) 53.26 kB - -
CDN Bundle (incl. Logs, Metrics) 34.47 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 55.2 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 75.88 kB - -
CDN Bundle (incl. Tracing, Replay) 91.47 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 93.4 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 97.58 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 99.58 kB - -
CDN Bundle - uncompressed 94.67 kB - -
CDN Bundle (incl. Tracing) - uncompressed 157.67 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 101.14 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 163.41 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 232.71 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 279.36 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 285.08 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 293.06 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 298.77 kB - -
@sentry/nextjs (client) 58.05 kB - -
@sentry/sveltekit (client) 53.24 kB - -
@sentry/core/server 44.25 kB - -
@sentry/core/browser 13.93 kB - -
@sentry/node 155.97 kB +0.02% +28 B 🔺
@sentry/node/import (ESM hook with diagnostics-channel injection) 82.28 kB - -
@sentry/node - without tracing 96.92 kB +0.03% +23 B 🔺
@sentry/node - without channel injection 136.29 kB +0.02% +19 B 🔺
@sentry/aws-serverless 105.12 kB +0.03% +22 B 🔺
@sentry/cloudflare (withSentry) - minified 200.67 kB - -
@sentry/cloudflare (withSentry) 499.05 kB - -
@sentry/nextjs/cloudflare (withSentry) - minified 222.67 kB - -

View base workflow run

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Optional E2E tests

Some optional checks were not successful. They do not block the merge. 6 not successful, 50 successful. See the run.

Check Status
❌ CI: Optional E2E Tests / E2E cloudflare-pi-durable (latest) Test (optional) Failing after 43s Details
❌ CI: Optional E2E Tests / E2E node-eve (latest) Test (optional) Failing after 51s Details
❌ CI: Optional E2E Tests / E2E node-eve Test (optional) Failing after 2m Details
❌ CI: Optional E2E Tests / E2E node-eve (orchestrion) Test (optional) Failing after 3m Details
❌ CI: Optional E2E Tests / E2E nextjs-16-cacheComponents (cloudflare) Test (optional) Failing after 10m Details
❌ CI: Optional E2E Tests / E2E node-flue (latest) Test (optional) Failing after 39s Details
Successful checks
Check Status
✅ CI: Optional E2E Tests / E2E cloudflare-workers-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E astro-7 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-mastra Test (optional) Successful in 51s Details
✅ CI: Optional E2E Tests / E2E nextjs-16-tunnel (cloudflare) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-agent Test (optional) Successful in 42s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (deno) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E deno (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-sourcemaps Test (optional) Successful in 39s Details
✅ CI: Optional E2E Tests / E2E astro-7-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-think Test (optional) Successful in 57s Details
✅ CI: Optional E2E Tests / E2E solidstart-2 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps Test (optional) Successful in 40s Details
✅ CI: Optional E2E Tests / E2E cloudflare-think (ai v6) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E supabase-nextjs Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E cloudflare-flue-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E react-router-7-framework-instrumentation Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E debug-id-sourcemaps Test (optional) Successful in 37s Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-cloudflare Test (optional) Successful in 52s Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (latest) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable Test (optional) Successful in 58s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (static / nuxt generate) Test (optional) Successful in 55s Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E vue-3 (no Options API) Test (optional) Successful in 8m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (cloudflare, latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (cloudflare) Test (optional) Successful in 56s Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable Test (optional) Successful in 41s Details
✅ CI: Optional E2E Tests / E2E react-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (turbo) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext, bun) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v1 Test (optional) Successful in 52s Details
✅ CI: Optional E2E Tests / E2E nextjs-orpc Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-14 (latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5 Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v2 Test (optional) Successful in 49s Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 Test (optional) Successful in 55s Details
✅ CI: Optional E2E Tests / E2E node-mastra (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable (latest) Test (optional) Successful in 48s Details
✅ CI: Optional E2E Tests / E2E node-flue Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 (tracer provider) Test (optional) Successful in 42s Details
✅ CI: Optional E2E Tests / E2E node-mastra Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (client source maps kept) Test (optional) Successful in 37s Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions (latest) Test (optional) Successful in 54s Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions Test (optional) Successful in 40s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (tunnel-dev) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (latest) Test (optional) Successful in 57s Details

nicohrubec and others added 9 commits November 4, 2026 15:36
Keep branch builds and SQL Server integration jobs on x86-64, and isolate architecture-specific caches.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
@nicohrubec
nicohrubec marked this pull request as ready for review November 5, 2026 13:21
@nicohrubec
nicohrubec requested review from a team as code owners November 5, 2026 13:21
@nicohrubec
nicohrubec requested review from Lms24, andreiborza, chargome, isaacs, logaretm and s1gr1d and removed request for a team November 5, 2026 13:21
@nicohrubec nicohrubec changed the title ci: benchmark GitHub ARM64 runners ci: use ARM runners for builds and selected tests Nov 5, 2026
@nicohrubec
nicohrubec added this pull request to stack #26149 November 5, 2026 14:29
@nicohrubec nicohrubec changed the title ci: use ARM runners for builds and selected tests perf(ci): use ARM runners for builds and selected tests Nov 6, 2026
@nicohrubec
nicohrubec removed this pull request from stack #26149 November 6, 2026 09:48
@nicohrubec
nicohrubec added this pull request to stack #26166 November 6, 2026 09:48
@nicohrubec
nicohrubec removed this pull request from stack #26166 November 6, 2026 10:30
@nicohrubec
nicohrubec added this pull request to stack #26168 November 6, 2026 10:30
@nicohrubec
nicohrubec merged commit b0128dc into develop Nov 6, 2026
418 of 424 checks passed
@nicohrubec
nicohrubec deleted the feat/benchmark-arm-ci branch November 6, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants