[sentry-junior]

Backend shards spend a large part of their time starting ClickHouse, Kafka and Snuba and waiting for them to be healthy, and ClickHouse keeps using CPU during tests. This lets full runs skip those services on shards whose test files don't use them. Measurements are in the comments.

Design

• Full runs split their 30 shards. The first BACKEND_SNUBA_SHARDS (23) keep devservices backend-ci. The rest are "lite": they use the new backend-ci-lite mode (postgres, redis, bigtable, redis-cluster, objectstore) and skip the per-worker Snuba bootstrap.
• pytest-selected.py assigns files before any test is imported, using backend-test-services.json.
  • Snuba shards get files that used Snuba, ClickHouse, Kafka or Symbolicator, files missing from the inventory, and files with # backend-ci: snuba or Relay/Symbolicator hints. Lite shards get the rest.
  • Each group is balanced by duration, and a misconfigured shard fails instead of falling back.
• pytest_service_guard.py watches socket connects and sends to the service ports, so it also sees pooled connections, plus Kafka client calls.
  • Snuba shards record per-file usage.
  • Lite shards fail any test that touches a service, even when the code swallows the error, and any test that needs Relay or Symbolicator fixtures.
• Selective PR runs are unchanged.

Inventory

Guarded full runs upload backend-service-usage-* artifacts, which backend-test-services.py merges. A file needs Snuba if any shard saw it use a service. A file that ran on a lite shard without failing is Snuba-free, because lite shards enforce. With BACKEND_SNUBA_SHARDS at 0, every shard keeps every service and only records usage.

Known gaps

• Relay and Symbolicator containers talk to Kafka themselves, so their tests are routed by source hints.
• A background thread that outlives its test can pin its usage on the next test.
• PR runs are selective and unguarded, so a file that starts using Snuba first fails a lite shard on master.

Open questions

• Is the saving worth this machinery? Writing group attributes to Snuba on every Group save looks like the main reason files need Snuba, so skipping that in tests that don't read it could move many files to lite shards.
• Should selective PR runs enforce the inventory?
• Should the inventory be refreshed together with backend-test-durations.json?

via @joshuarli.

--

View Session [Sentry]

[sentry-junior]

Measurements

All runs below are full backend suites (backend.yml, 30 shards). This PR's runs used Arm PR runners, and every shard reported the same CPU (arm implementer=0x41 part=0xd49), so the CPU-model grouping has only one row per run. Master runs on x64, so it's a reference, not a like-for-like comparison. Seconds are per shard, shown as median/max. "Pre-test" is the time from job start to the pytest step.

Run	Shards	Wall	Pre-test	Setup sentry env	Test step	Total runner time
43539505183: all services, record-only (baseline)	30	258/285	93/101	80/88	154/175	7,775s
43541416173: 21 Snuba + 9 lite	21 Snuba	263/299	95/106	82/92	160/182
	9 lite	185/208	28/38	21/24	147/172	7,285s (−6.3%)
43542483015: 23 Snuba + 7 lite, all green	23 Snuba	249/289	96/106	79/90	140/177
	7 lite	224/250	30/35	21/22	177/205	7,349s (−5.5%)

43541416173 had one failure on a lite shard, from a nested pytest-selected.py test that inherited the shard's color (fixed since). The guard caught nothing in that run.

Master reference, x64 (43535410847, 30 shards, all services):

CPU	Shards	Wall	Pre-test	Setup sentry env	Test step
AMD EPYC 7763	6	386/397	122/128	103/105	258/268
AMD EPYC 9V45	8	266/288	102/105	89/92	156/179
AMD EPYC 9V74	8	346/384	122/131	108/110	210/251
Intel Xeon Platinum 8573C	6	342/361	124/128	107/110	212/223
Intel Xeon 6973P-C	1	277	106	91	163
Intel Xeon Platinum 8370C	1	360	124	106	229

43537601902 is similar: median wall 335s, max 406s, pre-test 116s.

What needs Snuba

• 783–792 of the 2,637 measured test files used Snuba, ClickHouse, Kafka or Symbolicator. Those files are ~64% of test-seconds by backend-test-durations.json, so 23 of the 30 shards need the full stack.
• tests/snuba/ is only part of it: 79 of its 84 files need Snuba. About 713 files elsewhere need it too, mostly in api, issues, replays, seer and integrations.
• This is an inference from one shard's logs: lite shard 19 in 43540389481, which ran an inventory that missed requests on pooled connections. Most refused requests were the group-attributes snapshot written on Group save:
  • /tests/entities/group_attributes/insert: 3,740
  • /tests/events/eventstream (from store_event): 510
  • Snuba queries and subscriptions: fewer than 70

Takeaways

• Lite shards spend ~65s less before pytest, and their tests run without ClickHouse competing for CPU.
• Only 7–9 shards can be lite, so the saving is ~5–6% of runner time. The slowest shard is unchanged (285s → 289s), because Snuba shards stay on the critical path.
• The guard works. One run's inventory had missed requests on pooled urllib3 connections (it only watched connects). The lite shards failed 407 files loudly instead of passing, which is what exposed the gap.