Skip to content

Read the composite propagator's carrier cache in place - #13606

Draft
dougqh wants to merge 1 commit into
dougqh/kafka-header-keysfrom
dougqh/kafka-carrier-cache
Draft

dougqh wants to merge 1 commit into
dougqh/kafka-header-keysfrom
dougqh/kafka-carrier-cache

Conversation

@dougqh

@dougqh dougqh commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Extraction stops copying the carrier's headers twice. The composite propagator's cache becomes a small indexed array, and the tracing extractor reads it in place instead of copying every pair into a second list. Extracted contexts are unchanged.

Stacked on #13605.

Motivation

On a Kafka consumer, after #13600 removed RecordHeader's copies, the two caches were the next-largest extraction allocation: an ArrayList(32) each per record, about 3 GiB per profiled run (~12% of what the tracer still allocated), and their replay loops ~7–12% of the consumer thread's tracer time.

Additional Notes

  • IndexedCarrier (new, components/context): read-only size() / key(i) / value(i) over pairs already in memory, valid only for the extraction it's handed to.
  • CompositePropagator.CarrierCache implements it over a String[] sized for 8 pairs, growing on demand, instead of an ArrayList(32); with key filtering a Kafka record keeps a handful of headers.
  • HttpCodec.ExtractionCache reads an IndexedCarrier in place (computing the Datadog span id hex by one scan, as before); any other carrier is still copied.
  • Tests: CarrierCacheTest (indexed reads, growth, visiting order). CompoundExtractionGoldenTest extracts through the composite propagator, so it covers the in-place read against its golden file.
  • Numbers (with Skip Kafka headers no propagator reads, before decoding them #13605, laptop Kafka macrobenchmark, details there): consumer flat, enricher +3.6%. A JFR allocation profile of the consumer shows ExtractionCache.<init> gone (was 1.39 GiB per run) and CarrierCache.<init> down from 1.60 to 1.03 GiB; tracer-attributed allocation per run 24.2 → 22.0 GiB (−9%).

🤖 Generated with Claude Code

@dougqh dougqh added comp: core Tracer core tag: performance Performance related changes type: refactoring tag: ai generated Largely based on code generated by an AI or LLM labels Nov 21, 2026
@datadog-datadog-us1-prod

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.89 s 14.84 s [-0.5%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 13.67 s 13.91 s [-2.6%; -0.8%] (maybe better)
startup:petclinic:appsec:Agent 17.13 s 17.04 s [-0.2%; +1.2%] (no difference)
startup:petclinic:iast:Agent 17.02 s 16.47 s [-1.3%; +8.1%] (no difference)
startup:petclinic:profiling:Agent 16.51 s 16.79 s [-2.4%; -1.0%] (maybe better)
startup:petclinic:sca:Agent 17.15 s 17.00 s [+0.1%; +1.6%] (maybe worse)
startup:petclinic:tracing:Agent 16.21 s 16.32 s [-1.4%; +0.2%] (no difference)

Commit: 5bc558da · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@pr-commenter

pr-commenter Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch dougqh/kafka-header-keys dougqh/kafka-carrier-cache
git_commit_date 1795289298 1795289307
git_commit_sha d78803d 5bc558d
See matching parameters
Baseline Candidate
ci_job_date 1795291161 1795291161
ci_job_id 2286653005 2286653005
ci_pipeline_id 155480154 155480154
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-4wqp8x47 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-4wqp8x47 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce unsure
[+543.302op/s; +9179.074op/s] or [+0.329%; +5.563%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter

pr-commenter Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch dougqh/kafka-header-keys dougqh/kafka-carrier-cache
git_commit_date 1795289298 1795289307
git_commit_sha d78803d 5bc558d
See matching parameters
Baseline Candidate
ci_job_date 1795291087 1795291087
ci_job_id 2286653007 2286653007
ci_pipeline_id 155480154 155480154
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-ruxvrllx 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-ruxvrllx 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

CompositePropagator.CarrierCache now implements a new IndexedCarrier
(size/key/value by index) over a String[] sized for the handful of
headers left after key filtering, growing on demand, instead of an
ArrayList(32). HttpCodec.ExtractionCache reads an IndexedCarrier in place
rather than copying every pair into a second ArrayList(32); other
carriers are still copied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/kafka-carrier-cache branch from 8131802 to 5bc558d Compare November 21, 2026 19:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant