Skip to content

Skip Kafka headers no propagator reads, before decoding them - #13605

Draft
dougqh wants to merge 3 commits into
dougqh/kafka-header-viewfrom
dougqh/kafka-header-keys
Draft

dougqh wants to merge 3 commits into
dougqh/kafka-header-viewfrom
dougqh/kafka-header-keys

Conversation

@dougqh

@dougqh dougqh commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Propagators now declare which carrier keys they read, and the Kafka 3.8+ consumer skips every other header before decoding it: an application's own headers cost neither a key decode nor a value copy. A fetched header's key is matched against the declared keys from its bytes, once; a matching name comes back as that name's constant. Extracted contexts are unchanged.

Stacked on #13600.

Motivation

Extraction decoded every header on a record, then let each propagator ignore the ones it doesn't read. Real records usually carry application headers (content type, schema id, correlation ids) next to the trace headers, and each paid a key decode, a value copy and a value decode for nothing. Profiling also showed that #13600's known-name scan cost ~5% of the consumer thread's tracer time; matching once against a length-indexed set replaces it.

Additional Notes

  • CarrierKeys (components/context): exact names and prefixes, matched ignoring ASCII case, from a String or from raw key bytes. all() means "any key".

  • Propagator.keys() defaults to all(), so a propagator that doesn't declare its keys turns filtering off rather than losing headers. The composite propagator passes the union of its propagators' keys to a new CarrierVisitor.forEachKeyValue(carrier, keys, visitor) overload, whose default visits every key: every existing carrier behaves exactly as before. Only the Kafka 3.8 adapter overrides it, in this PR.

  • Core declares its keys:

    • each ContextInterpreter declares its own headers (Datadog, W3C, B3, Haystack, X-Ray, None);
    • ContextInterpreter adds the ones every style reads: forwarding, user agent and client IP headers (always included, since AppSec / AI Guard activation can change at run time; TODO: let them declare what they need), the configured client IP header, and the current trace config's request header tags and baggage mapping;
    • baggage, DSM and inferred-proxy declare theirs;
    • keys are cached per trace config instance and rebuilt when it changes (for instance, remote config changing header tags). When header names are logged on extraction, keys are all().
  • Per-record cost of the caching: confirming the cached union is current walks the propagators' keys() (reference compares and two trace-config lookups, a few ns), and the adapter's overload wraps the visitor in a lambda (16 B unless scalar-replaced).

  • Tests:

    • CarrierKeysTest, CompositePropagatorKeysTest (union, caching, rebuild on change, an undeclared propagator disabling filtering);
    • ExtractorKeysForkedTest: for every style, extraction over only the headers in the extractor's keys gives the same context as over all of them (trace headers, every shared header, a mapped header tag, baggage prefixes, application headers); a reflection check that every *_KEY header constant in the codecs is in the keys; keys following a trace config change. Forked because other propagation tests turn on header-name logging, a JVM-wide static, which makes every extractor keep all keys;
    • three Kafka adapter tests (skipped headers stay undecoded, all() visits everything, Base64 mode).
  • Numbers (with Read the composite propagator's carrier cache in place #13606, laptop Kafka macrobenchmark): consumer and enricher, 10% sampled upstream traces, 256m, v0.5, 3 reps, the in-review dense-store/TagLookup stack plus Read fetched Kafka headers without copying them first #13600, without and with this PR and Read the composite propagator's carrier cache in place #13606:

    workload throughput young GCs per 1M records
    consumer 270k → 269k/s (flat) 27 → 26
    enricher 170k → 176k/s (+3.6%) 55 → 54

    Tracer-attributed allocation per profiled consumer run fell from 24.2 to 22.0 GiB (−9%), mostly Read the composite propagator's carrier cache in place #13606's. This benchmark's records carry only trace headers, so it doesn't exercise the filtering itself.

  • Numbers (KafkaFetchedHeadersBenchmark, this PR against Read fetched Kafka headers without copying them first #13600, JDK 21): filtering pays off when records carry application headers, and costs a little when they carry only trace headers:

    shape appHeaders=0 (time) appHeaders=3 (time) appHeaders=3 (alloc)
    fetched +2.3% −3.7% −150 B/record
    fetchedAccessor (with Read fetched Kafka headers without copying them first #13600's accessor) +3.3% −8.3% −352 B/record
    built (producer-built headers) +1.9% −4.1%

    Times are a local single-threaded run (±1–2%) after the two fixes below; allocation is from a multi-fork A/B on a Bits worker (which also measured −5.3% time for fetchedAccessor at appHeaders=3 before the fixes). The first version cost up to +3.9% with no application headers, for two reasons:

    • contains(String) compared ignoring case only, even when the key was the declared name itself. It now checks equals() first (the usual case: header names are lower case).
    • nameAscii lower-cased each byte as it compared. It now compares the key's bytes exactly with each name's precomputed ASCII bytes, and compares ignoring case only if the key has an upper-case byte.

    What remains (~2–3%) is spread out: about 1% is confirming the cached keys are current, and the rest is the lambda wrapper and the extra call per header. It is a trade-off: records with any application headers come out ahead.

🤖 Generated with Claude Code

Propagators declare the carrier keys they read (CarrierKeys: names and
prefixes, ASCII case-insensitive). The composite propagator passes their
union to a new CarrierVisitor.forEachKeyValue overload, whose default
visits every key, so other carriers behave as before. The Kafka 3.8
adapter overrides it: a fetched header's key is matched from its bytes
once, a declared name comes back as that name's constant, a prefix match
is decoded, and anything else is skipped without decoding the key or
copying the value.

Core interpreters declare their own headers; ContextInterpreter adds the
forwarding, user agent and client IP headers, the configured client IP
header, and the trace config's header tags and baggage mapping, rebuilt
when the trace config changes. Baggage, DSM and inferred-proxy declare
theirs. Anything undeclared stays on all(), which disables filtering.

ExtractorKeysForkedTest checks, for every style, that skipping keys
outside an extractor's keys extracts the same context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added comp: core Tracer core tag: performance Performance related changes type: refactoring inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM labels Nov 21, 2026
@datadog-official

datadog-official Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 92.52%
• Overall Coverage: 59.55% (+0.25%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d78803d | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.02 s 14.02 s [-0.6%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent 13.00 s 13.08 s [-1.3%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.34 s 17.22 s [-0.1%; +1.6%] (no difference)
startup:petclinic:iast:Agent 17.11 s 17.23 s [-1.5%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 16.59 s 16.94 s [-3.2%; -1.0%] (maybe better)
startup:petclinic:sca:Agent 17.27 s 17.10 s [+0.1%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 16.27 s 16.27 s [-1.0%; +1.0%] (no difference)

Commit: d78803d7 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@pr-commenter

pr-commenter Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch dougqh/kafka-header-view dougqh/kafka-header-keys
git_commit_date 1795229242 1795289298
git_commit_sha ee355b9 d78803d
See matching parameters
Baseline Candidate
ci_job_date 1795290506 1795290506
ci_job_id 2286646605 2286646605
ci_pipeline_id 155479597 155479597
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-gnl77na3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-gnl77na3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter

pr-commenter Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch dougqh/kafka-header-view dougqh/kafka-header-keys
git_commit_date 1795229242 1795289298
git_commit_sha ee355b9 d78803d
See matching parameters
Baseline Candidate
ci_job_date 1795290555 1795290555
ci_job_id 2286646606 2286646606
ci_pipeline_id 155479597 155479597
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-5vqvp5hd 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-5vqvp5hd 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

dougqh and others added 2 commits November 21, 2026 12:24
CarrierKeys.contains(String) compared every key with regionMatches ignoring
case. Keys are usually already lower case (the tracer's own constants, or
names a carrier decoded), so String.equals, intrinsified and with an
identity check, catches them first. Recovers the per-record cost the key
filter added on Kafka records with no application headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CarrierKeys.nameAscii read every byte through ByteBuffer.get and folded
its case, against the union of all propagators' names. For a heap buffer
it now compares the backing array's bytes exactly with each name's
precomputed lower-case ASCII bytes, and only compares ignoring case if the
key has an upper-case byte at all. In KafkaFetchedHeadersBenchmark this
more than halves the matcher's cost: fetched headers with the accessor go
from -3.6% to -8.3% against #13600 with three application headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant