Skip to content

test(ffe): add scale-suite benchmark and conformance test - #2888

Draft
leoromanovsky wants to merge 1 commit into
mainfrom
leo.romanovsky/ffe-json-scale-bench
Draft

leoromanovsky wants to merge 1 commit into
mainfrom
leo.romanovsky/ffe-json-scale-bench

Conversation

@leoromanovsky

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a benchmark and a conformance test for the scale and scale-json evaluation suites of ffe-system-test-data, and repins the fixture submodule to 52205a7, the head of DataDog/ffe-system-test-data#78, which adds those suites. It changes no shipped code.

  • tests/scale_suites.rs evaluates every case in both suites and checks value, reason, and, where a case asserts them, serialId, doLog and split features.
  • benches/scale.rs (ffe-scale) is a criterion benchmark. It checks each case's value and reason, then measures each case file, and each scale-json case on its own. A counting global allocator prints bytes and allocations per evaluation on ALLOC lines.
  • At the new pin the dependent-flags suite has one more case, so its count guard moves from 15 to 16.

Motivation

A performance-sensitive customer serves large JSON flag values. The same suites now back benchmarks in each server SDK (DataDog/dd-trace-java#13485 pins the same commit), so the evaluator can be compared on equal inputs. The next PR in this stack caches the Python object that the pyo3 binding builds for a JSON variant. This PR provides the core baseline it is measured against: the core already shares each parsed JSON value, so the cost sits in the binding, not here.

flowchart LR
  F[ffe-system-test-data 52205a7<br/>scale + scale-json] --> T[tests/scale_suites.rs<br/>value, reason, metadata]
  F --> B[benches/scale.rs<br/>time + bytes/allocs per eval]
Loading

Baseline (cargo bench -p libdd-ffe-test-suite --bench ffe-scale, criterion median, 16 vCPU workspace with load average about 6):

Case Time Bytes / eval Allocs / eval
scale-json 719 nodes (scale-json-large) 120 ns 0 0
scale-json 100 KB (match) 140 ns 0 0
scale-json 1 MB (match) 143 ns 0 0
scale-json 4 MB (match) 145 ns 0 0
scale-json shapes (number array, 1 MB string, escapes, nested arrays) 108–114 ns 0 0
scale/one-of 16.7 µs 2 0.2
scale/one-of-many 57.6 µs 0 0
scale/regex-many 62.3 µs 0 0
scale/rules-sweep 16.5 µs 0 0
scale/semver 32.2 µs 8,901 368
scale/splits 6.1 µs 0 0
scale/variations 31.4 µs 0 0

The core returns a JSON value in constant time at every size because the variant is parsed once and shared. scale/semver is the one suite that allocates on every evaluation.

Additional Notes

The scale-json suite holds about 16 MB of JSON, so the conformance test takes about 10 s in a debug build. The benchmark uses short criterion windows (0.5 s warm-up, 2 s measurement) to keep the CI benchmark job reasonable.

How to test the change?

cargo nextest run -p libdd-ffe-test-suite
cargo bench -p libdd-ffe-test-suite --bench ffe-scale

🤖 Generated with Claude Code

Repin ffe-system-test-data to 52205a7, the head of
DataDog/ffe-system-test-data#78, which adds the `scale` and `scale-json`
evaluation suites. The dependent-flags suite gains one case at this pin.

`tests/scale_suites.rs` checks value, reason, serialId, doLog and
features for every case in both suites. The `ffe-scale` criterion
benchmark evaluates each case file, and each `scale-json` case on its
own, after checking its value and reason, and reports bytes and
allocations per evaluation through a counting allocator.

No shipped code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

✅ No documentation warnings found!

📦 libdd-ffe-test-suite - ✅ No warnings


Updated: 2026-11-18 00:30:19 UTC | Commit: 5e6e5e8 | missing-docs job results

@github-actions

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

⚠️ 7 issue(s) found, showing only errors (advisories, bans, sources)

📦 libdd-ffe-test-suite - 7 error(s)

Show output
info: syncing channel updates for 1.91.1-x86_64-unknown-linux-gnu
info: latest update on 2025-11-10 for version 1.91.1 (ed61e7d7e 2025-11-07)
info: downloading 6 components
warn: the missing active toolchain `1.91.1-x86_64-unknown-linux-gnu` has been auto-installed
warn: this might cause rustup commands to take longer time to finish than expected
info: you may opt out with `RUSTUP_AUTO_INSTALL=0` or `rustup set auto-install disable`
error[unmaintained]: Bincode is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:24:1
   │
24 │ bincode 1.3.3 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2025-0141
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2025-0141
   ├ Due to a doxxing and harassment incident, the bincode team has taken the decision to cease development permanently.
     
     The team considers version 1.3.3 a complete version of bincode that is not in need of any updates.
     
     ## Alternatives to consider
     
     * [wincode](https://crates.io/crates/wincode)
     * [postcard](https://crates.io/crates/postcard)
     * [bitcode](https://crates.io/crates/bitcode)
     * [rkyv](https://crates.io/crates/rkyv)
   ├ Announcement: https://git.sr.ht/~stygianentity/bincode/tree/v3.0/item/README.md
   ├ Solution: No safe upgrade is available!
   ├ bincode v1.3.3
     └── (dev) libdd-ffe v2.0.0
         └── (dev) libdd-ffe-test-suite v49.0.0

error[vulnerability]: TLS 1.3 handshake messages incorrectly accepted across encryption level boundaries
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:200:1
    │
200 │ rustls 0.23.37 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0285
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0285
    ├ Rustls accepted TLS 1.3 handshake messages sent at the wrong encryption level
      when they followed a key-changing message in the same record. For example,
      a plaintext `EncryptedExtensions` message packed into the same record as the
      `ServerHello` was accepted.
      
      RFC 8446 section 5.1 requires that handshake messages do not span key changes,
      and that implementations terminate the connection with an "unexpected_message"
      alert if they do.
      
      The handshake transcript is still authenticated, so a network-position attacker
      cannot use this to alter or complete a handshake; the practical effect is that
      a peer could send handshake messages that should be encrypted in plaintext
      without rustls rejecting the connection.
      
      This is functionally the same bug as Go's
      [GO-2026-4340](https://pkg.go.dev/vuln/GO-2026-4340) (CVE-2025-61730).
    ├ Announcement: https://github.com/rustls/rustls/pull/3265
    ├ Solution: Upgrade to >=0.23.45 (try `cargo update -p rustls`)
    ├ rustls v0.23.37
      ├── hyper-rustls v0.27.7
      │   └── libdd-networking v1.0.0
      │       ├── libdd-capabilities-impl v8.0.0
      │       │   ├── (dev) libdd-ffe v2.0.0
      │       │   │   └── (dev) libdd-ffe-test-suite v49.0.0
      │       │   └── libdd-remote-config v8.0.0
      │       │       ├── libdd-ffe v2.0.0 (*)
      │       │       └── (dev) libdd-remote-config v8.0.0 (*)
      │       ├── libdd-ffe v2.0.0 (*)
      │       └── libdd-remote-config v8.0.0 (*)
      ├── libdd-networking v1.0.0 (*)
      ├── rustls-platform-verifier v0.6.2
      │   └── libdd-networking v1.0.0 (*)
      └── tokio-rustls v0.26.0
          ├── hyper-rustls v0.27.7 (*)
          └── (dev) libdd-networking v1.0.0 (*)

error[vulnerability]: `Debug` impls for client session values expose TLS secrets
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:200:1
    │
200 │ rustls 0.23.37 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0346
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0346
    ├ The `Debug` implementations of `Tls12ClientSessionValue` and
      `Tls13ClientSessionValue` were derived, and so included the TLS 1.2 master
      secret or TLS 1.3 resumption PSK in their output.
      
      Custom `ClientSessionStore` implementations that format these values with
      `Debug` (for example, when logging) could disclose these secrets to readers of
      the logs. This conflicts with rustls's
      [key-material redaction policy](https://github.com/rustls/rustls/blob/99f2358cae2954837dbb866faf6727de75489ab9/SECURITY.md#boundary-public-api).
      
      Ordinary connection and built-in session cache `Debug` output is redacted and
      is not affected.
    ├ Announcement: https://github.com/rustls/rustls/pull/3299
    ├ Solution: Upgrade to >=0.23.46 (try `cargo update -p rustls`)
    ├ rustls v0.23.37
      ├── hyper-rustls v0.27.7
      │   └── libdd-networking v1.0.0
      │       ├── libdd-capabilities-impl v8.0.0
      │       │   ├── (dev) libdd-ffe v2.0.0
      │       │   │   └── (dev) libdd-ffe-test-suite v49.0.0
      │       │   └── libdd-remote-config v8.0.0
      │       │       ├── libdd-ffe v2.0.0 (*)
      │       │       └── (dev) libdd-remote-config v8.0.0 (*)
      │       ├── libdd-ffe v2.0.0 (*)
      │       └── libdd-remote-config v8.0.0 (*)
      ├── libdd-networking v1.0.0 (*)
      ├── rustls-platform-verifier v0.6.2
      │   └── libdd-networking v1.0.0 (*)
      └── tokio-rustls v0.26.0
          ├── hyper-rustls v0.27.7 (*)
          └── (dev) libdd-networking v1.0.0 (*)

error[vulnerability]: Brotli certificate decompression can allocate up to 1 GiB before authentication
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:200:1
    │
200 │ rustls 0.23.37 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0348
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0348
    ├ The built-in Brotli certificate decompressor (enabled by the non-default
      `brotli` crate feature) bounded the size of the decompressed output, but
      accepted a peer-selected history window. Brotli's large-window extension was
      enabled, so a five-byte header could cause an allocation of 1 GiB. A peer can
      abort a process whose allocator cannot satisfy that request.
      
      No trusted certificate is required by the attacker. Affected configurations are:
      
      - clients that enable the `brotli` feature and connect to a malicious server, and
      - servers that enable the `brotli` feature and request client certificates.
      
      Applications that do not enable the `brotli` feature are not affected.
    ├ Announcement: https://github.com/rustls/rustls/pull/3297
    ├ Solution: Upgrade to >=0.23.46 (try `cargo update -p rustls`)
    ├ rustls v0.23.37
      ├── hyper-rustls v0.27.7
      │   └── libdd-networking v1.0.0
      │       ├── libdd-capabilities-impl v8.0.0
      │       │   ├── (dev) libdd-ffe v2.0.0
      │       │   │   └── (dev) libdd-ffe-test-suite v49.0.0
      │       │   └── libdd-remote-config v8.0.0
      │       │       ├── libdd-ffe v2.0.0 (*)
      │       │       └── (dev) libdd-remote-config v8.0.0 (*)
      │       ├── libdd-ffe v2.0.0 (*)
      │       └── libdd-remote-config v8.0.0 (*)
      ├── libdd-networking v1.0.0 (*)
      ├── rustls-platform-verifier v0.6.2
      │   └── libdd-networking v1.0.0 (*)
      └── tokio-rustls v0.26.0
          ├── hyper-rustls v0.27.7 (*)
          └── (dev) libdd-networking v1.0.0 (*)

error[vulnerability]: Unauthenticated ServerHello can downgrade later connections to classical key exchange
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:200:1
    │
200 │ rustls 0.23.37 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0349
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0349
    ├ The client recorded the server-selected key exchange group for each server
      name when processing the ServerHello, before the server was authenticated.
      This record is used to predict which key share to send in future ClientHellos
      to that server name.
      
      A network attacker can answer one connection attempt with a ServerHello
      selecting X25519, and then close the connection. Later ClientHellos for that
      name carry only an X25519 key share, while `supported_groups` still lists
      X25519MLKEM768 first. If the genuine server accepts that share rather than
      sending a HelloRetryRequest for its preferred hybrid group, the handshake
      completes with classical-only key exchange and no error. This persists for as
      long as the entry stays in the client's session store.
      
      The impact is loss of post-quantum confidentiality for affected connections
      ("harvest now, decrypt later"). There is no key compromise and no
      authentication bypass.
      
      The following are not affected:
      
      - connections to servers which send a HelloRetryRequest for their preferred group,
      - clients with resumption disabled (as nothing is stored), and
      - other server names.
      
      This behaviour is discussed in
      [draft-ietf-tls-key-share-prediction](https://www.ietf.org/archive/id/draft-ietf-tls-key-share-prediction-01.html#name-security-considerations),
      so the practical impact depends largely on server policy.
    ├ Announcement: https://github.com/rustls/rustls/pull/3355
    ├ Solution: Upgrade to >=0.23.46 (try `cargo update -p rustls`)
    ├ rustls v0.23.37
      ├── hyper-rustls v0.27.7
      │   └── libdd-networking v1.0.0
      │       ├── libdd-capabilities-impl v8.0.0
      │       │   ├── (dev) libdd-ffe v2.0.0
      │       │   │   └── (dev) libdd-ffe-test-suite v49.0.0
      │       │   └── libdd-remote-config v8.0.0
      │       │       ├── libdd-ffe v2.0.0 (*)
      │       │       └── (dev) libdd-remote-config v8.0.0 (*)
      │       ├── libdd-ffe v2.0.0 (*)
      │       └── libdd-remote-config v8.0.0 (*)
      ├── libdd-networking v1.0.0 (*)
      ├── rustls-platform-verifier v0.6.2
      │   └── libdd-networking v1.0.0 (*)
      └── tokio-rustls v0.26.0
          ├── hyper-rustls v0.27.7 (*)
          └── (dev) libdd-networking v1.0.0 (*)

error[unsound]: `AsyncFd::new` and related constructors violate I/O safety
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:252:1
    │
252 │ tokio 1.49.0 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0386
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0386
    ├ `tokio::io::unix::AsyncFd` provides safe constructors (`new`, `with_interest`, `try_new`, and `try_with_interest`) that accept any `T: AsRawFd` rather than requiring proof of ownership or validity of the file descriptor. Because `RawFd` implements `AsRawFd`, safe code can construct an `AsyncFd` from an arbitrary or closed file descriptor, or close/replace the underlying file descriptor while the `AsyncFd` is still alive.
      
      This violates I/O safety in two ways:
      
      1. `AsyncFd<T>` implements `AsFd` for all `T: AsRawFd` by calling `BorrowedFd::borrow_raw(self.as_raw_fd())`, allowing safe code to obtain an arbitrary `BorrowedFd<'_>` without owning the file descriptor.
      2. Tokio's I/O driver stores a pointer to an internal allocation in the `epoll`/`kqueue` user data for registered file descriptors. If the underlying file descriptor is duplicated and then closed while an `AsyncFd` is alive, and the OS reuses that file descriptor number for a another IO resource on the same runtime, dropping the first `AsyncFd` deregisters the reused file descriptor number and frees the first while the original file description remains registered in `epoll`, leading to a use-after-free when a readiness event fires.
      
      Starting in Tokio 1.51.6, 1.53.3, and 1.54.0, `AsyncFd::new`, `AsyncFd::with_interest`, `AsyncFd::try_new`, and `AsyncFd::try_with_interest` are deprecated in favor of the `unsafe` constructors `AsyncFd::register` and `AsyncFd::register_with_interest`. Callers must ensure that the file descriptor remains valid, open, and unchanged for the lifetime of the `AsyncFd`.
    ├ Announcement: https://github.com/tokio-rs/tokio/pull/8587
    ├ Solution: Upgrade to >=1.51.6, <1.52.0 OR >=1.53.3 (try `cargo update -p tokio`)
    ├ tokio v1.49.0
      ├── h2 v0.4.19
      │   └── hyper v1.11.1
      │       ├── httpmock v0.8.0-alpha.1
      │       │   └── (dev) libdd-ffe v2.0.0
      │       │       └── (dev) libdd-ffe-test-suite v49.0.0
      │       ├── hyper-rustls v0.27.7
      │       │   └── libdd-networking v1.0.0
      │       │       ├── libdd-capabilities-impl v8.0.0
      │       │       │   ├── (dev) libdd-ffe v2.0.0 (*)
      │       │       │   └── libdd-remote-config v8.0.0
      │       │       │       ├── libdd-ffe v2.0.0 (*)
      │       │       │       └── (dev) libdd-remote-config v8.0.0 (*)
      │       │       ├── libdd-ffe v2.0.0 (*)
      │       │       └── libdd-remote-config v8.0.0 (*)
      │       ├── hyper-util v0.1.17
      │       │   ├── httpmock v0.8.0-alpha.1 (*)
      │       │   ├── hyper-rustls v0.27.7 (*)
      │       │   ├── libdd-networking v1.0.0 (*)
      │       │   └── libdd-remote-config v8.0.0 (*)
      │       ├── libdd-networking v1.0.0 (*)
      │       └── libdd-remote-config v8.0.0 (*)
      ├── httpmock v0.8.0-alpha.1 (*)
      ├── hyper v1.11.1 (*)
      ├── hyper-rustls v0.27.7 (*)
      ├── hyper-util v0.1.17 (*)
      ├── (dev) libdd-capabilities-impl v8.0.0 (*)
      ├── libdd-ffe v2.0.0 (*)
      ├── (dev) libdd-networking v1.0.0 (*)
      ├── libdd-remote-config v8.0.0 (*)
      ├── (dev) libdd-trace-protobuf v5.1.1
      │   └── libdd-remote-config v8.0.0 (*)
      ├── tokio-rustls v0.26.0
      │   ├── hyper-rustls v0.27.7 (*)
      │   └── (dev) libdd-networking v1.0.0 (*)
      ├── tokio-util v0.7.12
      │   ├── h2 v0.4.19 (*)
      │   └── libdd-remote-config v8.0.0 (*)
      └── (dev) tools v49.0.0
          └── (dev) libdd-networking v1.0.0 (*)

error[unmaintained]: chrono is unmaintained
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:38:1
   │
38 │ chrono 0.4.38 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unmaintained advisory detected
   │
   ├ ID: RUSTSEC-2026-0543
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0543
   ├ The chrono crate is unmaintained; all versions are affected.
     
     Recommended alternatives:
     
     - [jiff](https://crates.io/crates/jiff)
     - [time](https://crates.io/crates/time)
   ├ Announcement: https://github.com/chronotope/chrono/issues/1768
   ├ Solution: No safe upgrade is available!
   ├ chrono v0.4.38
     ├── libdd-ffe v2.0.0
     │   └── (dev) libdd-ffe-test-suite v49.0.0
     └── (dev) libdd-ffe-test-suite v49.0.0 (*)

advisories FAILED, bans ok, sources ok

Updated: 2026-11-18 00:31:53 UTC | Commit: 5e6e5e8 | dependency-check job results

@pr-commenter

pr-commenter Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-11-18 00:41:43

Comparing candidate commit b31a251 in PR branch leo.romanovsky/ffe-json-scale-bench with baseline commit c24d1e0 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 25 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/memoized_16_references execution_time 3.535µs 3.545µs ± 0.004µs 3.544µs ± 0.002µs 3.546µs 3.552µs 3.558µs 3.571µs 0.76% 1.963 8.788 0.12% 0.000µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/memoized_16_references execution_time [3.545µs; 3.546µs] or [-0.017%; +0.017%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/one_prerequisite execution_time 1.226µs 1.238µs ± 0.003µs 1.238µs ± 0.001µs 1.239µs 1.240µs 1.245µs 1.265µs 2.11% 4.700 50.016 0.22% 0.000µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/one_prerequisite execution_time [1.238µs; 1.239µs] or [-0.030%; +0.030%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
scale-json/json-shapes execution_time 136.463ns 137.307ns ± 0.339ns 137.277ns ± 0.112ns 137.391ns 137.660ns 138.481ns 140.467ns 2.32% 4.479 38.232 0.25% 0.024ns 1 200
scale-json/json-shapes/0-scale-json-number-array execution_time 135.301ns 136.168ns ± 0.238ns 136.149ns ± 0.108ns 136.267ns 136.528ns 136.839ns 137.256ns 0.81% 0.296 5.754 0.17% 0.017ns 1 200
scale-json/json-shapes/1-scale-json-long-string execution_time 136.627ns 138.448ns ± 0.509ns 138.492ns ± 0.264ns 138.737ns 139.169ns 139.428ns 140.398ns 1.38% -0.576 2.693 0.37% 0.036ns 1 200
scale-json/json-shapes/2-scale-json-escapes execution_time 132.072ns 133.200ns ± 0.289ns 133.184ns ± 0.097ns 133.272ns 133.472ns 134.310ns 135.655ns 1.86% 3.267 28.467 0.22% 0.020ns 1 200
scale-json/json-shapes/3-scale-json-nested-arrays execution_time 133.682ns 134.635ns ± 0.266ns 134.650ns ± 0.129ns 134.756ns 134.988ns 135.407ns 136.103ns 1.08% 0.869 7.518 0.20% 0.019ns 1 200
scale-json/json-sizes execution_time 177.197ns 178.506ns ± 0.371ns 178.474ns ± 0.213ns 178.750ns 179.083ns 179.513ns 180.318ns 1.03% 0.641 3.141 0.21% 0.026ns 1 200
scale-json/json-sizes/0-scale-json-100kb execution_time 170.124ns 171.156ns ± 0.287ns 171.129ns ± 0.133ns 171.264ns 171.596ns 172.237ns 172.340ns 0.71% 0.496 4.811 0.17% 0.020ns 1 200
scale-json/json-sizes/1-scale-json-100kb execution_time 169.842ns 170.892ns ± 0.368ns 170.875ns ± 0.144ns 171.004ns 171.249ns 172.074ns 174.277ns 1.99% 4.190 36.149 0.21% 0.026ns 1 200
scale-json/json-sizes/2-scale-json-1mb execution_time 172.016ns 173.186ns ± 0.466ns 173.132ns ± 0.135ns 173.270ns 173.620ns 175.095ns 176.883ns 2.17% 4.188 29.237 0.27% 0.033ns 1 200
scale-json/json-sizes/3-scale-json-1mb execution_time 171.627ns 172.708ns ± 0.378ns 172.650ns ± 0.133ns 172.796ns 173.226ns 173.922ns 175.322ns 1.55% 2.877 16.156 0.22% 0.027ns 1 200
scale-json/json-sizes/4-scale-json-4mb execution_time 171.416ns 172.528ns ± 0.689ns 172.457ns ± 0.135ns 172.592ns 172.990ns 173.460ns 181.324ns 5.14% 10.497 131.499 0.40% 0.049ns 1 200
scale-json/json-sizes/5-scale-json-4mb execution_time 169.902ns 170.992ns ± 0.366ns 170.952ns ± 0.125ns 171.087ns 171.454ns 172.562ns 173.818ns 1.68% 2.908 20.894 0.21% 0.026ns 1 200
scale-json/json-values execution_time 143.792ns 144.624ns ± 0.256ns 144.586ns ± 0.092ns 144.698ns 144.979ns 145.699ns 146.562ns 1.37% 2.828 18.818 0.18% 0.018ns 1 200
scale-json/json-values/0-scale-json-large execution_time 128.709ns 129.533ns ± 0.365ns 129.506ns ± 0.096ns 129.602ns 129.782ns 130.243ns 133.658ns 3.21% 7.377 81.037 0.28% 0.026ns 1 200
scale-json/json-values/1-scale-json-wide execution_time 132.032ns 132.819ns ± 0.243ns 132.781ns ± 0.094ns 132.896ns 133.197ns 133.689ns 134.253ns 1.11% 1.555 9.067 0.18% 0.017ns 1 200
scale-json/json-values/2-scale-json-deep execution_time 133.697ns 134.534ns ± 0.357ns 134.495ns ± 0.091ns 134.585ns 134.883ns 135.509ns 138.204ns 2.76% 6.024 57.074 0.26% 0.025ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
scale-json/json-shapes execution_time [137.260ns; 137.354ns] or [-0.034%; +0.034%] None None None
scale-json/json-shapes/0-scale-json-number-array execution_time [136.135ns; 136.201ns] or [-0.024%; +0.024%] None None None
scale-json/json-shapes/1-scale-json-long-string execution_time [138.377ns; 138.519ns] or [-0.051%; +0.051%] None None None
scale-json/json-shapes/2-scale-json-escapes execution_time [133.160ns; 133.240ns] or [-0.030%; +0.030%] None None None
scale-json/json-shapes/3-scale-json-nested-arrays execution_time [134.598ns; 134.672ns] or [-0.027%; +0.027%] None None None
scale-json/json-sizes execution_time [178.455ns; 178.558ns] or [-0.029%; +0.029%] None None None
scale-json/json-sizes/0-scale-json-100kb execution_time [171.116ns; 171.196ns] or [-0.023%; +0.023%] None None None
scale-json/json-sizes/1-scale-json-100kb execution_time [170.841ns; 170.943ns] or [-0.030%; +0.030%] None None None
scale-json/json-sizes/2-scale-json-1mb execution_time [173.121ns; 173.251ns] or [-0.037%; +0.037%] None None None
scale-json/json-sizes/3-scale-json-1mb execution_time [172.655ns; 172.760ns] or [-0.030%; +0.030%] None None None
scale-json/json-sizes/4-scale-json-4mb execution_time [172.432ns; 172.623ns] or [-0.055%; +0.055%] None None None
scale-json/json-sizes/5-scale-json-4mb execution_time [170.941ns; 171.043ns] or [-0.030%; +0.030%] None None None
scale-json/json-values execution_time [144.589ns; 144.660ns] or [-0.025%; +0.025%] None None None
scale-json/json-values/0-scale-json-large execution_time [129.482ns; 129.584ns] or [-0.039%; +0.039%] None None None
scale-json/json-values/1-scale-json-wide execution_time [132.785ns; 132.852ns] or [-0.025%; +0.025%] None None None
scale-json/json-values/2-scale-json-deep execution_time [134.484ns; 134.583ns] or [-0.037%; +0.037%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
single_flag_killswitch/rules-based execution_time 195.728ns 198.177ns ± 1.782ns 197.891ns ± 1.230ns 199.131ns 201.060ns 203.782ns 204.464ns 3.32% 0.893 0.888 0.90% 0.126ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
single_flag_killswitch/rules-based execution_time [197.930ns; 198.424ns] or [-0.125%; +0.125%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
scale/flags execution_time 141.072ns 142.131ns ± 0.880ns 142.029ns ± 0.119ns 142.149ns 142.503ns 144.088ns 153.068ns 7.77% 10.201 119.901 0.62% 0.062ns 1 200
scale/numeric execution_time 2.624µs 2.674µs ± 0.037µs 2.671µs ± 0.002µs 2.674µs 2.678µs 2.689µs 3.180µs 19.02% 13.243 180.301 1.37% 0.003µs 1 200
scale/one-of execution_time 17.523µs 18.690µs ± 0.275µs 18.674µs ± 0.021µs 18.698µs 18.772µs 19.060µs 22.209µs 18.93% 10.178 134.357 1.47% 0.019µs 1 200
scale/one-of-many execution_time 57.982µs 69.131µs ± 6.435µs 67.981µs ± 0.546µs 68.619µs 73.837µs 89.691µs 146.163µs 115.00% 9.333 103.569 9.28% 0.455µs 1 200
scale/one-of-small-numbers execution_time 1.012µs 1.019µs ± 0.003µs 1.019µs ± 0.001µs 1.020µs 1.022µs 1.034µs 1.054µs 3.49% 6.286 53.538 0.34% 0.000µs 1 200
scale/regex execution_time 4.264µs 4.462µs ± 0.027µs 4.467µs ± 0.004µs 4.471µs 4.477µs 4.483µs 4.486µs 0.42% -5.284 30.897 0.61% 0.002µs 1 200
scale/regex-many execution_time 58.724µs 62.137µs ± 9.628µs 60.112µs ± 0.615µs 60.995µs 68.263µs 98.006µs 147.951µs 146.12% 7.396 60.074 15.46% 0.681µs 1 200
scale/rules execution_time 3.163µs 3.199µs ± 0.007µs 3.200µs ± 0.003µs 3.203µs 3.208µs 3.217µs 3.224µs 0.75% -0.869 6.246 0.20% 0.000µs 1 200
scale/rules-sweep execution_time 1.843µs 19.641µs ± 1.780µs 20.018µs ± 0.117µs 20.108µs 20.198µs 20.226µs 20.265µs 1.23% -7.883 68.836 9.04% 0.126µs 1 200
scale/semver execution_time 36.944µs 37.292µs ± 0.462µs 37.221µs ± 0.036µs 37.268µs 37.560µs 38.534µs 43.181µs 16.01% 10.788 131.141 1.24% 0.033µs 1 200
scale/splits execution_time 7.004µs 7.526µs ± 0.081µs 7.521µs ± 0.010µs 7.532µs 7.569µs 7.767µs 8.204µs 9.08% 2.450 38.788 1.08% 0.006µs 1 200
scale/time-windows execution_time 560.572ns 564.092ns ± 2.502ns 562.904ns ± 1.403ns 566.495ns 568.084ns 569.387ns 575.411ns 2.22% 0.888 0.624 0.44% 0.177ns 1 200
scale/variations execution_time 24.043µs 32.031µs ± 0.625µs 32.129µs ± 0.035µs 32.157µs 32.198µs 32.331µs 32.529µs 1.25% -10.925 133.072 1.95% 0.044µs 1 200
scale/wide-context execution_time 452.126ns 456.127ns ± 0.850ns 456.068ns ± 0.361ns 456.462ns 457.454ns 457.898ns 461.674ns 1.23% 0.509 11.510 0.19% 0.060ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
scale/flags execution_time [142.009ns; 142.253ns] or [-0.086%; +0.086%] None None None
scale/numeric execution_time [2.669µs; 2.679µs] or [-0.190%; +0.190%] None None None
scale/one-of execution_time [18.652µs; 18.728µs] or [-0.204%; +0.204%] None None None
scale/one-of-many execution_time [68.239µs; 70.023µs] or [-1.290%; +1.290%] None None None
scale/one-of-small-numbers execution_time [1.019µs; 1.020µs] or [-0.047%; +0.047%] None None None
scale/regex execution_time [4.458µs; 4.466µs] or [-0.084%; +0.084%] None None None
scale/regex-many execution_time [60.803µs; 63.471µs] or [-2.147%; +2.147%] None None None
scale/rules execution_time [3.199µs; 3.200µs] or [-0.028%; +0.028%] None None None
scale/rules-sweep execution_time [19.395µs; 19.888µs] or [-1.256%; +1.256%] None None None
scale/semver execution_time [37.228µs; 37.356µs] or [-0.172%; +0.172%] None None None
scale/splits execution_time [7.515µs; 7.538µs] or [-0.150%; +0.150%] None None None
scale/time-windows execution_time [563.745ns; 564.439ns] or [-0.061%; +0.061%] None None None
scale/variations execution_time [31.944µs; 32.118µs] or [-0.270%; +0.270%] None None None
scale/wide-context execution_time [456.009ns; 456.245ns] or [-0.026%; +0.026%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_255_failure execution_time 139.266µs 139.843µs ± 0.195µs 139.814µs ± 0.057µs 139.891µs 140.097µs 140.732µs 141.127µs 0.94% 2.777 14.734 0.14% 0.014µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_255_failure execution_time [139.816µs; 139.870µs] or [-0.019%; +0.019%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_127_success execution_time 122.426µs 122.882µs ± 0.305µs 122.844µs ± 0.059µs 122.908µs 123.036µs 123.726µs 126.666µs 3.11% 9.944 118.048 0.25% 0.022µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_127_success execution_time [122.839µs; 122.924µs] or [-0.034%; +0.034%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_255_success execution_time 242.877µs 243.556µs ± 0.398µs 243.491µs ± 0.091µs 243.593µs 243.920µs 245.039µs 248.015µs 1.86% 7.697 78.812 0.16% 0.028µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_255_success execution_time [243.501µs; 243.611µs] or [-0.023%; +0.023%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/payloads/features/10flags_1000users_max_features execution_time 66.670ms 66.941ms ± 0.638ms 66.826ms ± 0.064ms 66.863ms 66.956ms 69.258ms 69.982ms 4.72% 4.593 19.433 0.93% 0.128ms 1 25
flagevaluation_evp/payloads/features/10flags_1000users_max_features throughput 14289.363op/s 14939.825op/s ± 136.517op/s 14964.318op/s ± 14.216op/s 14980.679op/s 14990.722op/s 14997.462op/s 14999.362op/s 0.23% -4.583 19.375 0.90% 27.303op/s 1 25
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time 35.866ms 36.193ms ± 0.109ms 36.195ms ± 0.054ms 36.258ms 36.345ms 36.479ms 36.501ms 0.85% -0.123 1.942 0.30% 0.015ms 1 50
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput 68490.639op/s 69074.285op/s ± 207.715op/s 69070.892op/s ± 103.225op/s 69150.544op/s 69371.897op/s 69640.429op/s 69704.570op/s 0.92% 0.158 1.952 0.30% 29.375op/s 1 50
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time 185.969ms 186.316ms ± 0.246ms 186.300ms ± 0.200ms 186.514ms 186.673ms 186.780ms 186.810ms 0.27% 0.222 -1.083 0.13% 0.049ms 1 25
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput 5353.032op/s 5367.246op/s ± 7.081op/s 5367.679op/s ± 5.743op/s 5372.076op/s 5377.079op/s 5377.223op/s 5377.248op/s 0.18% -0.219 -1.086 0.13% 1.416op/s 1 25
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time 882.383µs 884.747µs ± 1.111µs 884.597µs ± 0.601µs 885.247µs 886.579µs 887.669µs 888.285µs 0.42% 0.605 0.539 0.12% 0.111µs 1 100
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput 112576.509op/s 113026.864op/s ± 141.836op/s 113045.822op/s ± 76.732op/s 113115.792op/s 113219.826op/s 113320.395op/s 113329.422op/s 0.25% -0.596 0.526 0.12% 14.184op/s 1 100
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/payloads/features/10flags_1000users_max_features execution_time [66.691ms; 67.191ms] or [-0.374%; +0.374%] None None None
flagevaluation_evp/payloads/features/10flags_1000users_max_features throughput [14886.311op/s; 14993.339op/s] or [-0.358%; +0.358%] None None None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time [36.163ms; 36.223ms] or [-0.083%; +0.083%] None None None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput [69016.710op/s; 69131.859op/s] or [-0.083%; +0.083%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time [186.219ms; 186.412ms] or [-0.052%; +0.052%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput [5364.470op/s; 5370.022op/s] or [-0.052%; +0.052%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time [884.529µs; 884.965µs] or [-0.025%; +0.025%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput [112999.065op/s; 113054.664op/s] or [-0.025%; +0.025%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/coalescer/features/10flags_1000users_max_features execution_time 61.333ms 61.638ms ± 0.149ms 61.635ms ± 0.102ms 61.723ms 61.858ms 61.991ms 62.031ms 0.64% 0.497 0.546 0.24% 0.030ms 1 25
flagevaluation_evp/coalescer/features/10flags_1000users_max_features throughput 16121.098op/s 16223.739op/s ± 39.057op/s 16224.644op/s ± 26.811op/s 16254.629op/s 16268.612op/s 16295.952op/s 16304.511op/s 0.49% -0.481 0.526 0.24% 7.811op/s 1 25
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time 31.850ms 32.019ms ± 0.072ms 32.016ms ± 0.040ms 32.055ms 32.129ms 32.200ms 32.216ms 0.62% 0.252 0.458 0.22% 0.010ms 1 50
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput 77601.630op/s 78079.800op/s ± 176.309op/s 78086.335op/s ± 96.873op/s 78181.392op/s 78373.620op/s 78444.135op/s 78492.695op/s 0.52% -0.236 0.446 0.22% 24.934op/s 1 50
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time 192.360ms 193.374ms ± 1.386ms 193.211ms ± 0.205ms 193.343ms 193.615ms 198.361ms 199.852ms 3.44% 4.296 17.727 0.70% 0.277ms 1 25
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput 5003.702op/s 5171.581op/s ± 35.991op/s 5175.682op/s ± 5.497op/s 5184.299op/s 5190.224op/s 5196.602op/s 5198.593op/s 0.44% -4.269 17.574 0.68% 7.198op/s 1 25
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time 696.222µs 706.382µs ± 4.041µs 706.147µs ± 1.588µs 708.054µs 713.177µs 714.588µs 714.742µs 1.22% -0.140 0.165 0.57% 0.404µs 1 100
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput 139910.614op/s 141571.008op/s ± 810.616op/s 141613.485op/s ± 318.531op/s 141879.511op/s 143107.378op/s 143518.298op/s 143632.347op/s 1.43% 0.176 0.194 0.57% 81.062op/s 1 100
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/coalescer/features/10flags_1000users_max_features execution_time [61.580ms; 61.697ms] or [-0.094%; +0.094%] None None None
flagevaluation_evp/coalescer/features/10flags_1000users_max_features throughput [16208.429op/s; 16239.049op/s] or [-0.094%; +0.094%] None None None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time [31.999ms; 32.039ms] or [-0.063%; +0.063%] None None None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput [78030.930op/s; 78128.669op/s] or [-0.063%; +0.063%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time [192.830ms; 193.917ms] or [-0.281%; +0.281%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput [5157.472op/s; 5185.689op/s] or [-0.273%; +0.273%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time [705.590µs; 707.174µs] or [-0.112%; +0.112%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput [141412.130op/s; 141729.886op/s] or [-0.112%; +0.112%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/legacy execution_time 125.849ns 126.683ns ± 0.238ns 126.666ns ± 0.050ns 126.719ns 126.840ns 127.234ns 129.232ns 2.03% 6.552 68.002 0.19% 0.017ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/legacy execution_time [126.650ns; 126.716ns] or [-0.026%; +0.026%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_127_failure execution_time 69.377µs 69.682µs ± 0.083µs 69.673µs ± 0.038µs 69.715µs 69.792µs 69.984µs 70.198µs 0.75% 2.135 11.805 0.12% 0.006µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_127_failure execution_time [69.670µs; 69.693µs] or [-0.016%; +0.016%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz b31a251 1794961330 leo.romanovsky/ffe-json-scale-bench
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sdk_test_data/rules-based execution_time 307.823µs 315.316µs ± 15.846µs 311.299µs ± 1.500µs 313.310µs 335.704µs 390.182µs 440.488µs 41.50% 5.575 35.362 5.01% 1.120µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sdk_test_data/rules-based execution_time [313.120µs; 317.512µs] or [-0.696%; +0.696%] None None None

Baseline

Baseline benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/memoized_16_references execution_time 3.486µs 3.503µs ± 0.004µs 3.503µs ± 0.002µs 3.505µs 3.510µs 3.514µs 3.517µs 0.39% -0.125 2.054 0.12% 0.000µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/memoized_16_references execution_time [3.503µs; 3.504µs] or [-0.016%; +0.016%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/one_prerequisite execution_time 1.240µs 1.249µs ± 0.002µs 1.249µs ± 0.001µs 1.250µs 1.251µs 1.255µs 1.267µs 1.42% 3.837 39.172 0.15% 0.000µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/one_prerequisite execution_time [1.249µs; 1.250µs] or [-0.021%; +0.021%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
single_flag_killswitch/rules-based execution_time 195.386ns 198.153ns ± 1.863ns 197.994ns ± 1.243ns 199.207ns 201.513ns 203.539ns 203.920ns 2.99% 0.728 0.226 0.94% 0.132ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
single_flag_killswitch/rules-based execution_time [197.895ns; 198.411ns] or [-0.130%; +0.130%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_255_failure execution_time 139.154µs 139.590µs ± 0.139µs 139.580µs ± 0.069µs 139.649µs 139.781µs 140.178µs 140.233µs 0.47% 1.587 6.422 0.10% 0.010µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_255_failure execution_time [139.571µs; 139.609µs] or [-0.014%; +0.014%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_127_success execution_time 122.398µs 122.822µs ± 0.226µs 122.791µs ± 0.061µs 122.852µs 123.008µs 123.761µs 124.687µs 1.54% 5.391 37.505 0.18% 0.016µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_127_success execution_time [122.791µs; 122.853µs] or [-0.025%; +0.025%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_255_success execution_time 242.692µs 243.416µs ± 0.213µs 243.417µs ± 0.088µs 243.501µs 243.711µs 244.094µs 244.526µs 0.46% 0.809 5.621 0.09% 0.015µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_255_success execution_time [243.386µs; 243.445µs] or [-0.012%; +0.012%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/payloads/features/10flags_1000users_max_features execution_time 66.195ms 66.470ms ± 0.376ms 66.385ms ± 0.073ms 66.458ms 66.767ms 67.822ms 68.151ms 2.66% 3.760 14.350 0.55% 0.075ms 1 25
flagevaluation_evp/payloads/features/10flags_1000users_max_features throughput 14673.353op/s 15044.879op/s ± 83.443op/s 15063.549op/s ± 16.460op/s 15078.025op/s 15103.923op/s 15106.322op/s 15106.866op/s 0.29% -3.723 14.133 0.54% 16.689op/s 1 25
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time 35.314ms 35.397ms ± 0.048ms 35.390ms ± 0.029ms 35.434ms 35.483ms 35.492ms 35.499ms 0.31% 0.249 -0.612 0.14% 0.007ms 1 50
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput 70424.635op/s 70627.167op/s ± 96.584op/s 70641.899op/s ± 57.487op/s 70682.753op/s 70786.721op/s 70791.110op/s 70793.807op/s 0.22% -0.244 -0.613 0.14% 13.659op/s 1 50
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time 186.060ms 186.864ms ± 0.374ms 186.790ms ± 0.280ms 187.194ms 187.384ms 187.511ms 187.548ms 0.41% 0.039 -0.818 0.20% 0.075ms 1 25
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput 5331.964op/s 5351.496op/s ± 10.720op/s 5353.594op/s ± 7.999op/s 5359.979op/s 5363.851op/s 5372.131op/s 5374.608op/s 0.39% -0.032 -0.812 0.20% 2.144op/s 1 25
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time 873.546µs 875.605µs ± 0.812µs 875.607µs ± 0.494µs 876.049µs 876.968µs 877.501µs 878.389µs 0.32% 0.386 0.747 0.09% 0.081µs 1 100
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput 113844.772op/s 114206.830op/s ± 105.895op/s 114206.425op/s ± 64.407op/s 114271.902op/s 114370.527op/s 114418.806op/s 114475.941op/s 0.24% -0.378 0.736 0.09% 10.590op/s 1 100
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/payloads/features/10flags_1000users_max_features execution_time [66.322ms; 66.617ms] or [-0.222%; +0.222%] None None None
flagevaluation_evp/payloads/features/10flags_1000users_max_features throughput [15012.170op/s; 15077.588op/s] or [-0.217%; +0.217%] None None None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields execution_time [35.384ms; 35.411ms] or [-0.038%; +0.038%] None None None
flagevaluation_evp/payloads/scale/2500flags_500users_20fields throughput [70600.395op/s; 70653.938op/s] or [-0.038%; +0.038%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields execution_time [186.718ms; 187.011ms] or [-0.079%; +0.079%] None None None
flagevaluation_evp/payloads/stress/10flags_1000users_250fields throughput [5347.294op/s; 5355.698op/s] or [-0.079%; +0.079%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields execution_time [875.446µs; 875.764µs] or [-0.018%; +0.018%] None None None
flagevaluation_evp/payloads/typical/100flags_50users_10fields throughput [114186.075op/s; 114227.585op/s] or [-0.018%; +0.018%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
flagevaluation_evp/coalescer/features/10flags_1000users_max_features execution_time 61.164ms 61.475ms ± 0.113ms 61.500ms ± 0.052ms 61.520ms 61.612ms 61.686ms 61.710ms 0.34% -0.655 1.204 0.18% 0.023ms 1 25
flagevaluation_evp/coalescer/features/10flags_1000users_max_features throughput 16204.880op/s 16266.829op/s ± 29.858op/s 16260.253op/s ± 13.853op/s 16278.371op/s 16317.618op/s 16342.746op/s 16349.489op/s 0.55% 0.670 1.224 0.18% 5.972op/s 1 25
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time 31.637ms 31.794ms ± 0.069ms 31.785ms ± 0.045ms 31.844ms 31.920ms 31.950ms 31.957ms 0.54% 0.351 -0.067 0.21% 0.010ms 1 50
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput 78230.850op/s 78631.455op/s ± 169.560op/s 78653.385op/s ± 111.745op/s 78739.857op/s 78858.871op/s 78979.498op/s 79021.172op/s 0.47% -0.339 -0.070 0.21% 23.979op/s 1 50
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time 192.411ms 192.879ms ± 0.287ms 192.816ms ± 0.218ms 193.103ms 193.246ms 193.489ms 193.565ms 0.39% 0.296 -0.335 0.15% 0.057ms 1 25
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput 5166.220op/s 5184.612op/s ± 7.713op/s 5186.304op/s ± 5.876op/s 5188.222op/s 5196.032op/s 5196.936op/s 5197.197op/s 0.21% -0.289 -0.344 0.15% 1.543op/s 1 25
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time 692.581µs 701.412µs ± 3.003µs 701.280µs ± 0.766µs 702.223µs 706.685µs 707.469µs 707.715µs 0.92% -0.588 1.686 0.43% 0.300µs 1 100
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput 141299.871op/s 142572.143op/s ± 612.077op/s 142596.310op/s ± 155.664op/s 142719.003op/s 144018.019op/s 144277.432op/s 144387.391op/s 1.26% 0.631 1.737 0.43% 61.208op/s 1 100
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
flagevaluation_evp/coalescer/features/10flags_1000users_max_features execution_time [61.431ms; 61.519ms] or [-0.072%; +0.072%] None None None
flagevaluation_evp/coalescer/features/10flags_1000users_max_features throughput [16255.125op/s; 16278.533op/s] or [-0.072%; +0.072%] None None None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields execution_time [31.775ms; 31.813ms] or [-0.060%; +0.060%] None None None
flagevaluation_evp/coalescer/scale/2500flags_500users_20fields throughput [78584.456op/s; 78678.454op/s] or [-0.060%; +0.060%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields execution_time [192.766ms; 192.991ms] or [-0.058%; +0.058%] None None None
flagevaluation_evp/coalescer/stress/10flags_1000users_250fields throughput [5181.588op/s; 5187.635op/s] or [-0.058%; +0.058%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields execution_time [700.823µs; 702.001µs] or [-0.084%; +0.084%] None None None
flagevaluation_evp/coalescer/typical/100flags_50users_10fields throughput [142452.179op/s; 142692.108op/s] or [-0.084%; +0.084%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/legacy execution_time 128.174ns 128.991ns ± 0.314ns 128.973ns ± 0.062ns 129.035ns 129.153ns 129.396ns 132.930ns 3.07% 9.837 122.506 0.24% 0.022ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/legacy execution_time [128.948ns; 129.035ns] or [-0.034%; +0.034%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
dependent_flags/chain_127_failure execution_time 69.092µs 69.476µs ± 0.111µs 69.464µs ± 0.034µs 69.503µs 69.571µs 69.742µs 70.594µs 1.63% 5.817 55.521 0.16% 0.008µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
dependent_flags/chain_127_failure execution_time [69.461µs; 69.492µs] or [-0.022%; +0.022%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz c24d1e0 1794957957 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sdk_test_data/rules-based execution_time 301.766µs 308.142µs ± 11.979µs 304.886µs ± 1.504µs 307.746µs 320.464µs 355.123µs 427.270µs 40.14% 6.538 53.956 3.88% 0.847µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sdk_test_data/rules-based execution_time [306.482µs; 309.802µs] or [-0.539%; +0.539%] None None None

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant