Repository navigation
test(ffe): add scale-suite benchmark and conformance test - #2888
leoromanovsky wants to merge 1 commit into
Conversation
Repin ffe-system-test-data to 52205a7, the head of DataDog/ffe-system-test-data#78, which adds the `scale` and `scale-json` evaluation suites. The dependent-flags suite gains one case at this pin. `tests/scale_suites.rs` checks value, reason, serialId, doLog and features for every case in both suites. The `ffe-scale` criterion benchmark evaluates each case file, and each `scale-json` case on its own, after checking its value and reason, and reports bytes and allocations per evaluation through a counting allocator. No shipped code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
📚 Documentation Check Results✅ No documentation warnings found! 📦
|
🔒 Cargo Deny Results📦
|
BenchmarksComparisonBenchmark execution time: 2026-11-18 00:41:43 Comparing candidate commit b31a251 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 25 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/memoized_16_references | execution_time | 3.535µs | 3.545µs ± 0.004µs | 3.544µs ± 0.002µs | 3.546µs | 3.552µs | 3.558µs | 3.571µs | 0.76% | 1.963 | 8.788 | 0.12% | 0.000µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/memoized_16_references | execution_time | [3.545µs; 3.546µs] or [-0.017%; +0.017%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/one_prerequisite | execution_time | 1.226µs | 1.238µs ± 0.003µs | 1.238µs ± 0.001µs | 1.239µs | 1.240µs | 1.245µs | 1.265µs | 2.11% | 4.700 | 50.016 | 0.22% | 0.000µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/one_prerequisite | execution_time | [1.238µs; 1.239µs] or [-0.030%; +0.030%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| scale-json/json-shapes | execution_time | 136.463ns | 137.307ns ± 0.339ns | 137.277ns ± 0.112ns | 137.391ns | 137.660ns | 138.481ns | 140.467ns | 2.32% | 4.479 | 38.232 | 0.25% | 0.024ns | 1 | 200 |
| scale-json/json-shapes/0-scale-json-number-array | execution_time | 135.301ns | 136.168ns ± 0.238ns | 136.149ns ± 0.108ns | 136.267ns | 136.528ns | 136.839ns | 137.256ns | 0.81% | 0.296 | 5.754 | 0.17% | 0.017ns | 1 | 200 |
| scale-json/json-shapes/1-scale-json-long-string | execution_time | 136.627ns | 138.448ns ± 0.509ns | 138.492ns ± 0.264ns | 138.737ns | 139.169ns | 139.428ns | 140.398ns | 1.38% | -0.576 | 2.693 | 0.37% | 0.036ns | 1 | 200 |
| scale-json/json-shapes/2-scale-json-escapes | execution_time | 132.072ns | 133.200ns ± 0.289ns | 133.184ns ± 0.097ns | 133.272ns | 133.472ns | 134.310ns | 135.655ns | 1.86% | 3.267 | 28.467 | 0.22% | 0.020ns | 1 | 200 |
| scale-json/json-shapes/3-scale-json-nested-arrays | execution_time | 133.682ns | 134.635ns ± 0.266ns | 134.650ns ± 0.129ns | 134.756ns | 134.988ns | 135.407ns | 136.103ns | 1.08% | 0.869 | 7.518 | 0.20% | 0.019ns | 1 | 200 |
| scale-json/json-sizes | execution_time | 177.197ns | 178.506ns ± 0.371ns | 178.474ns ± 0.213ns | 178.750ns | 179.083ns | 179.513ns | 180.318ns | 1.03% | 0.641 | 3.141 | 0.21% | 0.026ns | 1 | 200 |
| scale-json/json-sizes/0-scale-json-100kb | execution_time | 170.124ns | 171.156ns ± 0.287ns | 171.129ns ± 0.133ns | 171.264ns | 171.596ns | 172.237ns | 172.340ns | 0.71% | 0.496 | 4.811 | 0.17% | 0.020ns | 1 | 200 |
| scale-json/json-sizes/1-scale-json-100kb | execution_time | 169.842ns | 170.892ns ± 0.368ns | 170.875ns ± 0.144ns | 171.004ns | 171.249ns | 172.074ns | 174.277ns | 1.99% | 4.190 | 36.149 | 0.21% | 0.026ns | 1 | 200 |
| scale-json/json-sizes/2-scale-json-1mb | execution_time | 172.016ns | 173.186ns ± 0.466ns | 173.132ns ± 0.135ns | 173.270ns | 173.620ns | 175.095ns | 176.883ns | 2.17% | 4.188 | 29.237 | 0.27% | 0.033ns | 1 | 200 |
| scale-json/json-sizes/3-scale-json-1mb | execution_time | 171.627ns | 172.708ns ± 0.378ns | 172.650ns ± 0.133ns | 172.796ns | 173.226ns | 173.922ns | 175.322ns | 1.55% | 2.877 | 16.156 | 0.22% | 0.027ns | 1 | 200 |
| scale-json/json-sizes/4-scale-json-4mb | execution_time | 171.416ns | 172.528ns ± 0.689ns | 172.457ns ± 0.135ns | 172.592ns | 172.990ns | 173.460ns | 181.324ns | 5.14% | 10.497 | 131.499 | 0.40% | 0.049ns | 1 | 200 |
| scale-json/json-sizes/5-scale-json-4mb | execution_time | 169.902ns | 170.992ns ± 0.366ns | 170.952ns ± 0.125ns | 171.087ns | 171.454ns | 172.562ns | 173.818ns | 1.68% | 2.908 | 20.894 | 0.21% | 0.026ns | 1 | 200 |
| scale-json/json-values | execution_time | 143.792ns | 144.624ns ± 0.256ns | 144.586ns ± 0.092ns | 144.698ns | 144.979ns | 145.699ns | 146.562ns | 1.37% | 2.828 | 18.818 | 0.18% | 0.018ns | 1 | 200 |
| scale-json/json-values/0-scale-json-large | execution_time | 128.709ns | 129.533ns ± 0.365ns | 129.506ns ± 0.096ns | 129.602ns | 129.782ns | 130.243ns | 133.658ns | 3.21% | 7.377 | 81.037 | 0.28% | 0.026ns | 1 | 200 |
| scale-json/json-values/1-scale-json-wide | execution_time | 132.032ns | 132.819ns ± 0.243ns | 132.781ns ± 0.094ns | 132.896ns | 133.197ns | 133.689ns | 134.253ns | 1.11% | 1.555 | 9.067 | 0.18% | 0.017ns | 1 | 200 |
| scale-json/json-values/2-scale-json-deep | execution_time | 133.697ns | 134.534ns ± 0.357ns | 134.495ns ± 0.091ns | 134.585ns | 134.883ns | 135.509ns | 138.204ns | 2.76% | 6.024 | 57.074 | 0.26% | 0.025ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| scale-json/json-shapes | execution_time | [137.260ns; 137.354ns] or [-0.034%; +0.034%] | None | None | None |
| scale-json/json-shapes/0-scale-json-number-array | execution_time | [136.135ns; 136.201ns] or [-0.024%; +0.024%] | None | None | None |
| scale-json/json-shapes/1-scale-json-long-string | execution_time | [138.377ns; 138.519ns] or [-0.051%; +0.051%] | None | None | None |
| scale-json/json-shapes/2-scale-json-escapes | execution_time | [133.160ns; 133.240ns] or [-0.030%; +0.030%] | None | None | None |
| scale-json/json-shapes/3-scale-json-nested-arrays | execution_time | [134.598ns; 134.672ns] or [-0.027%; +0.027%] | None | None | None |
| scale-json/json-sizes | execution_time | [178.455ns; 178.558ns] or [-0.029%; +0.029%] | None | None | None |
| scale-json/json-sizes/0-scale-json-100kb | execution_time | [171.116ns; 171.196ns] or [-0.023%; +0.023%] | None | None | None |
| scale-json/json-sizes/1-scale-json-100kb | execution_time | [170.841ns; 170.943ns] or [-0.030%; +0.030%] | None | None | None |
| scale-json/json-sizes/2-scale-json-1mb | execution_time | [173.121ns; 173.251ns] or [-0.037%; +0.037%] | None | None | None |
| scale-json/json-sizes/3-scale-json-1mb | execution_time | [172.655ns; 172.760ns] or [-0.030%; +0.030%] | None | None | None |
| scale-json/json-sizes/4-scale-json-4mb | execution_time | [172.432ns; 172.623ns] or [-0.055%; +0.055%] | None | None | None |
| scale-json/json-sizes/5-scale-json-4mb | execution_time | [170.941ns; 171.043ns] or [-0.030%; +0.030%] | None | None | None |
| scale-json/json-values | execution_time | [144.589ns; 144.660ns] or [-0.025%; +0.025%] | None | None | None |
| scale-json/json-values/0-scale-json-large | execution_time | [129.482ns; 129.584ns] or [-0.039%; +0.039%] | None | None | None |
| scale-json/json-values/1-scale-json-wide | execution_time | [132.785ns; 132.852ns] or [-0.025%; +0.025%] | None | None | None |
| scale-json/json-values/2-scale-json-deep | execution_time | [134.484ns; 134.583ns] or [-0.037%; +0.037%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 195.728ns | 198.177ns ± 1.782ns | 197.891ns ± 1.230ns | 199.131ns | 201.060ns | 203.782ns | 204.464ns | 3.32% | 0.893 | 0.888 | 0.90% | 0.126ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [197.930ns; 198.424ns] or [-0.125%; +0.125%] | None | None | None |
Group 5
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| scale/flags | execution_time | 141.072ns | 142.131ns ± 0.880ns | 142.029ns ± 0.119ns | 142.149ns | 142.503ns | 144.088ns | 153.068ns | 7.77% | 10.201 | 119.901 | 0.62% | 0.062ns | 1 | 200 |
| scale/numeric | execution_time | 2.624µs | 2.674µs ± 0.037µs | 2.671µs ± 0.002µs | 2.674µs | 2.678µs | 2.689µs | 3.180µs | 19.02% | 13.243 | 180.301 | 1.37% | 0.003µs | 1 | 200 |
| scale/one-of | execution_time | 17.523µs | 18.690µs ± 0.275µs | 18.674µs ± 0.021µs | 18.698µs | 18.772µs | 19.060µs | 22.209µs | 18.93% | 10.178 | 134.357 | 1.47% | 0.019µs | 1 | 200 |
| scale/one-of-many | execution_time | 57.982µs | 69.131µs ± 6.435µs | 67.981µs ± 0.546µs | 68.619µs | 73.837µs | 89.691µs | 146.163µs | 115.00% | 9.333 | 103.569 | 9.28% | 0.455µs | 1 | 200 |
| scale/one-of-small-numbers | execution_time | 1.012µs | 1.019µs ± 0.003µs | 1.019µs ± 0.001µs | 1.020µs | 1.022µs | 1.034µs | 1.054µs | 3.49% | 6.286 | 53.538 | 0.34% | 0.000µs | 1 | 200 |
| scale/regex | execution_time | 4.264µs | 4.462µs ± 0.027µs | 4.467µs ± 0.004µs | 4.471µs | 4.477µs | 4.483µs | 4.486µs | 0.42% | -5.284 | 30.897 | 0.61% | 0.002µs | 1 | 200 |
| scale/regex-many | execution_time | 58.724µs | 62.137µs ± 9.628µs | 60.112µs ± 0.615µs | 60.995µs | 68.263µs | 98.006µs | 147.951µs | 146.12% | 7.396 | 60.074 | 15.46% | 0.681µs | 1 | 200 |
| scale/rules | execution_time | 3.163µs | 3.199µs ± 0.007µs | 3.200µs ± 0.003µs | 3.203µs | 3.208µs | 3.217µs | 3.224µs | 0.75% | -0.869 | 6.246 | 0.20% | 0.000µs | 1 | 200 |
| scale/rules-sweep | execution_time | 1.843µs | 19.641µs ± 1.780µs | 20.018µs ± 0.117µs | 20.108µs | 20.198µs | 20.226µs | 20.265µs | 1.23% | -7.883 | 68.836 | 9.04% | 0.126µs | 1 | 200 |
| scale/semver | execution_time | 36.944µs | 37.292µs ± 0.462µs | 37.221µs ± 0.036µs | 37.268µs | 37.560µs | 38.534µs | 43.181µs | 16.01% | 10.788 | 131.141 | 1.24% | 0.033µs | 1 | 200 |
| scale/splits | execution_time | 7.004µs | 7.526µs ± 0.081µs | 7.521µs ± 0.010µs | 7.532µs | 7.569µs | 7.767µs | 8.204µs | 9.08% | 2.450 | 38.788 | 1.08% | 0.006µs | 1 | 200 |
| scale/time-windows | execution_time | 560.572ns | 564.092ns ± 2.502ns | 562.904ns ± 1.403ns | 566.495ns | 568.084ns | 569.387ns | 575.411ns | 2.22% | 0.888 | 0.624 | 0.44% | 0.177ns | 1 | 200 |
| scale/variations | execution_time | 24.043µs | 32.031µs ± 0.625µs | 32.129µs ± 0.035µs | 32.157µs | 32.198µs | 32.331µs | 32.529µs | 1.25% | -10.925 | 133.072 | 1.95% | 0.044µs | 1 | 200 |
| scale/wide-context | execution_time | 452.126ns | 456.127ns ± 0.850ns | 456.068ns ± 0.361ns | 456.462ns | 457.454ns | 457.898ns | 461.674ns | 1.23% | 0.509 | 11.510 | 0.19% | 0.060ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| scale/flags | execution_time | [142.009ns; 142.253ns] or [-0.086%; +0.086%] | None | None | None |
| scale/numeric | execution_time | [2.669µs; 2.679µs] or [-0.190%; +0.190%] | None | None | None |
| scale/one-of | execution_time | [18.652µs; 18.728µs] or [-0.204%; +0.204%] | None | None | None |
| scale/one-of-many | execution_time | [68.239µs; 70.023µs] or [-1.290%; +1.290%] | None | None | None |
| scale/one-of-small-numbers | execution_time | [1.019µs; 1.020µs] or [-0.047%; +0.047%] | None | None | None |
| scale/regex | execution_time | [4.458µs; 4.466µs] or [-0.084%; +0.084%] | None | None | None |
| scale/regex-many | execution_time | [60.803µs; 63.471µs] or [-2.147%; +2.147%] | None | None | None |
| scale/rules | execution_time | [3.199µs; 3.200µs] or [-0.028%; +0.028%] | None | None | None |
| scale/rules-sweep | execution_time | [19.395µs; 19.888µs] or [-1.256%; +1.256%] | None | None | None |
| scale/semver | execution_time | [37.228µs; 37.356µs] or [-0.172%; +0.172%] | None | None | None |
| scale/splits | execution_time | [7.515µs; 7.538µs] or [-0.150%; +0.150%] | None | None | None |
| scale/time-windows | execution_time | [563.745ns; 564.439ns] or [-0.061%; +0.061%] | None | None | None |
| scale/variations | execution_time | [31.944µs; 32.118µs] or [-0.270%; +0.270%] | None | None | None |
| scale/wide-context | execution_time | [456.009ns; 456.245ns] or [-0.026%; +0.026%] | None | None | None |
Group 6
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_255_failure | execution_time | 139.266µs | 139.843µs ± 0.195µs | 139.814µs ± 0.057µs | 139.891µs | 140.097µs | 140.732µs | 141.127µs | 0.94% | 2.777 | 14.734 | 0.14% | 0.014µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_255_failure | execution_time | [139.816µs; 139.870µs] or [-0.019%; +0.019%] | None | None | None |
Group 7
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_127_success | execution_time | 122.426µs | 122.882µs ± 0.305µs | 122.844µs ± 0.059µs | 122.908µs | 123.036µs | 123.726µs | 126.666µs | 3.11% | 9.944 | 118.048 | 0.25% | 0.022µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_127_success | execution_time | [122.839µs; 122.924µs] or [-0.034%; +0.034%] | None | None | None |
Group 8
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_255_success | execution_time | 242.877µs | 243.556µs ± 0.398µs | 243.491µs ± 0.091µs | 243.593µs | 243.920µs | 245.039µs | 248.015µs | 1.86% | 7.697 | 78.812 | 0.16% | 0.028µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_255_success | execution_time | [243.501µs; 243.611µs] or [-0.023%; +0.023%] | None | None | None |
Group 9
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | execution_time | 66.670ms | 66.941ms ± 0.638ms | 66.826ms ± 0.064ms | 66.863ms | 66.956ms | 69.258ms | 69.982ms | 4.72% | 4.593 | 19.433 | 0.93% | 0.128ms | 1 | 25 |
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | throughput | 14289.363op/s | 14939.825op/s ± 136.517op/s | 14964.318op/s ± 14.216op/s | 14980.679op/s | 14990.722op/s | 14997.462op/s | 14999.362op/s | 0.23% | -4.583 | 19.375 | 0.90% | 27.303op/s | 1 | 25 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | 35.866ms | 36.193ms ± 0.109ms | 36.195ms ± 0.054ms | 36.258ms | 36.345ms | 36.479ms | 36.501ms | 0.85% | -0.123 | 1.942 | 0.30% | 0.015ms | 1 | 50 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | 68490.639op/s | 69074.285op/s ± 207.715op/s | 69070.892op/s ± 103.225op/s | 69150.544op/s | 69371.897op/s | 69640.429op/s | 69704.570op/s | 0.92% | 0.158 | 1.952 | 0.30% | 29.375op/s | 1 | 50 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | 185.969ms | 186.316ms ± 0.246ms | 186.300ms ± 0.200ms | 186.514ms | 186.673ms | 186.780ms | 186.810ms | 0.27% | 0.222 | -1.083 | 0.13% | 0.049ms | 1 | 25 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | 5353.032op/s | 5367.246op/s ± 7.081op/s | 5367.679op/s ± 5.743op/s | 5372.076op/s | 5377.079op/s | 5377.223op/s | 5377.248op/s | 0.18% | -0.219 | -1.086 | 0.13% | 1.416op/s | 1 | 25 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | 882.383µs | 884.747µs ± 1.111µs | 884.597µs ± 0.601µs | 885.247µs | 886.579µs | 887.669µs | 888.285µs | 0.42% | 0.605 | 0.539 | 0.12% | 0.111µs | 1 | 100 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | 112576.509op/s | 113026.864op/s ± 141.836op/s | 113045.822op/s ± 76.732op/s | 113115.792op/s | 113219.826op/s | 113320.395op/s | 113329.422op/s | 0.25% | -0.596 | 0.526 | 0.12% | 14.184op/s | 1 | 100 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | execution_time | [66.691ms; 67.191ms] or [-0.374%; +0.374%] | None | None | None |
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | throughput | [14886.311op/s; 14993.339op/s] or [-0.358%; +0.358%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | [36.163ms; 36.223ms] or [-0.083%; +0.083%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | [69016.710op/s; 69131.859op/s] or [-0.083%; +0.083%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | [186.219ms; 186.412ms] or [-0.052%; +0.052%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | [5364.470op/s; 5370.022op/s] or [-0.052%; +0.052%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | [884.529µs; 884.965µs] or [-0.025%; +0.025%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | [112999.065op/s; 113054.664op/s] or [-0.025%; +0.025%] | None | None | None |
Group 10
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | execution_time | 61.333ms | 61.638ms ± 0.149ms | 61.635ms ± 0.102ms | 61.723ms | 61.858ms | 61.991ms | 62.031ms | 0.64% | 0.497 | 0.546 | 0.24% | 0.030ms | 1 | 25 |
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | throughput | 16121.098op/s | 16223.739op/s ± 39.057op/s | 16224.644op/s ± 26.811op/s | 16254.629op/s | 16268.612op/s | 16295.952op/s | 16304.511op/s | 0.49% | -0.481 | 0.526 | 0.24% | 7.811op/s | 1 | 25 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | 31.850ms | 32.019ms ± 0.072ms | 32.016ms ± 0.040ms | 32.055ms | 32.129ms | 32.200ms | 32.216ms | 0.62% | 0.252 | 0.458 | 0.22% | 0.010ms | 1 | 50 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | 77601.630op/s | 78079.800op/s ± 176.309op/s | 78086.335op/s ± 96.873op/s | 78181.392op/s | 78373.620op/s | 78444.135op/s | 78492.695op/s | 0.52% | -0.236 | 0.446 | 0.22% | 24.934op/s | 1 | 50 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | 192.360ms | 193.374ms ± 1.386ms | 193.211ms ± 0.205ms | 193.343ms | 193.615ms | 198.361ms | 199.852ms | 3.44% | 4.296 | 17.727 | 0.70% | 0.277ms | 1 | 25 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | 5003.702op/s | 5171.581op/s ± 35.991op/s | 5175.682op/s ± 5.497op/s | 5184.299op/s | 5190.224op/s | 5196.602op/s | 5198.593op/s | 0.44% | -4.269 | 17.574 | 0.68% | 7.198op/s | 1 | 25 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | 696.222µs | 706.382µs ± 4.041µs | 706.147µs ± 1.588µs | 708.054µs | 713.177µs | 714.588µs | 714.742µs | 1.22% | -0.140 | 0.165 | 0.57% | 0.404µs | 1 | 100 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | 139910.614op/s | 141571.008op/s ± 810.616op/s | 141613.485op/s ± 318.531op/s | 141879.511op/s | 143107.378op/s | 143518.298op/s | 143632.347op/s | 1.43% | 0.176 | 0.194 | 0.57% | 81.062op/s | 1 | 100 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | execution_time | [61.580ms; 61.697ms] or [-0.094%; +0.094%] | None | None | None |
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | throughput | [16208.429op/s; 16239.049op/s] or [-0.094%; +0.094%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | [31.999ms; 32.039ms] or [-0.063%; +0.063%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | [78030.930op/s; 78128.669op/s] or [-0.063%; +0.063%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | [192.830ms; 193.917ms] or [-0.281%; +0.281%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | [5157.472op/s; 5185.689op/s] or [-0.273%; +0.273%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | [705.590µs; 707.174µs] or [-0.112%; +0.112%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | [141412.130op/s; 141729.886op/s] or [-0.112%; +0.112%] | None | None | None |
Group 11
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/legacy | execution_time | 125.849ns | 126.683ns ± 0.238ns | 126.666ns ± 0.050ns | 126.719ns | 126.840ns | 127.234ns | 129.232ns | 2.03% | 6.552 | 68.002 | 0.19% | 0.017ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/legacy | execution_time | [126.650ns; 126.716ns] or [-0.026%; +0.026%] | None | None | None |
Group 12
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_127_failure | execution_time | 69.377µs | 69.682µs ± 0.083µs | 69.673µs ± 0.038µs | 69.715µs | 69.792µs | 69.984µs | 70.198µs | 0.75% | 2.135 | 11.805 | 0.12% | 0.006µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_127_failure | execution_time | [69.670µs; 69.693µs] or [-0.016%; +0.016%] | None | None | None |
Group 13
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | b31a251 | 1794961330 | leo.romanovsky/ffe-json-scale-bench |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 307.823µs | 315.316µs ± 15.846µs | 311.299µs ± 1.500µs | 313.310µs | 335.704µs | 390.182µs | 440.488µs | 41.50% | 5.575 | 35.362 | 5.01% | 1.120µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [313.120µs; 317.512µs] or [-0.696%; +0.696%] | None | None | None |
Baseline
Baseline benchmark details
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/memoized_16_references | execution_time | 3.486µs | 3.503µs ± 0.004µs | 3.503µs ± 0.002µs | 3.505µs | 3.510µs | 3.514µs | 3.517µs | 0.39% | -0.125 | 2.054 | 0.12% | 0.000µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/memoized_16_references | execution_time | [3.503µs; 3.504µs] or [-0.016%; +0.016%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/one_prerequisite | execution_time | 1.240µs | 1.249µs ± 0.002µs | 1.249µs ± 0.001µs | 1.250µs | 1.251µs | 1.255µs | 1.267µs | 1.42% | 3.837 | 39.172 | 0.15% | 0.000µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/one_prerequisite | execution_time | [1.249µs; 1.250µs] or [-0.021%; +0.021%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 195.386ns | 198.153ns ± 1.863ns | 197.994ns ± 1.243ns | 199.207ns | 201.513ns | 203.539ns | 203.920ns | 2.99% | 0.728 | 0.226 | 0.94% | 0.132ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [197.895ns; 198.411ns] or [-0.130%; +0.130%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_255_failure | execution_time | 139.154µs | 139.590µs ± 0.139µs | 139.580µs ± 0.069µs | 139.649µs | 139.781µs | 140.178µs | 140.233µs | 0.47% | 1.587 | 6.422 | 0.10% | 0.010µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_255_failure | execution_time | [139.571µs; 139.609µs] or [-0.014%; +0.014%] | None | None | None |
Group 5
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_127_success | execution_time | 122.398µs | 122.822µs ± 0.226µs | 122.791µs ± 0.061µs | 122.852µs | 123.008µs | 123.761µs | 124.687µs | 1.54% | 5.391 | 37.505 | 0.18% | 0.016µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_127_success | execution_time | [122.791µs; 122.853µs] or [-0.025%; +0.025%] | None | None | None |
Group 6
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_255_success | execution_time | 242.692µs | 243.416µs ± 0.213µs | 243.417µs ± 0.088µs | 243.501µs | 243.711µs | 244.094µs | 244.526µs | 0.46% | 0.809 | 5.621 | 0.09% | 0.015µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_255_success | execution_time | [243.386µs; 243.445µs] or [-0.012%; +0.012%] | None | None | None |
Group 7
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | execution_time | 66.195ms | 66.470ms ± 0.376ms | 66.385ms ± 0.073ms | 66.458ms | 66.767ms | 67.822ms | 68.151ms | 2.66% | 3.760 | 14.350 | 0.55% | 0.075ms | 1 | 25 |
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | throughput | 14673.353op/s | 15044.879op/s ± 83.443op/s | 15063.549op/s ± 16.460op/s | 15078.025op/s | 15103.923op/s | 15106.322op/s | 15106.866op/s | 0.29% | -3.723 | 14.133 | 0.54% | 16.689op/s | 1 | 25 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | 35.314ms | 35.397ms ± 0.048ms | 35.390ms ± 0.029ms | 35.434ms | 35.483ms | 35.492ms | 35.499ms | 0.31% | 0.249 | -0.612 | 0.14% | 0.007ms | 1 | 50 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | 70424.635op/s | 70627.167op/s ± 96.584op/s | 70641.899op/s ± 57.487op/s | 70682.753op/s | 70786.721op/s | 70791.110op/s | 70793.807op/s | 0.22% | -0.244 | -0.613 | 0.14% | 13.659op/s | 1 | 50 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | 186.060ms | 186.864ms ± 0.374ms | 186.790ms ± 0.280ms | 187.194ms | 187.384ms | 187.511ms | 187.548ms | 0.41% | 0.039 | -0.818 | 0.20% | 0.075ms | 1 | 25 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | 5331.964op/s | 5351.496op/s ± 10.720op/s | 5353.594op/s ± 7.999op/s | 5359.979op/s | 5363.851op/s | 5372.131op/s | 5374.608op/s | 0.39% | -0.032 | -0.812 | 0.20% | 2.144op/s | 1 | 25 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | 873.546µs | 875.605µs ± 0.812µs | 875.607µs ± 0.494µs | 876.049µs | 876.968µs | 877.501µs | 878.389µs | 0.32% | 0.386 | 0.747 | 0.09% | 0.081µs | 1 | 100 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | 113844.772op/s | 114206.830op/s ± 105.895op/s | 114206.425op/s ± 64.407op/s | 114271.902op/s | 114370.527op/s | 114418.806op/s | 114475.941op/s | 0.24% | -0.378 | 0.736 | 0.09% | 10.590op/s | 1 | 100 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | execution_time | [66.322ms; 66.617ms] or [-0.222%; +0.222%] | None | None | None |
| flagevaluation_evp/payloads/features/10flags_1000users_max_features | throughput | [15012.170op/s; 15077.588op/s] or [-0.217%; +0.217%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | [35.384ms; 35.411ms] or [-0.038%; +0.038%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | [70600.395op/s; 70653.938op/s] or [-0.038%; +0.038%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | [186.718ms; 187.011ms] or [-0.079%; +0.079%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | [5347.294op/s; 5355.698op/s] or [-0.079%; +0.079%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | [875.446µs; 875.764µs] or [-0.018%; +0.018%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | [114186.075op/s; 114227.585op/s] or [-0.018%; +0.018%] | None | None | None |
Group 8
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | execution_time | 61.164ms | 61.475ms ± 0.113ms | 61.500ms ± 0.052ms | 61.520ms | 61.612ms | 61.686ms | 61.710ms | 0.34% | -0.655 | 1.204 | 0.18% | 0.023ms | 1 | 25 |
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | throughput | 16204.880op/s | 16266.829op/s ± 29.858op/s | 16260.253op/s ± 13.853op/s | 16278.371op/s | 16317.618op/s | 16342.746op/s | 16349.489op/s | 0.55% | 0.670 | 1.224 | 0.18% | 5.972op/s | 1 | 25 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | 31.637ms | 31.794ms ± 0.069ms | 31.785ms ± 0.045ms | 31.844ms | 31.920ms | 31.950ms | 31.957ms | 0.54% | 0.351 | -0.067 | 0.21% | 0.010ms | 1 | 50 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | 78230.850op/s | 78631.455op/s ± 169.560op/s | 78653.385op/s ± 111.745op/s | 78739.857op/s | 78858.871op/s | 78979.498op/s | 79021.172op/s | 0.47% | -0.339 | -0.070 | 0.21% | 23.979op/s | 1 | 50 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | 192.411ms | 192.879ms ± 0.287ms | 192.816ms ± 0.218ms | 193.103ms | 193.246ms | 193.489ms | 193.565ms | 0.39% | 0.296 | -0.335 | 0.15% | 0.057ms | 1 | 25 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | 5166.220op/s | 5184.612op/s ± 7.713op/s | 5186.304op/s ± 5.876op/s | 5188.222op/s | 5196.032op/s | 5196.936op/s | 5197.197op/s | 0.21% | -0.289 | -0.344 | 0.15% | 1.543op/s | 1 | 25 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | 692.581µs | 701.412µs ± 3.003µs | 701.280µs ± 0.766µs | 702.223µs | 706.685µs | 707.469µs | 707.715µs | 0.92% | -0.588 | 1.686 | 0.43% | 0.300µs | 1 | 100 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | 141299.871op/s | 142572.143op/s ± 612.077op/s | 142596.310op/s ± 155.664op/s | 142719.003op/s | 144018.019op/s | 144277.432op/s | 144387.391op/s | 1.26% | 0.631 | 1.737 | 0.43% | 61.208op/s | 1 | 100 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | execution_time | [61.431ms; 61.519ms] or [-0.072%; +0.072%] | None | None | None |
| flagevaluation_evp/coalescer/features/10flags_1000users_max_features | throughput | [16255.125op/s; 16278.533op/s] or [-0.072%; +0.072%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | [31.775ms; 31.813ms] or [-0.060%; +0.060%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | [78584.456op/s; 78678.454op/s] or [-0.060%; +0.060%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | [192.766ms; 192.991ms] or [-0.058%; +0.058%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | [5181.588op/s; 5187.635op/s] or [-0.058%; +0.058%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | [700.823µs; 702.001µs] or [-0.084%; +0.084%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | [142452.179op/s; 142692.108op/s] or [-0.084%; +0.084%] | None | None | None |
Group 9
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/legacy | execution_time | 128.174ns | 128.991ns ± 0.314ns | 128.973ns ± 0.062ns | 129.035ns | 129.153ns | 129.396ns | 132.930ns | 3.07% | 9.837 | 122.506 | 0.24% | 0.022ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/legacy | execution_time | [128.948ns; 129.035ns] or [-0.034%; +0.034%] | None | None | None |
Group 10
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dependent_flags/chain_127_failure | execution_time | 69.092µs | 69.476µs ± 0.111µs | 69.464µs ± 0.034µs | 69.503µs | 69.571µs | 69.742µs | 70.594µs | 1.63% | 5.817 | 55.521 | 0.16% | 0.008µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| dependent_flags/chain_127_failure | execution_time | [69.461µs; 69.492µs] or [-0.022%; +0.022%] | None | None | None |
Group 11
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | c24d1e0 | 1794957957 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 301.766µs | 308.142µs ± 11.979µs | 304.886µs ± 1.504µs | 307.746µs | 320.464µs | 355.123µs | 427.270µs | 40.14% | 6.538 | 53.956 | 3.88% | 0.847µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [306.482µs; 309.802µs] or [-0.539%; +0.539%] | None | None | None |
What does this PR do?
Adds a benchmark and a conformance test for the
scaleandscale-jsonevaluation suites of ffe-system-test-data, and repins the fixture submodule to52205a7, the head of DataDog/ffe-system-test-data#78, which adds those suites. It changes no shipped code.tests/scale_suites.rsevaluates every case in both suites and checks value, reason, and, where a case asserts them,serialId,doLogand split features.benches/scale.rs(ffe-scale) is a criterion benchmark. It checks each case's value and reason, then measures each case file, and eachscale-jsoncase on its own. A counting global allocator prints bytes and allocations per evaluation onALLOClines.Motivation
A performance-sensitive customer serves large JSON flag values. The same suites now back benchmarks in each server SDK (DataDog/dd-trace-java#13485 pins the same commit), so the evaluator can be compared on equal inputs. The next PR in this stack caches the Python object that the
pyo3binding builds for a JSON variant. This PR provides the core baseline it is measured against: the core already shares each parsed JSON value, so the cost sits in the binding, not here.Baseline (
cargo bench -p libdd-ffe-test-suite --bench ffe-scale, criterion median, 16 vCPU workspace with load average about 6):scale-json719 nodes (scale-json-large)scale-json100 KB (match)scale-json1 MB (match)scale-json4 MB (match)scale-jsonshapes (number array, 1 MB string, escapes, nested arrays)scale/one-ofscale/one-of-manyscale/regex-manyscale/rules-sweepscale/semverscale/splitsscale/variationsThe core returns a JSON value in constant time at every size because the variant is parsed once and shared.
scale/semveris the one suite that allocates on every evaluation.Additional Notes
The
scale-jsonsuite holds about 16 MB of JSON, so the conformance test takes about 10 s in a debug build. The benchmark uses short criterion windows (0.5 s warm-up, 2 s measurement) to keep the CI benchmark job reasonable.How to test the change?
🤖 Generated with Claude Code