Repository navigation
perf(ffe)!: cache the Python object for each JSON variant - #2889
Draft
leoromanovsky wants to merge 1 commit into
Draft
leoromanovsky wants to merge 1 commit into
leoromanovsky wants to merge 1 commit into
Conversation
The pyo3 conversion of `AssignmentValue::Json` built a new `dict` and `list` tree on every evaluation, and on every read of the value. That costs milliseconds per evaluation for megabyte values. `AssignmentValue::Json` now carries a `JsonValueCache` that every clone shares with the variation in the configuration. The pyo3 binding fills it on first use through `PyOnceLock`, which detaches from the interpreter while it waits and so cannot deadlock with the GIL, and which is thread-safe on free-threaded builds. Each repeat evaluation of a variation returns the same object. The cache is dropped with the configuration, and a new configuration starts empty. Callers must treat the returned object as read-only: a change is visible to every later evaluation of that variation. BREAKING CHANGE: `AssignmentValue::Json` has a new `cache` field. Patterns must use `..`, and constructors set `cache: JsonValueCache::default()`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
📚 Documentation Check Results📦
|
Contributor
🔒 Cargo Deny Results📦
|
Contributor
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 478310f | Docs | View more details | Give us feedback! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is stacked on #2888 (scale-suite benchmark and conformance test). Review only the commits after it: 478310f.
What does this PR do?
Caches the Python object that the
pyo3binding builds for each JSON variant, so that repeat evaluations of a JSON flag return the same object instead of rebuilding the tree.AssignmentValue::Jsongains acache: JsonValueCachefield. With thepyo3feature, the cache holds anArc<PyOnceLock<Py<PyAny>>>. Without it,JsonValueCacheis a zero-size type, so the Rust core and the C FFI are unchanged.AssignmentValue, so all assignments of one variant share one cell. The cache lives exactly as long as theConfiguration. There is no global cache, no expiry, and no key by flag name. A new configuration builds new cells, so the first evaluation after a configuration update converts again.IntoPyObject for &AssignmentValuefills the cell on first use with the existingjson_to_pyobjectconversion and returns the cached object on later calls. The number conversion (u128/i128/f64) is unchanged.Motivation
A performance-sensitive customer serves large JSON flag values from Python. The core already shares each parsed JSON value (#2888 measures 120–145 ns with no allocation for 719 nodes up to 4 MB), but every Python read rebuilt the whole object tree: about 106 µs for 719 nodes and 63 ms for 4 MB, on every evaluation.
flowchart LR subgraph Before C1[Configuration<br/>Arc<serde_json::Value>] -->|every read| J1[json_to_pyobject] --> P1[new dict tree] end subgraph After C2[Configuration<br/>Arc<serde_json::Value>] -->|first read| J2[json_to_pyobject] --> K[PyOnceLock per variant] K -->|every read| P2[same dict] endConcurrency.
PyOnceLockdetaches from the interpreter while it waits for another thread to finish initializing, so a thread that holds the GIL cannot deadlock against the initializer. On free-threaded builds it is a real once-lock, so concurrent first reads produce exactly one object. If two threads race, one waits; neither sees a partly built object.Behavior changes
AssignmentValue::Jsonhas a new public field, so code that constructs or exhaustively destructures it must addcache(or..). In this repository onlylibdd-ffe-ffineeded a change.Results
Measured through dd-trace-py's provider (
DataDogProvider.resolve_object_details) with theopenfeature_scale_jsonbenchmark from DataDog/dd-trace-py#21875. The before and after builds come from the same cargo command and differ only in thelibdd-fferevision (v49.0.0 vs this PR). Each pair ran back to back (after, before, after, before) on one 16 vCPU workspace, Python 3.12, pyperf--fastmedian. Load averages at the start of each run: 3.06, 3.25, 2.51, 2.88. The load rose to 16 near the end of the last run, which explains the wide spread on its last 4 MB case.Repeat evaluation, same configuration:
First evaluation after a configuration load (the cache is empty, so it still converts):
The first evaluation is 8–17% faster even though it does the same conversion. I have not isolated why. A likely cause is that the tree is now freed with the configuration, outside the timed call, rather than when the previous result is dropped. Peak memory for the first evaluation is unchanged.
The Rust core numbers in #2888 do not change: without
pyo3, the cache is a zero-size field.No number-precision bugs were found in these suites.
Additional Notes
libdd-ffeat this commit and documents the read-only value.How to test the change?
repeat_evaluations_share_one_python_objectchecks that two evaluations of the same variant return the same object and that a new configuration returns a different one.cargo nextest run --workspacepasses (178 tests), and clippy is clean with and without--features pyo3.🤖 Generated with Claude Code