diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md new file mode 100644 index 0000000..4e2e270 --- /dev/null +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -0,0 +1,80 @@ +# Cycle: Persisted Offset Indexes + +Branch: `spiral/CYC-20260917-TTZ7C-4-conversion-offsets`, created from clean +`master` and verified on 2026-09-17. + +## Commitment And Evidence + +After verification found 181,723 of 181,724 persisted offsets incorrect in the +local curriculum-store dataset, the maintainer clarified: "the offset index +code is incorrect, i want to correct that code, in simplystore first." +The ID index was correct. Inspection of SimplyStore's converter confirms that +index creation precedes the final serialization, although custom index hooks +can change record sizes. This request authorizes fixing that evidenced defect. + +## Goal And Scope + +Make conversion persist offsets matching the final OD-JSONTag output after +custom indexing. Preserve the CLI, custom create hook, and disk format. Verify +default conversion and hooks that grow, shrink, and add records, including +multibyte text. Do not change curriculum-store data or publish a package in +this cycle. + +On 2026-09-17 the maintainer extended the goal to command changesets after a +worker probe reproduced stale offsets and a missing new-entity entry, saying +"But it needs fixing ...". This is an explicit re-plan of the active cycle: +both conversion and successful commands must publish offsets derived from +their final serialized bytes. Preserve sparse changeset record numbers, +pre-serialization custom mutations, integrity recording, and failure behavior. + +The maintainer subsequently approved routing finalization through the index +module so curriculum-store's customized methods work out of the box. Add an +optional finalization hook with a default fallback for existing wrappers; +preserve the current create/update/load contracts. This is a further explicit +scope refinement within the same active cycle. + +## Plan Continuity + +Deliberate deviation at the maintainer's request from the durability/integrity +sequence in SRC-001, REQ-001, and DES-001. CYC-018 is accepted; this bounded +downstream-discovered correction does not replace that roadmap. The adopted +Muze engineering/library cultures apply; compatibility and simplicity are +explicit constraints. The command extension follows the same demonstrated +defect and now also serves the existing durability and handler-ordering goals. + +## Evaluation + +Implemented final offset generation from the completed serialization. All 22 +tests in `npm test` pass, including three conversion CLI regressions. The +readback/writer pass also produced exact byte ranges for all 181,724 records of +the downstream data file in a disposable directory, taking about five seconds; +the source file's SHA-256 remained unchanged. Test-file lint and diff checks +pass. EVD-20260917-TTZ7C-8 records the evidence and limits. + +The command correction is also implemented and verified: all 53 tests across +startup/offset, handler, ACID, and crash/recovery suites pass. Both conversion +and commands now use the final-byte scanner/writer. The 181,724-record temporary +dataset check still passes, taking about 0.23 seconds including verification +with the shared scanner. EVD-20260917-TTZ7C-12 records the current evidence. + +The index-module dispatch refinement is implemented and verified: all 59 tests +pass, with asynchronous custom finalizers, failure propagation, and automatic +default fallback covered. A disposable fixture using byte-identical downstream +index sources and shared dependencies passed conversion and command offset +checks along with parent/root/niveau checks. EVD-20260917-TTZ7C-16 records the +evidence and the cross-checkout dependency limitation found by the first probe. +No merge, package release, downstream installation, or downstream-file repair +was performed in this cycle. + +## Human Acceptance + +Accepted by the maintainer on 2026-09-18: "And accept the current cycle for +simplystore." Acceptance covers final-byte offset generation for conversion +and command changesets, including optional custom finalization with the default +fallback, as evaluated above and recorded in EVD-20260917-TTZ7C-16. + +The cycle branch was rechecked as +`spiral/CYC-20260917-TTZ7C-4-conversion-offsets` when recording acceptance. +The maintainer separately requested explicit delegation in curriculum-store +and chose to retain its published SimplyStore dependency. That downstream +change is outside this cycle; using the new hook there awaits a package release. diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl b/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl new file mode 100644 index 0000000..8e8596e --- /dev/null +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl @@ -0,0 +1,9 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:CYC-20260917-TTZ7C-4 + a sd:Cycle ; + dcterms:identifier "CYC-20260917-TTZ7C-4" ; + sd:repositoryPath ".spiral/cycles/CYC-20260917-TTZ7C-4.md" ; + sd:status sd:Accepted . diff --git a/.spiral/designs/DES-20260917-TTZ7C-10.md b/.spiral/designs/DES-20260917-TTZ7C-10.md new file mode 100644 index 0000000..facc0a4 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-10.md @@ -0,0 +1,33 @@ +# Design: Derive Offsets From Final Serialized Records + +Keep the legacy custom index hook in its existing pre-serialization position: +it can change canonical state, including records outside the initial changed +list. Derive physical offsets only from the final bytes, not that changed list. + +Extract the existing OD-JSONTag framing validator's loop into a shared scanner +with an optional record visitor. Track global record numbers through `+N` +skips, report payload byte ranges for emitted records, and preserve all current +framing failures. The existing validation entry point delegates without a +visitor or allocating a complete index. This avoids parsing entity bodies or +resolving references to records absent from a sparse changeset. + +Add a serialized-byte writer to the existing offset module. It builds the same +JSON mapping and atomically replaces the appropriate base or command sidecar +using the already-installed write-file-atomic library. Empty changesets produce +`{}`. Keep the old create/update methods for custom-hook compatibility; the +final pass overwrites their provisional output. + +The command worker awaits this final write after the changeset write and before +integrity recording, the after-write fault point, and returning success. Failure +propagates through the existing failed-command path. A crash can still leave +uncommitted files; existing status-driven recovery must ignore them. This is not +a new atomic multi-file transaction or power-loss durability guarantee. + +Use the same writer in conversion, replacing its extra full parse. This retains +the earlier conversion guarantee and avoids maintaining two finalization paths. + +Verify with command-worker regressions for edits, additions, Unicode, leading +and interior skips, hook-introduced changes, successive changesets, empty +changesets, and failed final writes. Compare disk bytes with an independent +framing oracle. Run startup, handler, ACID, and crash/recovery suites because +command completion now awaits an additional write. diff --git a/.spiral/designs/DES-20260917-TTZ7C-10.ttl b/.spiral/designs/DES-20260917-TTZ7C-10.ttl new file mode 100644 index 0000000..86ea562 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-10.ttl @@ -0,0 +1,24 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260917-TTZ7C-10 + a sd:Design ; + dcterms:identifier "DES-20260917-TTZ7C-10" ; + sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-10.md" ; + sd:status sd:Active ; + sd:supersedes [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-6 ; + sd:gitCommit "5a118df699914f42e541daccb43a96b409101af2" + ] ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-9 ; + sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + ] ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-5 ; + sd:gitCommit "55629a9da8860b909fe465a1a828129a8c53edf2" + ] . diff --git a/.spiral/designs/DES-20260917-TTZ7C-14.md b/.spiral/designs/DES-20260917-TTZ7C-14.md new file mode 100644 index 0000000..2800bb3 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-14.md @@ -0,0 +1,33 @@ +# Design: Optional Index Finalization With Legacy Fallback + +Add `finalize(serialized, meta, uuid = null)` to the default index module. Its +default behavior delegates to the existing serialized offset writer. The +argument is the exact OD-JSONTag output (a string for conversion or Uint8Array +for commands), metadata retains the existing shape, and uuid is null for base +conversion or the command ID for a changeset. + +Export one `finalizeIndex(index, serialized, meta, uuid = null)` dispatcher +from that module. It invokes the configured index's finalize when present, +otherwise the default. Preserve the configured object's method receiver and +return its promise so both callers await completion and propagate rejection. +Do not mutate or spread the configured object: frozen legacy wrappers and +methods inherited through prototypes should remain usable. + +The converter and command worker import only this dispatcher, not the offset +module. Existing create/update/load hooks retain their invocation and mutation +semantics. No additional method is required in curriculum-store. A custom +finalizer owns its finalization behavior and can explicitly delegate to the +default to retain standard offsets while adding custom outputs; the dispatcher +does not invoke both and duplicate writes. Serialized input is read-only by +contract, and finalization must not mutate canonical state. + +Keep the existing position relative to data persistence, integrity recording, +and command completion. The final-byte writer and sparse framing behavior do +not change. This supplements DES-20260917-TTZ7C-10 with a compatibility boundary, +not a general handler registration system. Legacy ID/offset writes from custom +pre-serialization hooks remain possible for compatibility. + +Verify async completion, rejection, method receiver, custom replacement and +delegation, and fallback for frozen/legacy wrappers. Run a disposable probe +against curriculum-store's actual unchanged index module for both conversion +and a command on a representative graph, then run existing lifecycle suites. diff --git a/.spiral/designs/DES-20260917-TTZ7C-14.ttl b/.spiral/designs/DES-20260917-TTZ7C-14.ttl new file mode 100644 index 0000000..39f64ab --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-14.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260917-TTZ7C-14 + a sd:Design ; + dcterms:identifier "DES-20260917-TTZ7C-14" ; + sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-14.md" ; + sd:status sd:Active ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" + ] ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-10 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] . diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.md b/.spiral/designs/DES-20260917-TTZ7C-6.md new file mode 100644 index 0000000..b387fe2 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-6.md @@ -0,0 +1,31 @@ +# Design: Finalize Offsets After Conversion + +Superseded by DES-20260917-TTZ7C-10 when the maintainer extended the correction +to command changesets. The shared framing scanner replaces the full readback +below while preserving the final-byte offset guarantee. + +The converter owns the final output bytes. Its custom index hook can mutate +canonical objects after calling the built-in index writer. Positions captured +before that hook completes cannot describe the final serialized file. + +Keep invoking the existing custom hook once. Serialize its completed result, +write the output, parse that exact serialization with a fresh lazy OD-JSONTag +parser, then use the existing offset writer with the fresh result array. This +overwrites provisional offsets, includes newly serialized records, and retains +the current JSON sidecar and CLI formats. The extra pass is linear in the data. +Inspection of the installed parser shows that reading its position symbol +materializes the entity body, so the current offset writer adds a full readback +pass. Conversion already materializes its input; this bounded repair accepts +that cost and makes no laziness-preservation claim for offset finalization. + +Changing the writer alone cannot fix later mutations by a caller. Introducing +a new create/finalize hook contract would expand the compatibility surface for +this bounded repair. Reuse the existing parser/writer at the point where the +final bytes are known. This design makes no new atomic multi-file publication +or crash-safety claim for the conversion command. + +Verify through the actual CLI with an independent byte-framing oracle. Cover +default indexing, a hook that calls base indexing before changing lengths and +adding an anonymous record, and a custom hook that writes no offset sidecar. +Include multibyte text and a reused output directory. Run the normal startup +suite in addition to these conversion regressions. diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.ttl b/.spiral/designs/DES-20260917-TTZ7C-6.ttl new file mode 100644 index 0000000..b5f9922 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-6.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260917-TTZ7C-6 + a sd:Design ; + dcterms:identifier "DES-20260917-TTZ7C-6" ; + sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-6.md" ; + sd:status sd:Superseded ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-6 ; + sd:gitCommit "d63d2dead6bfc4b79b4265afd54c4de106c9ef47" + ] ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-5 ; + sd:gitCommit "55629a9da8860b909fe465a1a828129a8c53edf2" + ] . diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-12.md b/.spiral/evidence/EVD-20260917-TTZ7C-12.md new file mode 100644 index 0000000..dd06c2c --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-12.md @@ -0,0 +1,44 @@ +# Evidence: Command And Conversion Offset Finalization + +Verified on 2026-09-17 with Node v24.7.0. + +## Regression Evidence + +All six command-offset tests failed before implementation. They exposed offsets +into the prior file, missing appended entities and hook-introduced mutations, +incorrect successive changesets, stale no-op indexes, and a missing failure +when the final sidecar could not be written. + +After implementation, all 53 tests passed: + +- `npm test`: 28 startup, conversion, and command-offset tests. +- `npm run test:handlers`: 4 tests, including final offset-write failure causing + failed status with the previous committed state visible and reconstructable. +- `npm run test:acid`: 5 atomicity, consistency, isolation, and retry tests. +- `npm run test:durability`: 16 tests including crash boundaries, restart, + timeouts, retry safety, and integrity-enabled command completion. + +Offsets are checked against an independent disk-byte framing oracle, including +UTF-8 text, leading and interior skips, newly allocated records, hook mutations, +successive changeset files, and empty changesets. Replaying base plus changesets +preserves the expected logical data. + +The shared writer was also run against the local downstream 181,724-record base +file with output confined to a temporary directory. Every offset matched the +independent byte scan. Writer plus verification took 0.230 seconds in this one +local run. The temporary output was removed; downstream files were not written. + +Scoped ESLint checks on command-offset tests, handler tests, and recovery code +passed. Staged Spiral validation and whitespace checks passed before the +implementation commit. Tests requiring processes/sockets ran with sandbox +escalation; provenance validation used the existing temporary Python environment +with rdflib. + +## Limits + +Existing stale sidecars are not repaired automatically. The corrected package +has not been released or installed in curriculum-store. The final writer makes +successful commands await complete offsets; it does not make the changeset, +offset sidecar, and integrity record one atomic filesystem transaction. Failed +or interrupted commands may leave uncommitted derived files, which the existing +status-driven recovery excludes. Human acceptance remains pending. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-12.ttl b/.spiral/evidence/EVD-20260917-TTZ7C-12.ttl new file mode 100644 index 0000000..abeb1b6 --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-12.ttl @@ -0,0 +1,24 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260917-TTZ7C-12 + a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260917-TTZ7C-12" ; + sd:repositoryPath ".spiral/evidence/EVD-20260917-TTZ7C-12.md" ; + sd:status sd:Active ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-11 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-9 ; + sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + ] . diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-16.md b/.spiral/evidence/EVD-20260917-TTZ7C-16.md new file mode 100644 index 0000000..63c90f2 --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-16.md @@ -0,0 +1,55 @@ +# Evidence: Configured Index Finalization + +Verified on 2026-09-17 with Node v24.7.0. + +## Regression And Lifecycle Checks + +Before implementation, all five focused custom-finalization tests failed: +conversion and commands ignored the configured finalizer, including rejection. +After implementation all 59 tests passed: + +- `npm test`: 33 startup, conversion, and command-offset tests. +- `npm run test:handlers`: 5 tests, including custom finalizer rejection keeping + the prior committed state visible and independently reconstructable. +- `npm run test:acid`: 5 tests. +- `npm run test:durability`: 16 crash/recovery, timeout, retry, and integrity tests. + +Focused tests verify asynchronous completion, receiver and argument preservation, +explicit delegation to default offsets, replacement without an unconditional +default write, rejection propagation, and fallback for frozen legacy wrappers. +An additional direct dispatcher probe verified an inherited finalizer on a +frozen object, preserving its receiver, default UUID, and returned promise value. + +Scoped ESLint on `src/index.mjs`, conversion and command offset tests, and handler +tests passed. Staged provenance and whitespace checks passed before committing +the implementation. Process/socket suites used sandbox escalation. + +## Curriculum-Store Compatibility + +A disposable miniature curriculum graph exercised the actual downstream +`index.mjs`, parent, root, niveau, and utility source files copied byte-for-byte +from `curriculum-restapi-dev/curriculum-store/src`. The fixture linked SimplyStore +to this checkout and shared its JSONTag/OD-JSONTag dependencies. No source edits +were made to the copied wrapper, which still has no `finalize` method. + +Conversion and a command that lengthened a child title both produced offsets +matching an independent byte-framing oracle. Conversion preserved the expected +parent, root, and niveau links; replaying the command changeset preserved the +new title and niveau index. Temporary data was removed afterward. + +The initial direct cross-checkout probe failed before finalization because the +installed old SimplyStore and this checkout resolved distinct OD-JSONTag symbol +instances: the old offset writer could not see the parsed entity's position. +The shared-dependency fixture avoids that test-environment mismatch and verifies +unchanged wrapper source against the fixed dependency; it does not prove that +mixing arbitrary installed dependency copies works. Installed downstream +dependencies and data were not changed. + +## Limits + +This refines dispatch only; the previously verified final-byte scanner remains +unchanged. Custom finalizers own their output and must delegate to retain default +offset files. Read-only finalization inputs are a documented contract, not an +enforced sandbox. Failure can leave uncommitted files, not a partial successful +command. Existing stale sidecars are not repaired automatically. No release, +downstream installation, merge, or human acceptance is claimed. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-16.ttl b/.spiral/evidence/EVD-20260917-TTZ7C-16.ttl new file mode 100644 index 0000000..e17fbdc --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-16.ttl @@ -0,0 +1,29 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260917-TTZ7C-16 + a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260917-TTZ7C-16" ; + sd:repositoryPath ".spiral/evidence/EVD-20260917-TTZ7C-16.md" ; + sd:status sd:Active ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-15 ; + sd:gitCommit "6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-11 ; + sd:gitCommit "6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" + ] . diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-8.md b/.spiral/evidence/EVD-20260917-TTZ7C-8.md new file mode 100644 index 0000000..7f5bce3 --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-8.md @@ -0,0 +1,33 @@ +# Evidence: Conversion Offset Correction + +Verified on 2026-09-17 using Node v24.7.0. + +- Before the fix, the default conversion regression passed; both custom-hook + cases failed on incorrect ranges. In the mutating-hook case, record 1's range + was `[25,61]` while its final on-disk framing required `[25,121]`. +- After the fix, `npm test` passed all 22 tests: three conversion regressions + plus the existing startup/durability suite. CLI regressions cover multibyte + text, increased and decreased record lengths, hidden reference properties, + appended anonymous records, and replacing a stale sidecar when a custom hook + does not write one. The oracle walks on-disk byte framing independently of + the OD-JSONTag parser's position metadata. +- `node node_modules/eslint/bin/eslint.js test/conversion-offsets.mjs` passed. +- The final parse/writer pass was exercised on the 181,724-record downstream + `curriculum-restapi-dev/curriculum-store/data/data.jsontag`, generating an + index only in a temporary directory. Every range matched an independent + byte-framing scan; the source SHA-256 was unchanged. Measured elapsed time + was 5.034 seconds, including verification. Temporary output was removed. +- `git diff --check` and staged Spiral provenance validation passed before + implementation commit. Validation used + `/tmp/simplystore-spiral-venv/bin/python`, since the default Python lacked + rdflib. CLI tests required child-process execution outside the sandbox. + +## Limits + +The complete downstream custom conversion was not rerun. Its existing offset +file remains stale until the corrected converter is adopted and run there. +The added pass reads/materializes records with the current parser/writer; +the timing above is one local observation, not a performance guarantee. +There is no new atomic multi-file publication or crash-safety guarantee for +conversion, and command-update index hooks are not changed by this cycle. +Human acceptance and package publication remain pending. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-8.ttl b/.spiral/evidence/EVD-20260917-TTZ7C-8.ttl new file mode 100644 index 0000000..b5bef3b --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-8.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260917-TTZ7C-8 + a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260917-TTZ7C-8" ; + sd:repositoryPath ".spiral/evidence/EVD-20260917-TTZ7C-8.md" ; + sd:status sd:Active ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "bac44924fc7a3e566bf472f1511381bfc5a0cd1c" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-5 ; + sd:gitCommit "55629a9da8860b909fe465a1a828129a8c53edf2" + ] . diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-11.md b/.spiral/implementations/IMP-20260917-TTZ7C-11.md new file mode 100644 index 0000000..ea8f5c9 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-11.md @@ -0,0 +1,33 @@ +# Implementation: Serialized Command Offset Finalization + +The command worker awaits `finalizeIndex` after persisting the +changeset and before integrity recording and its existing after-write fault +point. Configured finalizers receive the serialized bytes, metadata, and command +ID with their original receiver. Missing finalizers fall back to the default +shared writer. The shared writer scans final bytes, includes records introduced by +custom hooks, respects skipped record numbers, and atomically replaces the +sidecar. Empty input produces an empty index. Write failures propagate through +the existing command failure path. + +`scanOdJsonTagRecords` extracts the existing recovery framing loop and adds +record-number tracking plus an optional visitor. `assertOdJsonTagFraming` +delegates without allocating offsets or parsing payloads. Existing validation +errors and integrity/status semantics are retained. The converter now shares +this writer; its separately governed implementation lineage is updated. + +All six command-worker regressions failed before implementation, reproducing +stale existing positions, absent new and hook-created records, incorrect +successive changesets, stale no-op indexes, and missing final-write failure. +A server-level test additionally checks that final-write failure leaves the +command failed and the previous committed state visible and reconstructable. + +The latest revision replaces direct offset-writer dispatch with optional +configured finalization. Tests now also cover asynchronous delegation, custom +replacement without an unconditional default write, rejection propagation, and +server-level rejection keeping the previous committed state reconstructable. +The shared scanner and writer themselves are unchanged in this revision. + +The legacy custom create/update interface remains callable. Such hooks can +still leave provisional files when they fail; the final pass guarantees the +index for successful conversion/commands. No new transaction framework or +on-disk format is introduced. No package release or downstream repair occurs. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl new file mode 100644 index 0000000..5b497be --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl @@ -0,0 +1,46 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260917-TTZ7C-11 + a sd:Implementation ; + dcterms:identifier "IMP-20260917-TTZ7C-11" ; + sd:repositoryPath ".spiral/implementations/IMP-20260917-TTZ7C-11.md" ; + sd:status sd:Active ; + sd:implementationLocation "src/command-worker-module.mjs#runCommand" ; + sd:implementationLocation "src/index.offset.mjs#writeSerialized" ; + sd:implementationLocation "src/recovery.mjs#scanOdJsonTagRecords" ; + sd:implementationLocation "src/recovery.mjs#assertOdJsonTagFraming" ; + sd:implementationLocation "test/command-offsets.mjs" ; + sd:implementationLocation "test/after-change-handlers.mjs" ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-10 ; + sd:gitCommit "641285f12e2cc638bae3835c944c4515b8e00b2a" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-001 ; + sd:gitCommit "85913e982e163c7b05a8e616c00e3f2014e775cc" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260819-09ZEF-7 ; + sd:gitCommit "471e28e350e8095adc2f956d16389790e47efdd0" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-11 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-14 ; + sd:gitCommit "f1911db50615e28b2e4e97a4852c73e412b933aa" + ] ; + sd:changeCausedBy [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" + ] ; + sd:implementationChangeKind sd:MixedChange . diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-15.md b/.spiral/implementations/IMP-20260917-TTZ7C-15.md new file mode 100644 index 0000000..896a757 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-15.md @@ -0,0 +1,17 @@ +# Implementation: Index Finalization Dispatch + +The default index object exposes `finalize(serialized, meta, uuid = null)`, +delegating to the existing serialized offset writer. The named `finalizeIndex` +export chooses a configured finalizer or that default and calls it with the +configured object as receiver. It forwards the result/promise unchanged. +This keeps frozen objects and inherited methods usable without mutating the +configured module or merging defaults into it. + +The converter and command worker await this dispatcher at their existing +post-write boundary. Their governed implementations are updated separately. +Existing create/update/load implementations are unchanged. A custom finalizer +replaces the default, and can delegate explicitly to retain standard offsets. + +README documents the optional contract, argument types, default fallback, +receiver, awaiting, read-only input expectation, and failure limits. No +dependency, on-disk format, or downstream wrapper change is introduced. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-15.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-15.ttl new file mode 100644 index 0000000..6881b5d --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-15.ttl @@ -0,0 +1,17 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260917-TTZ7C-15 + a sd:Implementation ; + dcterms:identifier "IMP-20260917-TTZ7C-15" ; + sd:repositoryPath ".spiral/implementations/IMP-20260917-TTZ7C-15.md" ; + sd:status sd:Active ; + sd:implementationLocation "src/index.mjs#finalize" ; + sd:implementationLocation "src/index.mjs#finalizeIndex" ; + sd:implementationLocation "README.md#custom-index-modules" ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-14 ; + sd:gitCommit "f1911db50615e28b2e4e97a4852c73e412b933aa" + ] . diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.md b/.spiral/implementations/IMP-20260917-TTZ7C-7.md new file mode 100644 index 0000000..55dd3bd --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.md @@ -0,0 +1,23 @@ +# Implementation: Final Conversion Offsets + +`scripts/convert.mjs` passes the final serialization through `finalizeIndex` +after writing the data. Configured finalizers are awaited; wrappers without one +use the default shared offset writer. The framing scanner generates fresh positions without +parsing entity bodies. This replaces the initial full readback in response to +the maintainer's command-offset extension. The custom create hook still runs +once, and the CLI and offset format remain unchanged. + +`test/conversion-offsets.mjs` exercises the CLI and independently walks UTF-8 +record framing. Before the fix, default conversion passed while the two custom +hook cases failed: a changed record ended at byte 121 but its stored range ended +at byte 61, and a hook writing no sidecar left the previous index untouched. +The tests also cover shrinking text, hidden reference properties, appended +records, and stale extra entries. They are included in `npm test`. + +The predecessor first governed this converter using a full parser readback. +The current version preserves the default final-byte guarantee while routing +through the index module, so custom finalizers can extend or replace it. +Additional tests cover a frozen legacy wrapper, asynchronous custom delegation +with the original receiver and final bytes, and rejection propagation. The +immediate predecessor used the shared scanner/writer directly; its lineage and +the new coordination design are recorded in Turtle. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl new file mode 100644 index 0000000..59dbf33 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl @@ -0,0 +1,33 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260917-TTZ7C-7 + a sd:Implementation ; + dcterms:identifier "IMP-20260917-TTZ7C-7" ; + sd:repositoryPath ".spiral/implementations/IMP-20260917-TTZ7C-7.md" ; + sd:status sd:Active ; + sd:implementationLocation "scripts/convert.mjs" ; + sd:implementationLocation "test/conversion-offsets.mjs" ; + sd:implementationLocation "package.json#scripts.test" ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-10 ; + sd:gitCommit "641285f12e2cc638bae3835c944c4515b8e00b2a" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-14 ; + sd:gitCommit "f1911db50615e28b2e4e97a4852c73e412b933aa" + ] ; + sd:changeCausedBy [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" + ] ; + sd:implementationChangeKind sd:MixedChange . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 63ded97..9c333d1 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,7 +18,7 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Current active Spiral cycle: none. Latest accepted cycle: `.spiral/cycles/CYC-20260819-09ZEF-5.md` (`Integrity Detection`). The runtime slice was accepted on 2026-08-19; the compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16. Integration must still be validated against the current target. +Current active Spiral cycle: none. Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18. Conversion and command changesets now finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. Integration must still be validated against the current target. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16. ## Intake State diff --git a/.spiral/requests/REQ-20260917-TTZ7C-13.md b/.spiral/requests/REQ-20260917-TTZ7C-13.md new file mode 100644 index 0000000..0d81c92 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-13.md @@ -0,0 +1,17 @@ +# Request: Customizable Index Finalization + +The maintainer accepted the proposed two-stage indexing arrangement and said: +"I think we should do that, since curriculum-store changes the index.* methods, +and I want those to work out of the box." + +Conversion and command workers currently call the offset module directly after +custom create/update hooks. A custom index module cannot participate in or +override that final step. Curriculum-store's existing wrapper exposes only +create/update/load, so adding a mandatory finalize method would break it. + +Route post-serialization finalization through SimplyStore's index module. Honor +an optional custom finalize method, await completion, and propagate errors. +Existing wrappers without the method must receive default offset finalization +automatically. Retain correct final-byte offsets and legacy hook semantics. +Verify the unchanged curriculum-store wrapper on representative graph fixtures. +No package publication, downstream dependency changes, or live data repair. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-13.ttl b/.spiral/requests/REQ-20260917-TTZ7C-13.ttl new file mode 100644 index 0000000..5236365 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-13.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260917-TTZ7C-13 + a sd:Request ; + dcterms:identifier "REQ-20260917-TTZ7C-13" ; + sd:repositoryPath ".spiral/requests/REQ-20260917-TTZ7C-13.md" ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . diff --git a/.spiral/requests/REQ-20260917-TTZ7C-5.md b/.spiral/requests/REQ-20260917-TTZ7C-5.md new file mode 100644 index 0000000..21462e2 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-5.md @@ -0,0 +1,15 @@ +# Request: Correct Conversion Offsets + +The maintainer explicitly requested fixing SimplyStore's offset-index code +after a read-only verification of curriculum-store's generated files found +offsets describing the data before derived fields were serialized. The current +conversation is the direct source; the cycle retains the operative quotation. + +Successful conversion must produce `index.offset.json` with one exact UTF-8 +byte range per record of the final output file, including records changed or +introduced by the custom `index.create(data, meta)` hook. Its ranges must allow +reading complete record payloads without stale boundaries or missing entries. + +Preserve existing CLI arguments, custom-hook invocation, OD-JSONTag framing, +and the offset sidecar's JSON shape. Verify both default and mutating custom +hooks. Existing curriculum-store files and package publication are out of scope. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-5.ttl b/.spiral/requests/REQ-20260917-TTZ7C-5.ttl new file mode 100644 index 0000000..6aa8b39 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-5.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260917-TTZ7C-5 + a sd:Request ; + dcterms:identifier "REQ-20260917-TTZ7C-5" ; + sd:repositoryPath ".spiral/requests/REQ-20260917-TTZ7C-5.md" ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . diff --git a/.spiral/requests/REQ-20260917-TTZ7C-9.md b/.spiral/requests/REQ-20260917-TTZ7C-9.md new file mode 100644 index 0000000..a9bbea5 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-9.md @@ -0,0 +1,18 @@ +# Request: Correct Command Changeset Offsets + +After the maintainer asked whether commands suffer the same defect, a direct +command-worker probe produced stored offsets `{0:[4,20],1:[25,39]}` for a +changeset whose records required `{0:[4,20],1:[25,81],3:[89,110]}`. The new +record was omitted and the updated record retained its original end position. +The maintainer then explicitly required fixing it: "But it needs fixing ...". + +Successful commands must persist an offset sidecar matching every record in +the final changeset, including newly allocated records and index-hook changes. +Skip records must preserve global record numbers; unchanged records must not +appear. Offsets count UTF-8 bytes within that changeset file. Empty changesets +have an empty index. The sidecar must be complete before success is returned. + +Preserve custom hook mutation semantics, command status and integrity ordering, +existing disk shapes, and restart behavior. Index write failure must prevent +success. Reuse the final-byte derivation for base conversion where practical; +do not republish packages or repair downstream files as part of this change. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-9.ttl b/.spiral/requests/REQ-20260917-TTZ7C-9.ttl new file mode 100644 index 0000000..13aeab4 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-9.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260917-TTZ7C-9 + a sd:Request ; + dcterms:identifier "REQ-20260917-TTZ7C-9" ; + sd:repositoryPath ".spiral/requests/REQ-20260917-TTZ7C-9.md" ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . diff --git a/README.md b/README.md index 2939600..f822972 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ Note: _There are known security issues in VM2, so the project will switch to V8- - [Background](#background) - [Install](#install) - [Usage](#usage) +- [Custom Index Modules](#custom-index-modules) - [Example Query](#examples) - [Goals](#goals) - [Roadmap](#roadmap) @@ -60,6 +61,44 @@ You should be able to go http://localhost:3000/query/ and see something like thi ![image](https://github.com/SimplyEdit/SimplyStore/assets/1006453/3bec6b97-ffa1-4114-9ed4-51a68f73476e) +## Custom Index Modules + +Use `indexFile` to configure a module whose default export provides the existing +`create(data, meta)`, `update(data, meta, changes)`, and `load(meta, uuid)` methods. +Conversion calls `create`; commands call `update` when changes are present. +These hooks run before final serialization and may update derived data. + +After writing the serialized data, SimplyStore awaits +`finalize(serialized, meta, uuid)`. Existing modules without this optional method +automatically use the default finalizer, which writes correct offset indexes. +Wrappers that only override `create`, `update`, and `load` need no changes. + +To extend finalization, delegate to the default implementation: + +```javascript +import index from '@muze-nl/simplystore/src/index.mjs' + +export default { + ...index, + async finalize(serialized, meta, uuid = null) { + await index.finalize(serialized, meta, uuid) + // Write any additional derived files here. + } +} +``` + +`serialized` contains the final OD-JSONTag output: a string during conversion +or a `Uint8Array` for a command changeset. `meta.data` is the output directory; +`uuid` is `null` for conversion or the command ID for a changeset. Treat the +serialized input and canonical data as read-only during finalization. The custom +method receives its original object as `this` and is called once, including for +empty command changesets. + +A custom finalizer replaces the default, so delegate as above to retain standard +offset files. Rejections fail conversion or the command before success is +reported. Files already written may remain; finalization is not an atomic +transaction across all data and index files. + ## Example query diff --git a/package.json b/package.json index b61fa98..7d948b7 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "scripts": { "start": "node src/run.mjs", - "test": "node --test test/durability-startup.mjs", + "test": "node --test test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs", "test:acid": "node --test test/acid-baseline.mjs", "test:durability": "node --test test/durability-crash.mjs", "test:handlers": "node --test test/after-change-handlers.mjs" diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 978d58b..966bb52 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -1,6 +1,7 @@ import JSONTag from '@muze-nl/jsontag' import serialize, { stringify } from '@muze-nl/od-jsontag/src/serialize.mjs' import Parser from '@muze-nl/od-jsontag' +import { finalizeIndex } from '../src/index.mjs' import fs from 'node:fs' import path from 'node:path' @@ -73,7 +74,9 @@ async function main() { strData = stringify(serialize(odData)) fs.writeFileSync(outputFile, strData) + // Custom indexes may change record sizes or add records after initial parsing. + await finalizeIndex(index, strData, meta) console.log('Converted data written to ',outputFile) } -main() \ No newline at end of file +main() diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 48e9033..79f7109 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -5,6 +5,7 @@ import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import writeFileAtomic from 'write-file-atomic' import { faultPoint } from './faults.mjs' import { appendIntegrityRecord } from './integrity.mjs' +import { finalizeIndex } from './index.mjs' let commands = {} let index = {} @@ -119,6 +120,8 @@ export default async function runCommand(commandStr, request) { let newfilename = basefile + '.' + task.id + '.' + extension await faultPoint('before-command-changeset-write') await writeFileAtomic(newfilename, uint8sab) + // Final bytes include new records and mutations made by the custom index hook. + await finalizeIndex(index, uint8sab, meta, task.id) if (integrityFile) { await appendIntegrityRecord(integrityFile, newfilename, uint8sab) } diff --git a/src/index.mjs b/src/index.mjs index d0cf70c..ed8b33d 100644 --- a/src/index.mjs +++ b/src/index.mjs @@ -1,7 +1,7 @@ import idIndex from './index.id.mjs' import offsetIndex from './index.offset.mjs' -export default { +const defaultIndex = { create(data, meta) { idIndex.create(data, meta) offsetIndex.create(data, meta) @@ -10,10 +10,20 @@ export default { idIndex.update(data, meta, changes) offsetIndex.update(data, meta, changes) }, + finalize(serialized, meta, uuid = null) { + return offsetIndex.writeSerialized(serialized, meta, uuid) + }, load(meta, uuid=null) { return { id: idIndex.load(uuid), offset: offsetIndex.load(uuid) } } -} \ No newline at end of file +} + +export function finalizeIndex(index, serialized, meta, uuid = null) { + const finalize = index.finalize ?? defaultIndex.finalize + return finalize.call(index, serialized, meta, uuid) +} + +export default defaultIndex diff --git a/src/index.offset.mjs b/src/index.offset.mjs index f0d4ba8..c2a9b2a 100644 --- a/src/index.offset.mjs +++ b/src/index.offset.mjs @@ -1,8 +1,20 @@ import fs from 'fs' -import JSONTag from '@muze-nl/jsontag' +import path from 'node:path' +import { Buffer } from 'node:buffer' +import writeFileAtomic from 'write-file-atomic' import { getIndex, position } from '@muze-nl/od-jsontag/src/symbols.mjs' +import { scanOdJsonTagRecords } from './recovery.mjs' export default { + async writeSerialized(buffer, meta, uuid = null) { + const filename = path.join(meta.data, uuid === null ? 'index.offset.json' : `index.offset.${uuid}.json`) + const bytes = typeof buffer === 'string' ? Buffer.from(buffer) : Buffer.from(buffer.buffer, buffer.byteOffset, buffer.byteLength) + const index = {} + scanOdJsonTagRecords(bytes, filename, 'offset source OD-JSONTag data', (record, start, end) => { + index[record] = [start, end] + }) + await writeFileAtomic(filename, JSON.stringify(index)) + }, create(data, meta) { console.log('creating '+meta.data+'/index.offset.json') // jsontag parse automatically fills meta.index.offset, so no need to create anything diff --git a/src/recovery.mjs b/src/recovery.mjs index 77a7f57..dfc1458 100644 --- a/src/recovery.mjs +++ b/src/recovery.mjs @@ -173,7 +173,12 @@ export function assertChangesetExists(dataFile, commandId) { } export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag data') { + scanOdJsonTagRecords(buffer, file, recordKind) +} + +export function scanOdJsonTagRecords(buffer, file, recordKind = 'OD-JSONTag data', onRecord) { let offset = 0 + let record = 0 while (offset < buffer.length) { while (offset < buffer.length && (buffer[offset] === 10 || buffer[offset] === 13)) { offset++ @@ -193,6 +198,7 @@ export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag da recordKind }) } + record += Number.parseInt(buffer.subarray(skipStart, offset).toString('utf8'), 10) continue } if (buffer[offset] !== 40) { @@ -221,6 +227,7 @@ export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag da recordKind }) } + onRecord?.(record++, offset, payloadEnd) offset = payloadEnd } } diff --git a/test/after-change-handlers.mjs b/test/after-change-handlers.mjs index 8e1d751..eaab756 100644 --- a/test/after-change-handlers.mjs +++ b/test/after-change-handlers.mjs @@ -107,6 +107,46 @@ export default { await assert.rejects(fs.access(path.join(fixture.dir, 'data.index-writes-then-fails.jsontag')), /ENOENT/) }) +test('handler lifecycle: final offset write failure prevents command commit', async t => { + const fixture = await makeServerFixture(t, { + initialData: '{"persons":[{"name":"Initial"}]}', + commandsSource: baseCommands + }) + const commandId = 'offset-write-fails' + await fs.mkdir(path.join(fixture.dir, `index.offset.${commandId}.json`)) + const port = await getOpenPort() + const server = startServer(t, fixture, {port}) + await waitForServer(server.child, server.getOutput, port) + + const status = await postAndWait(port, { + id: commandId, + name: 'addPerson', + value: {name: 'Should Not Commit'} + }, 'failed') + + assert.match(status.message, /EISDIR|EEXIST|ENOTEMPTY/) + assert.deepEqual(await queryNames(port), ['Initial']) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) +}) + +test('handler lifecycle: custom finalizer rejection prevents command commit', async t => { + const fixture = await makeServerFixture(t, { + initialData: '{"persons":[{"name":"Initial"}]}', + commandsSource: baseCommands, + indexSource: 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}' + }) + const port = await getOpenPort() + const server = startServer(t, fixture, {port}) + await waitForServer(server.child, server.getOutput, port) + + const status = await postAndWait(port, { + id: 'custom-finalizer-fails', name: 'addPerson', value: {name: 'Should Not Commit'} + }, 'failed') + assert.match(status.message, /custom finalization failed/) + assert.deepEqual(await queryNames(port), ['Initial']) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) +}) + test('handler lifecycle: current index update can mutate canonical state before commit', async t => { const fixture = await makeServerFixture(t, { initialData: '{"persons":[{"name":"Initial"}]}', diff --git a/test/command-offsets.mjs b/test/command-offsets.mjs new file mode 100644 index 0000000..e24b1f3 --- /dev/null +++ b/test/command-offsets.mjs @@ -0,0 +1,171 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import {fileURLToPath} from 'node:url' +import JSONTag from '@muze-nl/jsontag' +import Parser from '@muze-nl/od-jsontag' +import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' +import runCommand, {initialize} from '../src/command-worker-module.mjs' + +const baseIndex = new URL('../src/index.mjs', import.meta.url) + +async function fixture(t, indexSource) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-command-offset-')) + t.after(() => fs.rm(dir, {recursive: true, force: true})) + const commandsFile = path.join(dir, 'commands.mjs') + await fs.writeFile(commandsFile, `export default { + edit(data, command) { data.items[command.item || 0].name = command.value }, + add(data, command) { data.items.push(command.value) }, + noop() {} +}`) + let indexFile = fileURLToPath(baseIndex) + if (indexSource) { + indexFile = path.join(dir, 'index.mjs') + await fs.writeFile(indexFile, indexSource) + } + const base = serialize(JSONTag.parse('{"items":[{"name":"first"},{"name":"untouched"}]}')) + const task = { + data: [base], datafile: path.join(dir, 'data.jsontag'), commandsFile, indexFile, + meta: {data: dir, parts: 0, index: {id: new Map()}, resultArray: []} + } + await initialize(task) + return {dir, base, task} +} + +async function verifyOffsets(dir, id) { + const bytes = await fs.readFile(path.join(dir, `data.${id}.jsontag`)) + const stored = JSON.parse(await fs.readFile(path.join(dir, `index.offset.${id}.json`), 'utf8')) + const expected = {} + // Independent oracle: read record lengths and skips directly from disk bytes. + let cursor = 0, record = 0 + while (cursor < bytes.length) { + const header = bytes.subarray(cursor, cursor + 32).toString('ascii') + const skip = /^\+(\d+)/.exec(header) + if (skip) { + record += Number(skip[1]) + cursor += skip[0].length + } else { + const length = /^\((\d+)\)/.exec(header) + assert.ok(length, `record ${record} has valid framing`) + const start = cursor + length[0].length + cursor = start + Number(length[1]) + assert.ok(cursor <= bytes.length) + expected[record++] = [start, cursor] + } + if (cursor < bytes.length) assert.equal(bytes[cursor++], 10) + } + assert.deepEqual(stored, expected) + return {bytes, offsets: stored} +} + +function replay(base, ...changesets) { + const parser = new Parser('') + let data = parser.parse(base) + for (const bytes of changesets) data = parser.parse(bytes) + return data +} + +test('command edit indexes only its changed record using changeset UTF-8 offsets', async t => { + const {dir, base} = await fixture(t) + const value = 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a longer name' + await runCommand(JSONTag.stringify({id: 'edit', name: 'edit', value})) + const {bytes, offsets} = await verifyOffsets(dir, 'edit') + assert.deepEqual(Object.keys(offsets), ['1']) + assert.match(bytes.toString('utf8'), /^\+1\n/) + assert.equal(JSON.parse(bytes.subarray(...offsets[1]).toString('utf8')).name, value) + assert.equal(replay(base, bytes).items[0].name, value) +}) + +test('command addition indexes new entities and preserves interior skip numbers', async t => { + const {dir, base} = await fixture(t) + await runCommand(JSONTag.stringify({id: 'add', name: 'add', value: {name: 'new entity'}})) + const {bytes, offsets} = await verifyOffsets(dir, 'add') + assert.deepEqual(Object.keys(offsets), ['0', '3']) + assert.match(bytes.toString('utf8'), /\n\+2\n/) + assert.equal(JSON.parse(bytes.subarray(...offsets[3]).toString('utf8')).name, 'new entity') + assert.deepEqual(replay(base, bytes).items.map(e => e.name), ['first', 'untouched', 'new entity']) +}) + +test('command offsets include mutations and new records introduced by the index hook', async t => { + const {dir, base} = await fixture(t, `import index from ${JSON.stringify(baseIndex.href)} +export default { + update(data, meta, changes) { + index.update(data, meta, changes) + data.items[1].name = 'x' + data.items.push({name: 'derived by index'}) + } +}`) + await runCommand(JSONTag.stringify({id: 'hook', name: 'edit', value: 'longer command value'})) + const {bytes, offsets} = await verifyOffsets(dir, 'hook') + assert.deepEqual(Object.keys(offsets), ['0', '1', '2', '3']) + assert.deepEqual(replay(base, bytes).items.map(e => e.name), ['longer command value', 'x', 'derived by index']) +}) + +test('successive command indexes are relative to their own changeset files', async t => { + const {dir, base, task} = await fixture(t) + await runCommand(JSONTag.stringify({id: 'first', name: 'edit', value: 'longer first value'})) + const first = await verifyOffsets(dir, 'first') + await initialize({...task, data: [base, first.bytes], meta: {data: dir, parts: 1, index: {id: new Map()}, resultArray: []}}) + await runCommand(JSONTag.stringify({id: 'second', name: 'edit', item: 1, value: 'x'})) + const second = await verifyOffsets(dir, 'second') + assert.deepEqual(Object.keys(second.offsets), ['2']) + assert.match(second.bytes.toString('utf8'), /^\+2\n/) + assert.deepEqual(replay(base, first.bytes, second.bytes).items.map(e => e.name), ['longer first value', 'x']) + assert.deepEqual((await verifyOffsets(dir, 'first')).offsets, first.offsets) +}) + +test('no-op commands replace any stale sidecar with an empty offset index', async t => { + const {dir} = await fixture(t) + await fs.writeFile(path.join(dir, 'index.offset.noop.json'), '{"99":[0,100]}') + await runCommand('{"id":"noop","name":"noop"}') + const {bytes, offsets} = await verifyOffsets(dir, 'noop') + assert.equal(bytes.length, 0) + assert.deepEqual(offsets, {}) +}) + +test('failure to write final offsets rejects the command before success', async t => { + const {dir} = await fixture(t, 'export default {update() {}}') + await fs.mkdir(path.join(dir, 'index.offset.blocked.json')) + await assert.rejects(runCommand('{"id":"blocked","name":"edit","value":"changed"}'), /EISDIR|EEXIST|ENOTEMPTY/) +}) + +test('commands await custom finalization that delegates to the default index', async t => { + const {dir} = await fixture(t, `import index from ${JSON.stringify(baseIndex.href)} +import fs from 'node:fs/promises' +import {setTimeout} from 'node:timers/promises' +export default { + marker: 'custom receiver', + update(data) { data.items[1].name = 'index mutation' }, + async finalize(serialized, meta, uuid) { + await setTimeout(20) + if (!(serialized instanceof Uint8Array) || uuid !== 'custom') throw new Error('wrong finalizer arguments') + await index.finalize(serialized, meta, uuid) + await fs.writeFile(meta.data + '/finalized.json', JSON.stringify({marker: this.marker, uuid})) + } +}`) + await runCommand('{"id":"custom","name":"edit","value":"command mutation"}') + const {offsets} = await verifyOffsets(dir, 'custom') + assert.deepEqual(Object.keys(offsets), ['1', '2']) + assert.deepEqual(JSON.parse(await fs.readFile(path.join(dir, 'finalized.json'), 'utf8')), {marker: 'custom receiver', uuid: 'custom'}) +}) + +test('custom finalization can replace the default without an unconditional offset write', async t => { + const {dir} = await fixture(t, `import fs from 'node:fs/promises' +export default { + update() {}, + async finalize(serialized, meta, uuid) { + await fs.writeFile(meta.data + '/custom.' + uuid, serialized) + } +}`) + await runCommand('{"id":"replacement","name":"edit","value":"changed"}') + assert.deepEqual(await fs.readFile(path.join(dir, 'custom.replacement')), await fs.readFile(path.join(dir, 'data.replacement.jsontag'))) + await assert.rejects(fs.access(path.join(dir, 'index.offset.replacement.json')), /ENOENT/) +}) + +test('custom finalization rejection propagates from the command worker', async t => { + const {dir} = await fixture(t, 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}') + await assert.rejects(runCommand('{"id":"rejected","name":"edit","value":"changed"}'), /custom finalization failed/) + await assert.rejects(fs.access(path.join(dir, 'index.offset.rejected.json')), /ENOENT/) +}) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs new file mode 100644 index 0000000..f228aca --- /dev/null +++ b/test/conversion-offsets.mjs @@ -0,0 +1,127 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { execFile } from 'node:child_process' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import process from 'node:process' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' +import Parser from '@muze-nl/od-jsontag' + +const run = promisify(execFile) +const converter = fileURLToPath(new URL('../scripts/convert.mjs', import.meta.url)) +const baseIndex = new URL('../src/index.mjs', import.meta.url).href + +async function fixture(t) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-convert-')) + t.after(() => fs.rm(dir, {recursive: true, force: true})) + const input = path.join(dir, 'input.jsontag') + const output = path.join(dir, 'data.jsontag') + await fs.writeFile(input, '{"items":[{"name":"first"},{"name":"a long original name"}]}') + return {dir, input, output} +} + +async function convert({input, output}, indexFile) { + await run(process.execPath, [converter, input, output, ...indexFile ? [indexFile] : []], {timeout: 10000}) +} + +async function verifyOffsets({dir, output}) { + const bytes = await fs.readFile(output) + const offsets = JSON.parse(await fs.readFile(path.join(dir, 'index.offset.json'), 'utf8')) + const payloads = [] + let cursor = 0 + // Derive byte ranges directly from disk framing, independently of Parser.meta. + while (cursor < bytes.length) { + const match = /^\((\d+)\)/.exec(bytes.subarray(cursor, cursor + 32).toString('ascii')) + assert.ok(match, `record ${payloads.length} has a length prefix`) + const start = cursor + match[0].length + const end = start + Number(match[1]) + assert.ok(end <= bytes.length, 'record ends within the file') + assert.deepEqual(offsets[payloads.length], [start, end], `record ${payloads.length} byte range`) + payloads.push(bytes.subarray(...offsets[payloads.length]).toString('utf8')) + cursor = end + if (cursor < bytes.length) { + assert.equal(bytes[cursor], 10, 'records separated by a newline') + cursor++ + } + } + assert.equal(Object.keys(offsets).length, payloads.length, 'no missing or extra offsets') + return {payloads, data: new Parser('').parse(bytes.toString('utf8'))} +} + +test('default conversion offsets match every final record', async t => { + const files = await fixture(t) + await convert(files) + const {payloads, data} = await verifyOffsets(files) + assert.equal(payloads.length, 3) + assert.equal(data.items[1].name, 'a long original name') + const ids = JSON.parse(await fs.readFile(path.join(files.dir, 'index.id.json'), 'utf8')) + assert.match(payloads[ids['/first']], /id="\/first"/) + assert.match(payloads[ids['/second']], /id="\/second"/) +}) + +test('conversion finalizes offsets after custom indexing changes record sizes and adds records', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} +export default { + create(data, meta) { + index.create(data, meta) + data.items[0].name = 'caf\\u00e9 \\u6f22\\u5b57 \\ud83d\\ude00 with a much longer derived name' + Object.defineProperty(data.items[0], 'derived', {value: [data.items[1]], enumerable: false, configurable: true, writable: true}) + data.items[1].name = 'x' + data.items.push({name: 'new derived record'}) + } +} +`) + await convert(files, indexFile) + const {payloads, data} = await verifyOffsets(files) + assert.equal(payloads.length, 4) + assert.equal(data.items[0].name, 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a much longer derived name') + assert.equal(data.items[0].derived[0].name, 'x') + assert.equal(Object.getOwnPropertyDescriptor(data.items[0], 'derived').enumerable, false) + assert.equal(data.items[2].name, 'new derived record') + assert.match(payloads[3], /new derived record/) +}) + +test('conversion replaces stale offsets even when the custom hook writes no index', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, 'export default Object.freeze({ create(data) { data.items[0].name = "changed" } })\n') + await fs.writeFile(path.join(files.dir, 'index.offset.json'), '{"0":[0,1],"99":[2,3]}') + await convert(files, indexFile) + const {payloads, data} = await verifyOffsets(files) + assert.equal(payloads.length, 3) + assert.equal(data.items[0].name, 'changed') +}) + +test('conversion awaits custom finalization with the original receiver and final bytes', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} +import fs from 'node:fs/promises' +import {setTimeout} from 'node:timers/promises' +export default { + marker: 'custom receiver', + create(data) { data.items[0].name = 'final value' }, + async finalize(serialized, meta, uuid) { + await setTimeout(20) + if (uuid !== null || typeof serialized !== 'string' || !serialized.includes('final value')) throw new Error('wrong finalizer arguments') + await index.finalize(serialized, meta, uuid) + await fs.writeFile(meta.data + '/finalized.json', JSON.stringify({marker: this.marker, uuid})) + } +} +`) + await convert(files, indexFile) + await verifyOffsets(files) + assert.deepEqual(JSON.parse(await fs.readFile(path.join(files.dir, 'finalized.json'), 'utf8')), {marker: 'custom receiver', uuid: null}) +}) + +test('conversion propagates rejection from custom finalization', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, 'export default {create() {}, async finalize() { throw new Error("custom finalization failed") }}\n') + await assert.rejects(convert(files, indexFile), /custom finalization failed/) + await assert.rejects(fs.access(path.join(files.dir, 'index.offset.json')), /ENOENT/) +})