From 55629a9da8860b909fe465a1a828129a8c53edf2 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:33:07 +0200 Subject: [PATCH 01/14] Record conversion offset correction request and cycle --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 33 +++++++++++++++++++++++ .spiral/cycles/CYC-20260917-TTZ7C-4.ttl | 9 +++++++ .spiral/requests/REQ-20260917-TTZ7C-5.md | 15 +++++++++++ .spiral/requests/REQ-20260917-TTZ7C-5.ttl | 10 +++++++ 4 files changed, 67 insertions(+) create mode 100644 .spiral/cycles/CYC-20260917-TTZ7C-4.md create mode 100644 .spiral/cycles/CYC-20260917-TTZ7C-4.ttl create mode 100644 .spiral/requests/REQ-20260917-TTZ7C-5.md create mode 100644 .spiral/requests/REQ-20260917-TTZ7C-5.ttl diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md new file mode 100644 index 0000000..f2a82be --- /dev/null +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -0,0 +1,33 @@ +# Cycle: Conversion Offset Indexes + +Branch: `spiral/CYC-20260917-TTZ7C-4-conversion-offsets`, created from clean +`master` and verified on 2026-09-17. + +## Commitment And Evidence + +After verification found 181,723 of 181,724 persisted offsets incorrect in the +local curriculum-store dataset, the maintainer clarified: "the offset index +code is incorrect, i want to correct that code, in simplystore first." +The ID index was correct. Inspection of SimplyStore's converter confirms that +index creation precedes the final serialization, although custom index hooks +can change record sizes. This request authorizes fixing that evidenced defect. + +## Goal And Scope + +Make conversion persist offsets matching the final OD-JSONTag output after +custom indexing. Preserve the CLI, custom create hook, and disk format. Verify +default conversion and hooks that grow, shrink, and add records, including +multibyte text. Do not change curriculum-store data, publish a package, or +redesign command-update index hooks in this cycle. + +## Plan Continuity + +Deliberate deviation at the maintainer's request from the durability/integrity +sequence in SRC-001, REQ-001, and DES-001. CYC-018 is accepted; this bounded +downstream-discovered correction does not replace that roadmap. The adopted +Muze engineering/library cultures apply; compatibility and simplicity are +explicit constraints. Evidence claims remain limited to conversion. + +## Evaluation + +Pending implementation and verification. Human acceptance remains pending. diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl b/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl new file mode 100644 index 0000000..79daed7 --- /dev/null +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl @@ -0,0 +1,9 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:CYC-20260917-TTZ7C-4 + a sd:Cycle ; + dcterms:identifier "CYC-20260917-TTZ7C-4" ; + sd:repositoryPath ".spiral/cycles/CYC-20260917-TTZ7C-4.md" ; + sd:status sd:Active . diff --git a/.spiral/requests/REQ-20260917-TTZ7C-5.md b/.spiral/requests/REQ-20260917-TTZ7C-5.md new file mode 100644 index 0000000..21462e2 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-5.md @@ -0,0 +1,15 @@ +# Request: Correct Conversion Offsets + +The maintainer explicitly requested fixing SimplyStore's offset-index code +after a read-only verification of curriculum-store's generated files found +offsets describing the data before derived fields were serialized. The current +conversation is the direct source; the cycle retains the operative quotation. + +Successful conversion must produce `index.offset.json` with one exact UTF-8 +byte range per record of the final output file, including records changed or +introduced by the custom `index.create(data, meta)` hook. Its ranges must allow +reading complete record payloads without stale boundaries or missing entries. + +Preserve existing CLI arguments, custom-hook invocation, OD-JSONTag framing, +and the offset sidecar's JSON shape. Verify both default and mutating custom +hooks. Existing curriculum-store files and package publication are out of scope. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-5.ttl b/.spiral/requests/REQ-20260917-TTZ7C-5.ttl new file mode 100644 index 0000000..6aa8b39 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-5.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260917-TTZ7C-5 + a sd:Request ; + dcterms:identifier "REQ-20260917-TTZ7C-5" ; + sd:repositoryPath ".spiral/requests/REQ-20260917-TTZ7C-5.md" ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . From d63d2dead6bfc4b79b4265afd54c4de106c9ef47 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:34:49 +0200 Subject: [PATCH 02/14] Design offset finalization from serialized conversion output --- .spiral/designs/DES-20260917-TTZ7C-6.md | 24 ++++++++++++++++++++++++ .spiral/designs/DES-20260917-TTZ7C-6.ttl | 14 ++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 .spiral/designs/DES-20260917-TTZ7C-6.md create mode 100644 .spiral/designs/DES-20260917-TTZ7C-6.ttl diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.md b/.spiral/designs/DES-20260917-TTZ7C-6.md new file mode 100644 index 0000000..f228cfd --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-6.md @@ -0,0 +1,24 @@ +# Design: Finalize Offsets After Conversion + +The converter owns the final output bytes. Its custom index hook can mutate +canonical objects after calling the built-in index writer. Positions captured +before that hook completes cannot describe the final serialized file. + +Keep invoking the existing custom hook once. Serialize its completed result, +write the output, parse that exact serialization with a fresh lazy OD-JSONTag +parser, then use the existing offset writer with the fresh result array. This +overwrites provisional offsets, includes newly serialized records, and retains +the current JSON sidecar and CLI formats. The extra parse scans record framing; +it need not materialize entity bodies to recover positions. + +Changing the writer alone cannot fix later mutations by a caller. Introducing +a new create/finalize hook contract would expand the compatibility surface for +this bounded repair. Reuse the existing parser/writer at the point where the +final bytes are known. This design makes no new atomic multi-file publication +or crash-safety claim for the conversion command. + +Verify through the actual CLI with an independent byte-framing oracle. Cover +default indexing, a hook that calls base indexing before changing lengths and +adding an anonymous record, and a custom hook that writes no offset sidecar. +Include multibyte text and a reused output directory. Run the normal startup +suite in addition to these conversion regressions. diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.ttl b/.spiral/designs/DES-20260917-TTZ7C-6.ttl new file mode 100644 index 0000000..5638cc7 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-6.ttl @@ -0,0 +1,14 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260917-TTZ7C-6 + a sd:Design ; + dcterms:identifier "DES-20260917-TTZ7C-6" ; + sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-6.md" ; + sd:status sd:Active ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-5 ; + sd:gitCommit "55629a9da8860b909fe465a1a828129a8c53edf2" + ] . From 5a118df699914f42e541daccb43a96b409101af2 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:37:00 +0200 Subject: [PATCH 03/14] Clarify conversion offset readback cost from parser behavior --- .spiral/designs/DES-20260917-TTZ7C-6.md | 7 +++++-- .spiral/designs/DES-20260917-TTZ7C-6.ttl | 5 +++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.md b/.spiral/designs/DES-20260917-TTZ7C-6.md index f228cfd..21f2c14 100644 --- a/.spiral/designs/DES-20260917-TTZ7C-6.md +++ b/.spiral/designs/DES-20260917-TTZ7C-6.md @@ -8,8 +8,11 @@ Keep invoking the existing custom hook once. Serialize its completed result, write the output, parse that exact serialization with a fresh lazy OD-JSONTag parser, then use the existing offset writer with the fresh result array. This overwrites provisional offsets, includes newly serialized records, and retains -the current JSON sidecar and CLI formats. The extra parse scans record framing; -it need not materialize entity bodies to recover positions. +the current JSON sidecar and CLI formats. The extra pass is linear in the data. +Inspection of the installed parser shows that reading its position symbol +materializes the entity body, so the current offset writer adds a full readback +pass. Conversion already materializes its input; this bounded repair accepts +that cost and makes no laziness-preservation claim for offset finalization. Changing the writer alone cannot fix later mutations by a caller. Introducing a new create/finalize hook contract would expand the compatibility surface for diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.ttl b/.spiral/designs/DES-20260917-TTZ7C-6.ttl index 5638cc7..1a8b9e0 100644 --- a/.spiral/designs/DES-20260917-TTZ7C-6.ttl +++ b/.spiral/designs/DES-20260917-TTZ7C-6.ttl @@ -7,6 +7,11 @@ project:DES-20260917-TTZ7C-6 dcterms:identifier "DES-20260917-TTZ7C-6" ; sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-6.md" ; sd:status sd:Active ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-6 ; + sd:gitCommit "d63d2dead6bfc4b79b4265afd54c4de106c9ef47" + ] ; sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260917-TTZ7C-5 ; From bac44924fc7a3e566bf472f1511381bfc5a0cd1c Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:37:34 +0200 Subject: [PATCH 04/14] Fix conversion offsets after custom index mutations --- .../implementations/IMP-20260917-TTZ7C-7.md | 16 +++ .../implementations/IMP-20260917-TTZ7C-7.ttl | 17 ++++ package.json | 2 +- scripts/convert.mjs | 7 +- test/conversion-offsets.mjs | 97 +++++++++++++++++++ 5 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 .spiral/implementations/IMP-20260917-TTZ7C-7.md create mode 100644 .spiral/implementations/IMP-20260917-TTZ7C-7.ttl create mode 100644 test/conversion-offsets.mjs diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.md b/.spiral/implementations/IMP-20260917-TTZ7C-7.md new file mode 100644 index 0000000..657c330 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.md @@ -0,0 +1,16 @@ +# Implementation: Final Conversion Offsets + +`scripts/convert.mjs` reparses the final serialization after writing it and +regenerates the offset sidecar through the existing writer with fresh positions. +The legacy custom create hook still runs once. No format or CLI changes. + +`test/conversion-offsets.mjs` exercises the CLI and independently walks UTF-8 +record framing. Before the fix, default conversion passed while the two custom +hook cases failed: a changed record ended at byte 121 but its stored range ended +at byte 61, and a hook writing no sidecar left the previous index untouched. +The tests also cover shrinking text, hidden reference properties, appended +records, and stale extra entries. They are included in `npm test`. + +This previously ungoverned converter has no prior implementation artifact to +transform. Its existing hook/write behavior was characterized directly from +source and the failing CLI probes. Command-update indexing remains out of scope. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl new file mode 100644 index 0000000..2a028c0 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl @@ -0,0 +1,17 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260917-TTZ7C-7 + a sd:Implementation ; + dcterms:identifier "IMP-20260917-TTZ7C-7" ; + sd:repositoryPath ".spiral/implementations/IMP-20260917-TTZ7C-7.md" ; + sd:status sd:Active ; + sd:implementationLocation "scripts/convert.mjs" ; + sd:implementationLocation "test/conversion-offsets.mjs" ; + sd:implementationLocation "package.json#scripts.test" ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-6 ; + sd:gitCommit "5a118df699914f42e541daccb43a96b409101af2" + ] . diff --git a/package.json b/package.json index b61fa98..e89e6ea 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "scripts": { "start": "node src/run.mjs", - "test": "node --test test/durability-startup.mjs", + "test": "node --test test/durability-startup.mjs test/conversion-offsets.mjs", "test:acid": "node --test test/acid-baseline.mjs", "test:durability": "node --test test/durability-crash.mjs", "test:handlers": "node --test test/after-change-handlers.mjs" diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 978d58b..00d306f 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -1,6 +1,7 @@ import JSONTag from '@muze-nl/jsontag' import serialize, { stringify } from '@muze-nl/od-jsontag/src/serialize.mjs' import Parser from '@muze-nl/od-jsontag' +import offsetIndex from '../src/index.offset.mjs' import fs from 'node:fs' import path from 'node:path' @@ -73,7 +74,11 @@ async function main() { strData = stringify(serialize(odData)) fs.writeFileSync(outputFile, strData) + // Custom indexes may change record sizes or add records after initial parsing. + const finalParser = new Parser('') + const finalData = finalParser.parse(strData) + offsetIndex.create(finalData, {...meta, resultArray: finalParser.meta.resultArray}) console.log('Converted data written to ',outputFile) } -main() \ No newline at end of file +main() diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs new file mode 100644 index 0000000..7896e08 --- /dev/null +++ b/test/conversion-offsets.mjs @@ -0,0 +1,97 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { execFile } from 'node:child_process' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import process from 'node:process' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' +import Parser from '@muze-nl/od-jsontag' + +const run = promisify(execFile) +const converter = fileURLToPath(new URL('../scripts/convert.mjs', import.meta.url)) +const baseIndex = new URL('../src/index.mjs', import.meta.url).href + +async function fixture(t) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-convert-')) + t.after(() => fs.rm(dir, {recursive: true, force: true})) + const input = path.join(dir, 'input.jsontag') + const output = path.join(dir, 'data.jsontag') + await fs.writeFile(input, '{"items":[{"name":"first"},{"name":"a long original name"}]}') + return {dir, input, output} +} + +async function convert({input, output}, indexFile) { + await run(process.execPath, [converter, input, output, ...indexFile ? [indexFile] : []], {timeout: 10000}) +} + +async function verifyOffsets({dir, output}) { + const bytes = await fs.readFile(output) + const offsets = JSON.parse(await fs.readFile(path.join(dir, 'index.offset.json'), 'utf8')) + const payloads = [] + let cursor = 0 + // Derive byte ranges directly from disk framing, independently of Parser.meta. + while (cursor < bytes.length) { + const match = /^\((\d+)\)/.exec(bytes.subarray(cursor, cursor + 32).toString('ascii')) + assert.ok(match, `record ${payloads.length} has a length prefix`) + const start = cursor + match[0].length + const end = start + Number(match[1]) + assert.ok(end <= bytes.length, 'record ends within the file') + assert.deepEqual(offsets[payloads.length], [start, end], `record ${payloads.length} byte range`) + payloads.push(bytes.subarray(...offsets[payloads.length]).toString('utf8')) + cursor = end + if (cursor < bytes.length) { + assert.equal(bytes[cursor], 10, 'records separated by a newline') + cursor++ + } + } + assert.equal(Object.keys(offsets).length, payloads.length, 'no missing or extra offsets') + return {payloads, data: new Parser('').parse(bytes.toString('utf8'))} +} + +test('default conversion offsets match every final record', async t => { + const files = await fixture(t) + await convert(files) + const {payloads, data} = await verifyOffsets(files) + assert.equal(payloads.length, 3) + assert.equal(data.items[1].name, 'a long original name') + const ids = JSON.parse(await fs.readFile(path.join(files.dir, 'index.id.json'), 'utf8')) + assert.match(payloads[ids['/first']], /id="\/first"/) + assert.match(payloads[ids['/second']], /id="\/second"/) +}) + +test('conversion finalizes offsets after custom indexing changes record sizes and adds records', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} +export default { + create(data, meta) { + index.create(data, meta) + data.items[0].name = 'caf\\u00e9 \\u6f22\\u5b57 \\ud83d\\ude00 with a much longer derived name' + Object.defineProperty(data.items[0], 'derived', {value: [data.items[1]], enumerable: false, configurable: true, writable: true}) + data.items[1].name = 'x' + data.items.push({name: 'new derived record'}) + } +} +`) + await convert(files, indexFile) + const {payloads, data} = await verifyOffsets(files) + assert.equal(payloads.length, 4) + assert.equal(data.items[0].name, 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a much longer derived name') + assert.equal(data.items[0].derived[0].name, 'x') + assert.equal(Object.getOwnPropertyDescriptor(data.items[0], 'derived').enumerable, false) + assert.equal(data.items[2].name, 'new derived record') + assert.match(payloads[3], /new derived record/) +}) + +test('conversion replaces stale offsets even when the custom hook writes no index', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, 'export default { create(data) { data.items[0].name = "changed" } }\n') + await fs.writeFile(path.join(files.dir, 'index.offset.json'), '{"0":[0,1],"99":[2,3]}') + await convert(files, indexFile) + const {payloads, data} = await verifyOffsets(files) + assert.equal(payloads.length, 3) + assert.equal(data.items[0].name, 'changed') +}) From e54e615e50f0d095930dea2071cafad63aa85b24 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:38:32 +0200 Subject: [PATCH 05/14] Record conversion offset regression and dataset verification --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 10 ++++++- .spiral/evidence/EVD-20260917-TTZ7C-8.md | 33 +++++++++++++++++++++++ .spiral/evidence/EVD-20260917-TTZ7C-8.ttl | 19 +++++++++++++ 3 files changed, 61 insertions(+), 1 deletion(-) create mode 100644 .spiral/evidence/EVD-20260917-TTZ7C-8.md create mode 100644 .spiral/evidence/EVD-20260917-TTZ7C-8.ttl diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md index f2a82be..3bbbaee 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.md +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -30,4 +30,12 @@ explicit constraints. Evidence claims remain limited to conversion. ## Evaluation -Pending implementation and verification. Human acceptance remains pending. +Implemented final offset generation from the completed serialization. All 22 +tests in `npm test` pass, including three conversion CLI regressions. The +readback/writer pass also produced exact byte ranges for all 181,724 records of +the downstream data file in a disposable directory, taking about five seconds; +the source file's SHA-256 remained unchanged. Test-file lint and diff checks +pass. EVD-20260917-TTZ7C-8 records the evidence and limits. + +The code is ready for human evaluation on this branch. Human acceptance remains +pending; no merge, package release, or downstream-file repair was performed. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-8.md b/.spiral/evidence/EVD-20260917-TTZ7C-8.md new file mode 100644 index 0000000..7f5bce3 --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-8.md @@ -0,0 +1,33 @@ +# Evidence: Conversion Offset Correction + +Verified on 2026-09-17 using Node v24.7.0. + +- Before the fix, the default conversion regression passed; both custom-hook + cases failed on incorrect ranges. In the mutating-hook case, record 1's range + was `[25,61]` while its final on-disk framing required `[25,121]`. +- After the fix, `npm test` passed all 22 tests: three conversion regressions + plus the existing startup/durability suite. CLI regressions cover multibyte + text, increased and decreased record lengths, hidden reference properties, + appended anonymous records, and replacing a stale sidecar when a custom hook + does not write one. The oracle walks on-disk byte framing independently of + the OD-JSONTag parser's position metadata. +- `node node_modules/eslint/bin/eslint.js test/conversion-offsets.mjs` passed. +- The final parse/writer pass was exercised on the 181,724-record downstream + `curriculum-restapi-dev/curriculum-store/data/data.jsontag`, generating an + index only in a temporary directory. Every range matched an independent + byte-framing scan; the source SHA-256 was unchanged. Measured elapsed time + was 5.034 seconds, including verification. Temporary output was removed. +- `git diff --check` and staged Spiral provenance validation passed before + implementation commit. Validation used + `/tmp/simplystore-spiral-venv/bin/python`, since the default Python lacked + rdflib. CLI tests required child-process execution outside the sandbox. + +## Limits + +The complete downstream custom conversion was not rerun. Its existing offset +file remains stale until the corrected converter is adopted and run there. +The added pass reads/materializes records with the current parser/writer; +the timing above is one local observation, not a performance guarantee. +There is no new atomic multi-file publication or crash-safety guarantee for +conversion, and command-update index hooks are not changed by this cycle. +Human acceptance and package publication remain pending. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-8.ttl b/.spiral/evidence/EVD-20260917-TTZ7C-8.ttl new file mode 100644 index 0000000..b5bef3b --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-8.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260917-TTZ7C-8 + a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260917-TTZ7C-8" ; + sd:repositoryPath ".spiral/evidence/EVD-20260917-TTZ7C-8.md" ; + sd:status sd:Active ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "bac44924fc7a3e566bf472f1511381bfc5a0cd1c" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-5 ; + sd:gitCommit "55629a9da8860b909fe465a1a828129a8c53edf2" + ] . From 5a4fd47ae8b59468067fbe164294472550abb274 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:43:49 +0200 Subject: [PATCH 06/14] Extend offset correction to command changesets --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 19 ++++++++++++++----- .spiral/requests/REQ-20260917-TTZ7C-9.md | 18 ++++++++++++++++++ .spiral/requests/REQ-20260917-TTZ7C-9.ttl | 10 ++++++++++ 3 files changed, 42 insertions(+), 5 deletions(-) create mode 100644 .spiral/requests/REQ-20260917-TTZ7C-9.md create mode 100644 .spiral/requests/REQ-20260917-TTZ7C-9.ttl diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md index 3bbbaee..f376cdf 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.md +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -1,4 +1,4 @@ -# Cycle: Conversion Offset Indexes +# Cycle: Persisted Offset Indexes Branch: `spiral/CYC-20260917-TTZ7C-4-conversion-offsets`, created from clean `master` and verified on 2026-09-17. @@ -18,7 +18,14 @@ Make conversion persist offsets matching the final OD-JSONTag output after custom indexing. Preserve the CLI, custom create hook, and disk format. Verify default conversion and hooks that grow, shrink, and add records, including multibyte text. Do not change curriculum-store data, publish a package, or -redesign command-update index hooks in this cycle. +redesign the custom index-hook interface in this cycle. + +On 2026-09-17 the maintainer extended the goal to command changesets after a +worker probe reproduced stale offsets and a missing new-entity entry, saying +"But it needs fixing ...". This is an explicit re-plan of the active cycle: +both conversion and successful commands must publish offsets derived from +their final serialized bytes. Preserve sparse changeset record numbers, +pre-serialization custom mutations, integrity recording, and failure behavior. ## Plan Continuity @@ -26,7 +33,8 @@ Deliberate deviation at the maintainer's request from the durability/integrity sequence in SRC-001, REQ-001, and DES-001. CYC-018 is accepted; this bounded downstream-discovered correction does not replace that roadmap. The adopted Muze engineering/library cultures apply; compatibility and simplicity are -explicit constraints. Evidence claims remain limited to conversion. +explicit constraints. The command extension follows the same demonstrated +defect and now also serves the existing durability and handler-ordering goals. ## Evaluation @@ -37,5 +45,6 @@ the downstream data file in a disposable directory, taking about five seconds; the source file's SHA-256 remained unchanged. Test-file lint and diff checks pass. EVD-20260917-TTZ7C-8 records the evidence and limits. -The code is ready for human evaluation on this branch. Human acceptance remains -pending; no merge, package release, or downstream-file repair was performed. +The conversion result is verified. The newly authorized command correction is +in progress. Human acceptance remains pending; no merge, package release, or +downstream-file repair was performed. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-9.md b/.spiral/requests/REQ-20260917-TTZ7C-9.md new file mode 100644 index 0000000..a9bbea5 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-9.md @@ -0,0 +1,18 @@ +# Request: Correct Command Changeset Offsets + +After the maintainer asked whether commands suffer the same defect, a direct +command-worker probe produced stored offsets `{0:[4,20],1:[25,39]}` for a +changeset whose records required `{0:[4,20],1:[25,81],3:[89,110]}`. The new +record was omitted and the updated record retained its original end position. +The maintainer then explicitly required fixing it: "But it needs fixing ...". + +Successful commands must persist an offset sidecar matching every record in +the final changeset, including newly allocated records and index-hook changes. +Skip records must preserve global record numbers; unchanged records must not +appear. Offsets count UTF-8 bytes within that changeset file. Empty changesets +have an empty index. The sidecar must be complete before success is returned. + +Preserve custom hook mutation semantics, command status and integrity ordering, +existing disk shapes, and restart behavior. Index write failure must prevent +success. Reuse the final-byte derivation for base conversion where practical; +do not republish packages or repair downstream files as part of this change. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-9.ttl b/.spiral/requests/REQ-20260917-TTZ7C-9.ttl new file mode 100644 index 0000000..13aeab4 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-9.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260917-TTZ7C-9 + a sd:Request ; + dcterms:identifier "REQ-20260917-TTZ7C-9" ; + sd:repositoryPath ".spiral/requests/REQ-20260917-TTZ7C-9.md" ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . From 641285f12e2cc638bae3835c944c4515b8e00b2a Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:46:11 +0200 Subject: [PATCH 07/14] Design offset finalization for sparse command changesets --- .spiral/designs/DES-20260917-TTZ7C-10.md | 33 +++++++++++++++++++++++ .spiral/designs/DES-20260917-TTZ7C-10.ttl | 19 +++++++++++++ 2 files changed, 52 insertions(+) create mode 100644 .spiral/designs/DES-20260917-TTZ7C-10.md create mode 100644 .spiral/designs/DES-20260917-TTZ7C-10.ttl diff --git a/.spiral/designs/DES-20260917-TTZ7C-10.md b/.spiral/designs/DES-20260917-TTZ7C-10.md new file mode 100644 index 0000000..facc0a4 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-10.md @@ -0,0 +1,33 @@ +# Design: Derive Offsets From Final Serialized Records + +Keep the legacy custom index hook in its existing pre-serialization position: +it can change canonical state, including records outside the initial changed +list. Derive physical offsets only from the final bytes, not that changed list. + +Extract the existing OD-JSONTag framing validator's loop into a shared scanner +with an optional record visitor. Track global record numbers through `+N` +skips, report payload byte ranges for emitted records, and preserve all current +framing failures. The existing validation entry point delegates without a +visitor or allocating a complete index. This avoids parsing entity bodies or +resolving references to records absent from a sparse changeset. + +Add a serialized-byte writer to the existing offset module. It builds the same +JSON mapping and atomically replaces the appropriate base or command sidecar +using the already-installed write-file-atomic library. Empty changesets produce +`{}`. Keep the old create/update methods for custom-hook compatibility; the +final pass overwrites their provisional output. + +The command worker awaits this final write after the changeset write and before +integrity recording, the after-write fault point, and returning success. Failure +propagates through the existing failed-command path. A crash can still leave +uncommitted files; existing status-driven recovery must ignore them. This is not +a new atomic multi-file transaction or power-loss durability guarantee. + +Use the same writer in conversion, replacing its extra full parse. This retains +the earlier conversion guarantee and avoids maintaining two finalization paths. + +Verify with command-worker regressions for edits, additions, Unicode, leading +and interior skips, hook-introduced changes, successive changesets, empty +changesets, and failed final writes. Compare disk bytes with an independent +framing oracle. Run startup, handler, ACID, and crash/recovery suites because +command completion now awaits an additional write. diff --git a/.spiral/designs/DES-20260917-TTZ7C-10.ttl b/.spiral/designs/DES-20260917-TTZ7C-10.ttl new file mode 100644 index 0000000..40a858e --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-10.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260917-TTZ7C-10 + a sd:Design ; + dcterms:identifier "DES-20260917-TTZ7C-10" ; + sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-10.md" ; + sd:status sd:Active ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-9 ; + sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + ] ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-5 ; + sd:gitCommit "55629a9da8860b909fe465a1a828129a8c53edf2" + ] . From c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:50:24 +0200 Subject: [PATCH 08/14] Finalize changeset offsets from serialized bytes before command success --- .spiral/designs/DES-20260917-TTZ7C-10.ttl | 5 + .spiral/designs/DES-20260917-TTZ7C-6.md | 4 + .spiral/designs/DES-20260917-TTZ7C-6.ttl | 2 +- .../implementations/IMP-20260917-TTZ7C-11.md | 25 ++++ .../implementations/IMP-20260917-TTZ7C-11.ttl | 51 +++++++ .../implementations/IMP-20260917-TTZ7C-7.md | 15 +- .../implementations/IMP-20260917-TTZ7C-7.ttl | 17 ++- package.json | 2 +- scripts/convert.mjs | 4 +- src/command-worker-module.mjs | 3 + src/index.offset.mjs | 14 +- src/recovery.mjs | 7 + test/after-change-handlers.mjs | 22 +++ test/command-offsets.mjs | 132 ++++++++++++++++++ 14 files changed, 288 insertions(+), 15 deletions(-) create mode 100644 .spiral/implementations/IMP-20260917-TTZ7C-11.md create mode 100644 .spiral/implementations/IMP-20260917-TTZ7C-11.ttl create mode 100644 test/command-offsets.mjs diff --git a/.spiral/designs/DES-20260917-TTZ7C-10.ttl b/.spiral/designs/DES-20260917-TTZ7C-10.ttl index 40a858e..86ea562 100644 --- a/.spiral/designs/DES-20260917-TTZ7C-10.ttl +++ b/.spiral/designs/DES-20260917-TTZ7C-10.ttl @@ -7,6 +7,11 @@ project:DES-20260917-TTZ7C-10 dcterms:identifier "DES-20260917-TTZ7C-10" ; sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-10.md" ; sd:status sd:Active ; + sd:supersedes [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-6 ; + sd:gitCommit "5a118df699914f42e541daccb43a96b409101af2" + ] ; sd:satisfies [ a sd:ArtifactReference ; sd:artifact project:REQ-20260917-TTZ7C-9 ; diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.md b/.spiral/designs/DES-20260917-TTZ7C-6.md index 21f2c14..b387fe2 100644 --- a/.spiral/designs/DES-20260917-TTZ7C-6.md +++ b/.spiral/designs/DES-20260917-TTZ7C-6.md @@ -1,5 +1,9 @@ # Design: Finalize Offsets After Conversion +Superseded by DES-20260917-TTZ7C-10 when the maintainer extended the correction +to command changesets. The shared framing scanner replaces the full readback +below while preserving the final-byte offset guarantee. + The converter owns the final output bytes. Its custom index hook can mutate canonical objects after calling the built-in index writer. Positions captured before that hook completes cannot describe the final serialized file. diff --git a/.spiral/designs/DES-20260917-TTZ7C-6.ttl b/.spiral/designs/DES-20260917-TTZ7C-6.ttl index 1a8b9e0..b5f9922 100644 --- a/.spiral/designs/DES-20260917-TTZ7C-6.ttl +++ b/.spiral/designs/DES-20260917-TTZ7C-6.ttl @@ -6,7 +6,7 @@ project:DES-20260917-TTZ7C-6 a sd:Design ; dcterms:identifier "DES-20260917-TTZ7C-6" ; sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-6.md" ; - sd:status sd:Active ; + sd:status sd:Superseded ; sd:transforms [ a sd:ArtifactReference ; sd:artifact project:DES-20260917-TTZ7C-6 ; diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-11.md b/.spiral/implementations/IMP-20260917-TTZ7C-11.md new file mode 100644 index 0000000..e652cf7 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-11.md @@ -0,0 +1,25 @@ +# Implementation: Serialized Command Offset Finalization + +The command worker awaits `offsetIndex.writeSerialized` after persisting the +changeset and before integrity recording and its existing after-write fault +point. The shared writer scans final bytes, includes records introduced by +custom hooks, respects skipped record numbers, and atomically replaces the +sidecar. Empty input produces an empty index. Write failures propagate through +the existing command failure path. + +`scanOdJsonTagRecords` extracts the existing recovery framing loop and adds +record-number tracking plus an optional visitor. `assertOdJsonTagFraming` +delegates without allocating offsets or parsing payloads. Existing validation +errors and integrity/status semantics are retained. The converter now shares +this writer; its separately governed implementation lineage is updated. + +All six command-worker regressions failed before implementation, reproducing +stale existing positions, absent new and hook-created records, incorrect +successive changesets, stale no-op indexes, and missing final-write failure. +A server-level test additionally checks that final-write failure leaves the +command failed and the previous committed state visible and reconstructable. + +The legacy custom create/update interface remains callable. Such hooks can +still leave provisional files when they fail; the final pass guarantees the +index for successful conversion/commands. No new transaction framework or +on-disk format is introduced. No package release or downstream repair occurs. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl new file mode 100644 index 0000000..940d223 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl @@ -0,0 +1,51 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260917-TTZ7C-11 + a sd:Implementation ; + dcterms:identifier "IMP-20260917-TTZ7C-11" ; + sd:repositoryPath ".spiral/implementations/IMP-20260917-TTZ7C-11.md" ; + sd:status sd:Active ; + sd:implementationLocation "src/command-worker-module.mjs#runCommand" ; + sd:implementationLocation "src/index.offset.mjs#writeSerialized" ; + sd:implementationLocation "src/recovery.mjs#scanOdJsonTagRecords" ; + sd:implementationLocation "src/recovery.mjs#assertOdJsonTagFraming" ; + sd:implementationLocation "test/command-offsets.mjs" ; + sd:implementationLocation "test/after-change-handlers.mjs" ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-10 ; + sd:gitCommit "641285f12e2cc638bae3835c944c4515b8e00b2a" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-001 ; + sd:gitCommit "85913e982e163c7b05a8e616c00e3f2014e775cc" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260819-09ZEF-7 ; + sd:gitCommit "471e28e350e8095adc2f956d16389790e47efdd0" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:IMP-004 ; + sd:gitCommit "adccdd13f4de9c0501a5f6e0c10a2778aa2b0eb8" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:IMP-006 ; + sd:gitCommit "adccdd13f4de9c0501a5f6e0c10a2778aa2b0eb8" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260819-09ZEF-8 ; + sd:gitCommit "adccdd13f4de9c0501a5f6e0c10a2778aa2b0eb8" + ] ; + sd:changeCausedBy [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-9 ; + sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + ] ; + sd:implementationChangeKind sd:MixedChange . diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.md b/.spiral/implementations/IMP-20260917-TTZ7C-7.md index 657c330..18b1177 100644 --- a/.spiral/implementations/IMP-20260917-TTZ7C-7.md +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.md @@ -1,8 +1,10 @@ # Implementation: Final Conversion Offsets -`scripts/convert.mjs` reparses the final serialization after writing it and -regenerates the offset sidecar through the existing writer with fresh positions. -The legacy custom create hook still runs once. No format or CLI changes. +`scripts/convert.mjs` passes the final serialization to the shared offset writer +after writing the data. The framing scanner generates fresh positions without +parsing entity bodies. This replaces the initial full readback in response to +the maintainer's command-offset extension. The custom create hook still runs +once, and the CLI and offset format remain unchanged. `test/conversion-offsets.mjs` exercises the CLI and independently walks UTF-8 record framing. Before the fix, default conversion passed while the two custom @@ -11,6 +13,7 @@ at byte 61, and a hook writing no sidecar left the previous index untouched. The tests also cover shrinking text, hidden reference properties, appended records, and stale extra entries. They are included in `npm test`. -This previously ungoverned converter has no prior implementation artifact to -transform. Its existing hook/write behavior was characterized directly from -source and the failing CLI probes. Command-update indexing remains out of scope. +The predecessor first governed this converter using a full parser readback. +The current version preserves that correction's final-byte guarantee while +sharing the scanner/writer introduced for command changesets in +IMP-20260917-TTZ7C-11. Its lineage and new design are recorded in Turtle. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl index 2a028c0..e7fb82c 100644 --- a/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl @@ -12,6 +12,17 @@ project:IMP-20260917-TTZ7C-7 sd:implementationLocation "package.json#scripts.test" ; sd:implements [ a sd:ArtifactReference ; - sd:artifact project:DES-20260917-TTZ7C-6 ; - sd:gitCommit "5a118df699914f42e541daccb43a96b409101af2" - ] . + sd:artifact project:DES-20260917-TTZ7C-10 ; + sd:gitCommit "641285f12e2cc638bae3835c944c4515b8e00b2a" + ] ; + sd:transforms [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "bac44924fc7a3e566bf472f1511381bfc5a0cd1c" + ] ; + sd:changeCausedBy [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-9 ; + sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + ] ; + sd:implementationChangeKind sd:MixedChange . diff --git a/package.json b/package.json index e89e6ea..7d948b7 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "scripts": { "start": "node src/run.mjs", - "test": "node --test test/durability-startup.mjs test/conversion-offsets.mjs", + "test": "node --test test/durability-startup.mjs test/conversion-offsets.mjs test/command-offsets.mjs", "test:acid": "node --test test/acid-baseline.mjs", "test:durability": "node --test test/durability-crash.mjs", "test:handlers": "node --test test/after-change-handlers.mjs" diff --git a/scripts/convert.mjs b/scripts/convert.mjs index 00d306f..bbadeb0 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -75,9 +75,7 @@ async function main() { fs.writeFileSync(outputFile, strData) // Custom indexes may change record sizes or add records after initial parsing. - const finalParser = new Parser('') - const finalData = finalParser.parse(strData) - offsetIndex.create(finalData, {...meta, resultArray: finalParser.meta.resultArray}) + await offsetIndex.writeSerialized(strData, meta) console.log('Converted data written to ',outputFile) } diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 48e9033..53f9e79 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -5,6 +5,7 @@ import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import writeFileAtomic from 'write-file-atomic' import { faultPoint } from './faults.mjs' import { appendIntegrityRecord } from './integrity.mjs' +import offsetIndex from './index.offset.mjs' let commands = {} let index = {} @@ -119,6 +120,8 @@ export default async function runCommand(commandStr, request) { let newfilename = basefile + '.' + task.id + '.' + extension await faultPoint('before-command-changeset-write') await writeFileAtomic(newfilename, uint8sab) + // Final bytes include new records and mutations made by the custom index hook. + await offsetIndex.writeSerialized(uint8sab, meta, task.id) if (integrityFile) { await appendIntegrityRecord(integrityFile, newfilename, uint8sab) } diff --git a/src/index.offset.mjs b/src/index.offset.mjs index f0d4ba8..c2a9b2a 100644 --- a/src/index.offset.mjs +++ b/src/index.offset.mjs @@ -1,8 +1,20 @@ import fs from 'fs' -import JSONTag from '@muze-nl/jsontag' +import path from 'node:path' +import { Buffer } from 'node:buffer' +import writeFileAtomic from 'write-file-atomic' import { getIndex, position } from '@muze-nl/od-jsontag/src/symbols.mjs' +import { scanOdJsonTagRecords } from './recovery.mjs' export default { + async writeSerialized(buffer, meta, uuid = null) { + const filename = path.join(meta.data, uuid === null ? 'index.offset.json' : `index.offset.${uuid}.json`) + const bytes = typeof buffer === 'string' ? Buffer.from(buffer) : Buffer.from(buffer.buffer, buffer.byteOffset, buffer.byteLength) + const index = {} + scanOdJsonTagRecords(bytes, filename, 'offset source OD-JSONTag data', (record, start, end) => { + index[record] = [start, end] + }) + await writeFileAtomic(filename, JSON.stringify(index)) + }, create(data, meta) { console.log('creating '+meta.data+'/index.offset.json') // jsontag parse automatically fills meta.index.offset, so no need to create anything diff --git a/src/recovery.mjs b/src/recovery.mjs index 77a7f57..dfc1458 100644 --- a/src/recovery.mjs +++ b/src/recovery.mjs @@ -173,7 +173,12 @@ export function assertChangesetExists(dataFile, commandId) { } export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag data') { + scanOdJsonTagRecords(buffer, file, recordKind) +} + +export function scanOdJsonTagRecords(buffer, file, recordKind = 'OD-JSONTag data', onRecord) { let offset = 0 + let record = 0 while (offset < buffer.length) { while (offset < buffer.length && (buffer[offset] === 10 || buffer[offset] === 13)) { offset++ @@ -193,6 +198,7 @@ export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag da recordKind }) } + record += Number.parseInt(buffer.subarray(skipStart, offset).toString('utf8'), 10) continue } if (buffer[offset] !== 40) { @@ -221,6 +227,7 @@ export function assertOdJsonTagFraming(buffer, file, recordKind = 'OD-JSONTag da recordKind }) } + onRecord?.(record++, offset, payloadEnd) offset = payloadEnd } } diff --git a/test/after-change-handlers.mjs b/test/after-change-handlers.mjs index 8e1d751..9d178e2 100644 --- a/test/after-change-handlers.mjs +++ b/test/after-change-handlers.mjs @@ -107,6 +107,28 @@ export default { await assert.rejects(fs.access(path.join(fixture.dir, 'data.index-writes-then-fails.jsontag')), /ENOENT/) }) +test('handler lifecycle: final offset write failure prevents command commit', async t => { + const fixture = await makeServerFixture(t, { + initialData: '{"persons":[{"name":"Initial"}]}', + commandsSource: baseCommands + }) + const commandId = 'offset-write-fails' + await fs.mkdir(path.join(fixture.dir, `index.offset.${commandId}.json`)) + const port = await getOpenPort() + const server = startServer(t, fixture, {port}) + await waitForServer(server.child, server.getOutput, port) + + const status = await postAndWait(port, { + id: commandId, + name: 'addPerson', + value: {name: 'Should Not Commit'} + }, 'failed') + + assert.match(status.message, /EISDIR|EEXIST|ENOTEMPTY/) + assert.deepEqual(await queryNames(port), ['Initial']) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) +}) + test('handler lifecycle: current index update can mutate canonical state before commit', async t => { const fixture = await makeServerFixture(t, { initialData: '{"persons":[{"name":"Initial"}]}', diff --git a/test/command-offsets.mjs b/test/command-offsets.mjs new file mode 100644 index 0000000..4ab5bec --- /dev/null +++ b/test/command-offsets.mjs @@ -0,0 +1,132 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import {fileURLToPath} from 'node:url' +import JSONTag from '@muze-nl/jsontag' +import Parser from '@muze-nl/od-jsontag' +import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' +import runCommand, {initialize} from '../src/command-worker-module.mjs' + +const baseIndex = new URL('../src/index.mjs', import.meta.url) + +async function fixture(t, indexSource) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'simplystore-command-offset-')) + t.after(() => fs.rm(dir, {recursive: true, force: true})) + const commandsFile = path.join(dir, 'commands.mjs') + await fs.writeFile(commandsFile, `export default { + edit(data, command) { data.items[command.item || 0].name = command.value }, + add(data, command) { data.items.push(command.value) }, + noop() {} +}`) + let indexFile = fileURLToPath(baseIndex) + if (indexSource) { + indexFile = path.join(dir, 'index.mjs') + await fs.writeFile(indexFile, indexSource) + } + const base = serialize(JSONTag.parse('{"items":[{"name":"first"},{"name":"untouched"}]}')) + const task = { + data: [base], datafile: path.join(dir, 'data.jsontag'), commandsFile, indexFile, + meta: {data: dir, parts: 0, index: {id: new Map()}, resultArray: []} + } + await initialize(task) + return {dir, base, task} +} + +async function verifyOffsets(dir, id) { + const bytes = await fs.readFile(path.join(dir, `data.${id}.jsontag`)) + const stored = JSON.parse(await fs.readFile(path.join(dir, `index.offset.${id}.json`), 'utf8')) + const expected = {} + // Independent oracle: read record lengths and skips directly from disk bytes. + let cursor = 0, record = 0 + while (cursor < bytes.length) { + const header = bytes.subarray(cursor, cursor + 32).toString('ascii') + const skip = /^\+(\d+)/.exec(header) + if (skip) { + record += Number(skip[1]) + cursor += skip[0].length + } else { + const length = /^\((\d+)\)/.exec(header) + assert.ok(length, `record ${record} has valid framing`) + const start = cursor + length[0].length + cursor = start + Number(length[1]) + assert.ok(cursor <= bytes.length) + expected[record++] = [start, cursor] + } + if (cursor < bytes.length) assert.equal(bytes[cursor++], 10) + } + assert.deepEqual(stored, expected) + return {bytes, offsets: stored} +} + +function replay(base, ...changesets) { + const parser = new Parser('') + let data = parser.parse(base) + for (const bytes of changesets) data = parser.parse(bytes) + return data +} + +test('command edit indexes only its changed record using changeset UTF-8 offsets', async t => { + const {dir, base} = await fixture(t) + const value = 'caf\u00e9 \u6f22\u5b57 \ud83d\ude00 with a longer name' + await runCommand(JSONTag.stringify({id: 'edit', name: 'edit', value})) + const {bytes, offsets} = await verifyOffsets(dir, 'edit') + assert.deepEqual(Object.keys(offsets), ['1']) + assert.match(bytes.toString('utf8'), /^\+1\n/) + assert.equal(JSON.parse(bytes.subarray(...offsets[1]).toString('utf8')).name, value) + assert.equal(replay(base, bytes).items[0].name, value) +}) + +test('command addition indexes new entities and preserves interior skip numbers', async t => { + const {dir, base} = await fixture(t) + await runCommand(JSONTag.stringify({id: 'add', name: 'add', value: {name: 'new entity'}})) + const {bytes, offsets} = await verifyOffsets(dir, 'add') + assert.deepEqual(Object.keys(offsets), ['0', '3']) + assert.match(bytes.toString('utf8'), /\n\+2\n/) + assert.equal(JSON.parse(bytes.subarray(...offsets[3]).toString('utf8')).name, 'new entity') + assert.deepEqual(replay(base, bytes).items.map(e => e.name), ['first', 'untouched', 'new entity']) +}) + +test('command offsets include mutations and new records introduced by the index hook', async t => { + const {dir, base} = await fixture(t, `import index from ${JSON.stringify(baseIndex.href)} +export default { + update(data, meta, changes) { + index.update(data, meta, changes) + data.items[1].name = 'x' + data.items.push({name: 'derived by index'}) + } +}`) + await runCommand(JSONTag.stringify({id: 'hook', name: 'edit', value: 'longer command value'})) + const {bytes, offsets} = await verifyOffsets(dir, 'hook') + assert.deepEqual(Object.keys(offsets), ['0', '1', '2', '3']) + assert.deepEqual(replay(base, bytes).items.map(e => e.name), ['longer command value', 'x', 'derived by index']) +}) + +test('successive command indexes are relative to their own changeset files', async t => { + const {dir, base, task} = await fixture(t) + await runCommand(JSONTag.stringify({id: 'first', name: 'edit', value: 'longer first value'})) + const first = await verifyOffsets(dir, 'first') + await initialize({...task, data: [base, first.bytes], meta: {data: dir, parts: 1, index: {id: new Map()}, resultArray: []}}) + await runCommand(JSONTag.stringify({id: 'second', name: 'edit', item: 1, value: 'x'})) + const second = await verifyOffsets(dir, 'second') + assert.deepEqual(Object.keys(second.offsets), ['2']) + assert.match(second.bytes.toString('utf8'), /^\+2\n/) + assert.deepEqual(replay(base, first.bytes, second.bytes).items.map(e => e.name), ['longer first value', 'x']) + assert.deepEqual((await verifyOffsets(dir, 'first')).offsets, first.offsets) +}) + +test('no-op commands replace any stale sidecar with an empty offset index', async t => { + const {dir} = await fixture(t) + await fs.writeFile(path.join(dir, 'index.offset.noop.json'), '{"99":[0,100]}') + await runCommand('{"id":"noop","name":"noop"}') + const {bytes, offsets} = await verifyOffsets(dir, 'noop') + assert.equal(bytes.length, 0) + assert.deepEqual(offsets, {}) +}) + +test('failure to write final offsets rejects the command before success', async t => { + const {dir} = await fixture(t, 'export default {update() {}}') + await fs.mkdir(path.join(dir, 'index.offset.blocked.json')) + await assert.rejects(runCommand('{"id":"blocked","name":"edit","value":"changed"}'), /EISDIR|EEXIST|ENOTEMPTY/) +}) From 53f1d453da35a5deb5f1208307d0229b5ac88293 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 16:51:28 +0200 Subject: [PATCH 09/14] Record verified command offset and recovery behavior --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 11 ++++-- .spiral/evidence/EVD-20260917-TTZ7C-12.md | 44 ++++++++++++++++++++++ .spiral/evidence/EVD-20260917-TTZ7C-12.ttl | 24 ++++++++++++ 3 files changed, 76 insertions(+), 3 deletions(-) create mode 100644 .spiral/evidence/EVD-20260917-TTZ7C-12.md create mode 100644 .spiral/evidence/EVD-20260917-TTZ7C-12.ttl diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md index f376cdf..d3bb305 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.md +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -45,6 +45,11 @@ the downstream data file in a disposable directory, taking about five seconds; the source file's SHA-256 remained unchanged. Test-file lint and diff checks pass. EVD-20260917-TTZ7C-8 records the evidence and limits. -The conversion result is verified. The newly authorized command correction is -in progress. Human acceptance remains pending; no merge, package release, or -downstream-file repair was performed. +The command correction is also implemented and verified: all 53 tests across +startup/offset, handler, ACID, and crash/recovery suites pass. Both conversion +and commands now use the final-byte scanner/writer. The 181,724-record temporary +dataset check still passes, taking about 0.23 seconds including verification +with the shared scanner. EVD-20260917-TTZ7C-12 records the current evidence. + +The integrated result is ready for human evaluation. Human acceptance remains +pending; no merge, package release, or downstream-file repair was performed. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-12.md b/.spiral/evidence/EVD-20260917-TTZ7C-12.md new file mode 100644 index 0000000..dd06c2c --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-12.md @@ -0,0 +1,44 @@ +# Evidence: Command And Conversion Offset Finalization + +Verified on 2026-09-17 with Node v24.7.0. + +## Regression Evidence + +All six command-offset tests failed before implementation. They exposed offsets +into the prior file, missing appended entities and hook-introduced mutations, +incorrect successive changesets, stale no-op indexes, and a missing failure +when the final sidecar could not be written. + +After implementation, all 53 tests passed: + +- `npm test`: 28 startup, conversion, and command-offset tests. +- `npm run test:handlers`: 4 tests, including final offset-write failure causing + failed status with the previous committed state visible and reconstructable. +- `npm run test:acid`: 5 atomicity, consistency, isolation, and retry tests. +- `npm run test:durability`: 16 tests including crash boundaries, restart, + timeouts, retry safety, and integrity-enabled command completion. + +Offsets are checked against an independent disk-byte framing oracle, including +UTF-8 text, leading and interior skips, newly allocated records, hook mutations, +successive changeset files, and empty changesets. Replaying base plus changesets +preserves the expected logical data. + +The shared writer was also run against the local downstream 181,724-record base +file with output confined to a temporary directory. Every offset matched the +independent byte scan. Writer plus verification took 0.230 seconds in this one +local run. The temporary output was removed; downstream files were not written. + +Scoped ESLint checks on command-offset tests, handler tests, and recovery code +passed. Staged Spiral validation and whitespace checks passed before the +implementation commit. Tests requiring processes/sockets ran with sandbox +escalation; provenance validation used the existing temporary Python environment +with rdflib. + +## Limits + +Existing stale sidecars are not repaired automatically. The corrected package +has not been released or installed in curriculum-store. The final writer makes +successful commands await complete offsets; it does not make the changeset, +offset sidecar, and integrity record one atomic filesystem transaction. Failed +or interrupted commands may leave uncommitted derived files, which the existing +status-driven recovery excludes. Human acceptance remains pending. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-12.ttl b/.spiral/evidence/EVD-20260917-TTZ7C-12.ttl new file mode 100644 index 0000000..abeb1b6 --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-12.ttl @@ -0,0 +1,24 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260917-TTZ7C-12 + a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260917-TTZ7C-12" ; + sd:repositoryPath ".spiral/evidence/EVD-20260917-TTZ7C-12.md" ; + sd:status sd:Active ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-11 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-9 ; + sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + ] . From 59214a11f730c9657024c2f338513c86b1088dad Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 17:01:01 +0200 Subject: [PATCH 10/14] Record backward-compatible index finalization request --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 15 +++++++++++---- .spiral/requests/REQ-20260917-TTZ7C-13.md | 17 +++++++++++++++++ .spiral/requests/REQ-20260917-TTZ7C-13.ttl | 10 ++++++++++ 3 files changed, 38 insertions(+), 4 deletions(-) create mode 100644 .spiral/requests/REQ-20260917-TTZ7C-13.md create mode 100644 .spiral/requests/REQ-20260917-TTZ7C-13.ttl diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md index d3bb305..2e70ff9 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.md +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -17,8 +17,8 @@ can change record sizes. This request authorizes fixing that evidenced defect. Make conversion persist offsets matching the final OD-JSONTag output after custom indexing. Preserve the CLI, custom create hook, and disk format. Verify default conversion and hooks that grow, shrink, and add records, including -multibyte text. Do not change curriculum-store data, publish a package, or -redesign the custom index-hook interface in this cycle. +multibyte text. Do not change curriculum-store data or publish a package in +this cycle. On 2026-09-17 the maintainer extended the goal to command changesets after a worker probe reproduced stale offsets and a missing new-entity entry, saying @@ -27,6 +27,12 @@ both conversion and successful commands must publish offsets derived from their final serialized bytes. Preserve sparse changeset record numbers, pre-serialization custom mutations, integrity recording, and failure behavior. +The maintainer subsequently approved routing finalization through the index +module so curriculum-store's customized methods work out of the box. Add an +optional finalization hook with a default fallback for existing wrappers; +preserve the current create/update/load contracts. This is a further explicit +scope refinement within the same active cycle. + ## Plan Continuity Deliberate deviation at the maintainer's request from the durability/integrity @@ -51,5 +57,6 @@ and commands now use the final-byte scanner/writer. The 181,724-record temporary dataset check still passes, taking about 0.23 seconds including verification with the shared scanner. EVD-20260917-TTZ7C-12 records the current evidence. -The integrated result is ready for human evaluation. Human acceptance remains -pending; no merge, package release, or downstream-file repair was performed. +The offset correction is verified. The newly authorized index-module dispatch +refinement is in progress. Human acceptance remains pending; no merge, package +release, or downstream-file repair was performed. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-13.md b/.spiral/requests/REQ-20260917-TTZ7C-13.md new file mode 100644 index 0000000..0d81c92 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-13.md @@ -0,0 +1,17 @@ +# Request: Customizable Index Finalization + +The maintainer accepted the proposed two-stage indexing arrangement and said: +"I think we should do that, since curriculum-store changes the index.* methods, +and I want those to work out of the box." + +Conversion and command workers currently call the offset module directly after +custom create/update hooks. A custom index module cannot participate in or +override that final step. Curriculum-store's existing wrapper exposes only +create/update/load, so adding a mandatory finalize method would break it. + +Route post-serialization finalization through SimplyStore's index module. Honor +an optional custom finalize method, await completion, and propagate errors. +Existing wrappers without the method must receive default offset finalization +automatically. Retain correct final-byte offsets and legacy hook semantics. +Verify the unchanged curriculum-store wrapper on representative graph fixtures. +No package publication, downstream dependency changes, or live data repair. diff --git a/.spiral/requests/REQ-20260917-TTZ7C-13.ttl b/.spiral/requests/REQ-20260917-TTZ7C-13.ttl new file mode 100644 index 0000000..5236365 --- /dev/null +++ b/.spiral/requests/REQ-20260917-TTZ7C-13.ttl @@ -0,0 +1,10 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:REQ-20260917-TTZ7C-13 + a sd:Request ; + dcterms:identifier "REQ-20260917-TTZ7C-13" ; + sd:repositoryPath ".spiral/requests/REQ-20260917-TTZ7C-13.md" ; + sd:provenanceConfidence sd:Explicit ; + sd:status sd:Active . From f1911db50615e28b2e4e97a4852c73e412b933aa Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 17:03:07 +0200 Subject: [PATCH 11/14] Design optional finalization dispatch for custom index modules --- .spiral/designs/DES-20260917-TTZ7C-14.md | 33 +++++++++++++++++++++++ .spiral/designs/DES-20260917-TTZ7C-14.ttl | 19 +++++++++++++ 2 files changed, 52 insertions(+) create mode 100644 .spiral/designs/DES-20260917-TTZ7C-14.md create mode 100644 .spiral/designs/DES-20260917-TTZ7C-14.ttl diff --git a/.spiral/designs/DES-20260917-TTZ7C-14.md b/.spiral/designs/DES-20260917-TTZ7C-14.md new file mode 100644 index 0000000..2800bb3 --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-14.md @@ -0,0 +1,33 @@ +# Design: Optional Index Finalization With Legacy Fallback + +Add `finalize(serialized, meta, uuid = null)` to the default index module. Its +default behavior delegates to the existing serialized offset writer. The +argument is the exact OD-JSONTag output (a string for conversion or Uint8Array +for commands), metadata retains the existing shape, and uuid is null for base +conversion or the command ID for a changeset. + +Export one `finalizeIndex(index, serialized, meta, uuid = null)` dispatcher +from that module. It invokes the configured index's finalize when present, +otherwise the default. Preserve the configured object's method receiver and +return its promise so both callers await completion and propagate rejection. +Do not mutate or spread the configured object: frozen legacy wrappers and +methods inherited through prototypes should remain usable. + +The converter and command worker import only this dispatcher, not the offset +module. Existing create/update/load hooks retain their invocation and mutation +semantics. No additional method is required in curriculum-store. A custom +finalizer owns its finalization behavior and can explicitly delegate to the +default to retain standard offsets while adding custom outputs; the dispatcher +does not invoke both and duplicate writes. Serialized input is read-only by +contract, and finalization must not mutate canonical state. + +Keep the existing position relative to data persistence, integrity recording, +and command completion. The final-byte writer and sparse framing behavior do +not change. This supplements DES-20260917-TTZ7C-10 with a compatibility boundary, +not a general handler registration system. Legacy ID/offset writes from custom +pre-serialization hooks remain possible for compatibility. + +Verify async completion, rejection, method receiver, custom replacement and +delegation, and fallback for frozen/legacy wrappers. Run a disposable probe +against curriculum-store's actual unchanged index module for both conversion +and a command on a representative graph, then run existing lifecycle suites. diff --git a/.spiral/designs/DES-20260917-TTZ7C-14.ttl b/.spiral/designs/DES-20260917-TTZ7C-14.ttl new file mode 100644 index 0000000..39f64ab --- /dev/null +++ b/.spiral/designs/DES-20260917-TTZ7C-14.ttl @@ -0,0 +1,19 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:DES-20260917-TTZ7C-14 + a sd:Design ; + dcterms:identifier "DES-20260917-TTZ7C-14" ; + sd:repositoryPath ".spiral/designs/DES-20260917-TTZ7C-14.md" ; + sd:status sd:Active ; + sd:satisfies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" + ] ; + sd:derivedFrom [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-10 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] . From 6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 17:10:08 +0200 Subject: [PATCH 12/14] Route serialized finalization through configured index modules --- .../implementations/IMP-20260917-TTZ7C-11.md | 12 +++++- .../implementations/IMP-20260917-TTZ7C-11.ttl | 19 ++++----- .../implementations/IMP-20260917-TTZ7C-15.md | 17 ++++++++ .../implementations/IMP-20260917-TTZ7C-15.ttl | 17 ++++++++ .../implementations/IMP-20260917-TTZ7C-7.md | 14 ++++--- .../implementations/IMP-20260917-TTZ7C-7.ttl | 11 ++++-- README.md | 39 +++++++++++++++++++ scripts/convert.mjs | 4 +- src/command-worker-module.mjs | 4 +- src/index.mjs | 14 ++++++- test/after-change-handlers.mjs | 18 +++++++++ test/command-offsets.mjs | 39 +++++++++++++++++++ test/conversion-offsets.mjs | 32 ++++++++++++++- 13 files changed, 211 insertions(+), 29 deletions(-) create mode 100644 .spiral/implementations/IMP-20260917-TTZ7C-15.md create mode 100644 .spiral/implementations/IMP-20260917-TTZ7C-15.ttl diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-11.md b/.spiral/implementations/IMP-20260917-TTZ7C-11.md index e652cf7..ea8f5c9 100644 --- a/.spiral/implementations/IMP-20260917-TTZ7C-11.md +++ b/.spiral/implementations/IMP-20260917-TTZ7C-11.md @@ -1,8 +1,10 @@ # Implementation: Serialized Command Offset Finalization -The command worker awaits `offsetIndex.writeSerialized` after persisting the +The command worker awaits `finalizeIndex` after persisting the changeset and before integrity recording and its existing after-write fault -point. The shared writer scans final bytes, includes records introduced by +point. Configured finalizers receive the serialized bytes, metadata, and command +ID with their original receiver. Missing finalizers fall back to the default +shared writer. The shared writer scans final bytes, includes records introduced by custom hooks, respects skipped record numbers, and atomically replaces the sidecar. Empty input produces an empty index. Write failures propagate through the existing command failure path. @@ -19,6 +21,12 @@ successive changesets, stale no-op indexes, and missing final-write failure. A server-level test additionally checks that final-write failure leaves the command failed and the previous committed state visible and reconstructable. +The latest revision replaces direct offset-writer dispatch with optional +configured finalization. Tests now also cover asynchronous delegation, custom +replacement without an unconditional default write, rejection propagation, and +server-level rejection keeping the previous committed state reconstructable. +The shared scanner and writer themselves are unchanged in this revision. + The legacy custom create/update interface remains callable. Such hooks can still leave provisional files when they fail; the final pass guarantees the index for successful conversion/commands. No new transaction framework or diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl index 940d223..5b497be 100644 --- a/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl +++ b/.spiral/implementations/IMP-20260917-TTZ7C-11.ttl @@ -30,22 +30,17 @@ project:IMP-20260917-TTZ7C-11 ] ; sd:transforms [ a sd:ArtifactReference ; - sd:artifact project:IMP-004 ; - sd:gitCommit "adccdd13f4de9c0501a5f6e0c10a2778aa2b0eb8" + sd:artifact project:IMP-20260917-TTZ7C-11 ; + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" ] ; - sd:transforms [ - a sd:ArtifactReference ; - sd:artifact project:IMP-006 ; - sd:gitCommit "adccdd13f4de9c0501a5f6e0c10a2778aa2b0eb8" - ] ; - sd:transforms [ + sd:implements [ a sd:ArtifactReference ; - sd:artifact project:IMP-20260819-09ZEF-8 ; - sd:gitCommit "adccdd13f4de9c0501a5f6e0c10a2778aa2b0eb8" + sd:artifact project:DES-20260917-TTZ7C-14 ; + sd:gitCommit "f1911db50615e28b2e4e97a4852c73e412b933aa" ] ; sd:changeCausedBy [ a sd:ArtifactReference ; - sd:artifact project:REQ-20260917-TTZ7C-9 ; - sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" ] ; sd:implementationChangeKind sd:MixedChange . diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-15.md b/.spiral/implementations/IMP-20260917-TTZ7C-15.md new file mode 100644 index 0000000..896a757 --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-15.md @@ -0,0 +1,17 @@ +# Implementation: Index Finalization Dispatch + +The default index object exposes `finalize(serialized, meta, uuid = null)`, +delegating to the existing serialized offset writer. The named `finalizeIndex` +export chooses a configured finalizer or that default and calls it with the +configured object as receiver. It forwards the result/promise unchanged. +This keeps frozen objects and inherited methods usable without mutating the +configured module or merging defaults into it. + +The converter and command worker await this dispatcher at their existing +post-write boundary. Their governed implementations are updated separately. +Existing create/update/load implementations are unchanged. A custom finalizer +replaces the default, and can delegate explicitly to retain standard offsets. + +README documents the optional contract, argument types, default fallback, +receiver, awaiting, read-only input expectation, and failure limits. No +dependency, on-disk format, or downstream wrapper change is introduced. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-15.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-15.ttl new file mode 100644 index 0000000..6881b5d --- /dev/null +++ b/.spiral/implementations/IMP-20260917-TTZ7C-15.ttl @@ -0,0 +1,17 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:IMP-20260917-TTZ7C-15 + a sd:Implementation ; + dcterms:identifier "IMP-20260917-TTZ7C-15" ; + sd:repositoryPath ".spiral/implementations/IMP-20260917-TTZ7C-15.md" ; + sd:status sd:Active ; + sd:implementationLocation "src/index.mjs#finalize" ; + sd:implementationLocation "src/index.mjs#finalizeIndex" ; + sd:implementationLocation "README.md#custom-index-modules" ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-14 ; + sd:gitCommit "f1911db50615e28b2e4e97a4852c73e412b933aa" + ] . diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.md b/.spiral/implementations/IMP-20260917-TTZ7C-7.md index 18b1177..55dd3bd 100644 --- a/.spiral/implementations/IMP-20260917-TTZ7C-7.md +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.md @@ -1,7 +1,8 @@ # Implementation: Final Conversion Offsets -`scripts/convert.mjs` passes the final serialization to the shared offset writer -after writing the data. The framing scanner generates fresh positions without +`scripts/convert.mjs` passes the final serialization through `finalizeIndex` +after writing the data. Configured finalizers are awaited; wrappers without one +use the default shared offset writer. The framing scanner generates fresh positions without parsing entity bodies. This replaces the initial full readback in response to the maintainer's command-offset extension. The custom create hook still runs once, and the CLI and offset format remain unchanged. @@ -14,6 +15,9 @@ The tests also cover shrinking text, hidden reference properties, appended records, and stale extra entries. They are included in `npm test`. The predecessor first governed this converter using a full parser readback. -The current version preserves that correction's final-byte guarantee while -sharing the scanner/writer introduced for command changesets in -IMP-20260917-TTZ7C-11. Its lineage and new design are recorded in Turtle. +The current version preserves the default final-byte guarantee while routing +through the index module, so custom finalizers can extend or replace it. +Additional tests cover a frozen legacy wrapper, asynchronous custom delegation +with the original receiver and final bytes, and rejection propagation. The +immediate predecessor used the shared scanner/writer directly; its lineage and +the new coordination design are recorded in Turtle. diff --git a/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl index e7fb82c..59dbf33 100644 --- a/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl +++ b/.spiral/implementations/IMP-20260917-TTZ7C-7.ttl @@ -18,11 +18,16 @@ project:IMP-20260917-TTZ7C-7 sd:transforms [ a sd:ArtifactReference ; sd:artifact project:IMP-20260917-TTZ7C-7 ; - sd:gitCommit "bac44924fc7a3e566bf472f1511381bfc5a0cd1c" + sd:gitCommit "c6bfb419cf48e2248e1bb0a7ae3d8ec069d0a6c9" + ] ; + sd:implements [ + a sd:ArtifactReference ; + sd:artifact project:DES-20260917-TTZ7C-14 ; + sd:gitCommit "f1911db50615e28b2e4e97a4852c73e412b933aa" ] ; sd:changeCausedBy [ a sd:ArtifactReference ; - sd:artifact project:REQ-20260917-TTZ7C-9 ; - sd:gitCommit "5a4fd47ae8b59468067fbe164294472550abb274" + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" ] ; sd:implementationChangeKind sd:MixedChange . diff --git a/README.md b/README.md index 2939600..f822972 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ Note: _There are known security issues in VM2, so the project will switch to V8- - [Background](#background) - [Install](#install) - [Usage](#usage) +- [Custom Index Modules](#custom-index-modules) - [Example Query](#examples) - [Goals](#goals) - [Roadmap](#roadmap) @@ -60,6 +61,44 @@ You should be able to go http://localhost:3000/query/ and see something like thi ![image](https://github.com/SimplyEdit/SimplyStore/assets/1006453/3bec6b97-ffa1-4114-9ed4-51a68f73476e) +## Custom Index Modules + +Use `indexFile` to configure a module whose default export provides the existing +`create(data, meta)`, `update(data, meta, changes)`, and `load(meta, uuid)` methods. +Conversion calls `create`; commands call `update` when changes are present. +These hooks run before final serialization and may update derived data. + +After writing the serialized data, SimplyStore awaits +`finalize(serialized, meta, uuid)`. Existing modules without this optional method +automatically use the default finalizer, which writes correct offset indexes. +Wrappers that only override `create`, `update`, and `load` need no changes. + +To extend finalization, delegate to the default implementation: + +```javascript +import index from '@muze-nl/simplystore/src/index.mjs' + +export default { + ...index, + async finalize(serialized, meta, uuid = null) { + await index.finalize(serialized, meta, uuid) + // Write any additional derived files here. + } +} +``` + +`serialized` contains the final OD-JSONTag output: a string during conversion +or a `Uint8Array` for a command changeset. `meta.data` is the output directory; +`uuid` is `null` for conversion or the command ID for a changeset. Treat the +serialized input and canonical data as read-only during finalization. The custom +method receives its original object as `this` and is called once, including for +empty command changesets. + +A custom finalizer replaces the default, so delegate as above to retain standard +offset files. Rejections fail conversion or the command before success is +reported. Files already written may remain; finalization is not an atomic +transaction across all data and index files. + ## Example query diff --git a/scripts/convert.mjs b/scripts/convert.mjs index bbadeb0..966bb52 100644 --- a/scripts/convert.mjs +++ b/scripts/convert.mjs @@ -1,7 +1,7 @@ import JSONTag from '@muze-nl/jsontag' import serialize, { stringify } from '@muze-nl/od-jsontag/src/serialize.mjs' import Parser from '@muze-nl/od-jsontag' -import offsetIndex from '../src/index.offset.mjs' +import { finalizeIndex } from '../src/index.mjs' import fs from 'node:fs' import path from 'node:path' @@ -75,7 +75,7 @@ async function main() { fs.writeFileSync(outputFile, strData) // Custom indexes may change record sizes or add records after initial parsing. - await offsetIndex.writeSerialized(strData, meta) + await finalizeIndex(index, strData, meta) console.log('Converted data written to ',outputFile) } diff --git a/src/command-worker-module.mjs b/src/command-worker-module.mjs index 53f9e79..79f7109 100644 --- a/src/command-worker-module.mjs +++ b/src/command-worker-module.mjs @@ -5,7 +5,7 @@ import serialize from '@muze-nl/od-jsontag/src/serialize.mjs' import writeFileAtomic from 'write-file-atomic' import { faultPoint } from './faults.mjs' import { appendIntegrityRecord } from './integrity.mjs' -import offsetIndex from './index.offset.mjs' +import { finalizeIndex } from './index.mjs' let commands = {} let index = {} @@ -121,7 +121,7 @@ export default async function runCommand(commandStr, request) { await faultPoint('before-command-changeset-write') await writeFileAtomic(newfilename, uint8sab) // Final bytes include new records and mutations made by the custom index hook. - await offsetIndex.writeSerialized(uint8sab, meta, task.id) + await finalizeIndex(index, uint8sab, meta, task.id) if (integrityFile) { await appendIntegrityRecord(integrityFile, newfilename, uint8sab) } diff --git a/src/index.mjs b/src/index.mjs index d0cf70c..ed8b33d 100644 --- a/src/index.mjs +++ b/src/index.mjs @@ -1,7 +1,7 @@ import idIndex from './index.id.mjs' import offsetIndex from './index.offset.mjs' -export default { +const defaultIndex = { create(data, meta) { idIndex.create(data, meta) offsetIndex.create(data, meta) @@ -10,10 +10,20 @@ export default { idIndex.update(data, meta, changes) offsetIndex.update(data, meta, changes) }, + finalize(serialized, meta, uuid = null) { + return offsetIndex.writeSerialized(serialized, meta, uuid) + }, load(meta, uuid=null) { return { id: idIndex.load(uuid), offset: offsetIndex.load(uuid) } } -} \ No newline at end of file +} + +export function finalizeIndex(index, serialized, meta, uuid = null) { + const finalize = index.finalize ?? defaultIndex.finalize + return finalize.call(index, serialized, meta, uuid) +} + +export default defaultIndex diff --git a/test/after-change-handlers.mjs b/test/after-change-handlers.mjs index 9d178e2..eaab756 100644 --- a/test/after-change-handlers.mjs +++ b/test/after-change-handlers.mjs @@ -129,6 +129,24 @@ test('handler lifecycle: final offset write failure prevents command commit', as assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) }) +test('handler lifecycle: custom finalizer rejection prevents command commit', async t => { + const fixture = await makeServerFixture(t, { + initialData: '{"persons":[{"name":"Initial"}]}', + commandsSource: baseCommands, + indexSource: 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}' + }) + const port = await getOpenPort() + const server = startServer(t, fixture, {port}) + await waitForServer(server.child, server.getOutput, port) + + const status = await postAndWait(port, { + id: 'custom-finalizer-fails', name: 'addPerson', value: {name: 'Should Not Commit'} + }, 'failed') + assert.match(status.message, /custom finalization failed/) + assert.deepEqual(await queryNames(port), ['Initial']) + assert.deepEqual(await reconstructCommittedPersonNames(fixture), ['Initial']) +}) + test('handler lifecycle: current index update can mutate canonical state before commit', async t => { const fixture = await makeServerFixture(t, { initialData: '{"persons":[{"name":"Initial"}]}', diff --git a/test/command-offsets.mjs b/test/command-offsets.mjs index 4ab5bec..e24b1f3 100644 --- a/test/command-offsets.mjs +++ b/test/command-offsets.mjs @@ -130,3 +130,42 @@ test('failure to write final offsets rejects the command before success', async await fs.mkdir(path.join(dir, 'index.offset.blocked.json')) await assert.rejects(runCommand('{"id":"blocked","name":"edit","value":"changed"}'), /EISDIR|EEXIST|ENOTEMPTY/) }) + +test('commands await custom finalization that delegates to the default index', async t => { + const {dir} = await fixture(t, `import index from ${JSON.stringify(baseIndex.href)} +import fs from 'node:fs/promises' +import {setTimeout} from 'node:timers/promises' +export default { + marker: 'custom receiver', + update(data) { data.items[1].name = 'index mutation' }, + async finalize(serialized, meta, uuid) { + await setTimeout(20) + if (!(serialized instanceof Uint8Array) || uuid !== 'custom') throw new Error('wrong finalizer arguments') + await index.finalize(serialized, meta, uuid) + await fs.writeFile(meta.data + '/finalized.json', JSON.stringify({marker: this.marker, uuid})) + } +}`) + await runCommand('{"id":"custom","name":"edit","value":"command mutation"}') + const {offsets} = await verifyOffsets(dir, 'custom') + assert.deepEqual(Object.keys(offsets), ['1', '2']) + assert.deepEqual(JSON.parse(await fs.readFile(path.join(dir, 'finalized.json'), 'utf8')), {marker: 'custom receiver', uuid: 'custom'}) +}) + +test('custom finalization can replace the default without an unconditional offset write', async t => { + const {dir} = await fixture(t, `import fs from 'node:fs/promises' +export default { + update() {}, + async finalize(serialized, meta, uuid) { + await fs.writeFile(meta.data + '/custom.' + uuid, serialized) + } +}`) + await runCommand('{"id":"replacement","name":"edit","value":"changed"}') + assert.deepEqual(await fs.readFile(path.join(dir, 'custom.replacement')), await fs.readFile(path.join(dir, 'data.replacement.jsontag'))) + await assert.rejects(fs.access(path.join(dir, 'index.offset.replacement.json')), /ENOENT/) +}) + +test('custom finalization rejection propagates from the command worker', async t => { + const {dir} = await fixture(t, 'export default {update() {}, async finalize() { throw new Error("custom finalization failed") }}') + await assert.rejects(runCommand('{"id":"rejected","name":"edit","value":"changed"}'), /custom finalization failed/) + await assert.rejects(fs.access(path.join(dir, 'index.offset.rejected.json')), /ENOENT/) +}) diff --git a/test/conversion-offsets.mjs b/test/conversion-offsets.mjs index 7896e08..f228aca 100644 --- a/test/conversion-offsets.mjs +++ b/test/conversion-offsets.mjs @@ -88,10 +88,40 @@ export default { test('conversion replaces stale offsets even when the custom hook writes no index', async t => { const files = await fixture(t) const indexFile = path.join(files.dir, 'index.mjs') - await fs.writeFile(indexFile, 'export default { create(data) { data.items[0].name = "changed" } }\n') + await fs.writeFile(indexFile, 'export default Object.freeze({ create(data) { data.items[0].name = "changed" } })\n') await fs.writeFile(path.join(files.dir, 'index.offset.json'), '{"0":[0,1],"99":[2,3]}') await convert(files, indexFile) const {payloads, data} = await verifyOffsets(files) assert.equal(payloads.length, 3) assert.equal(data.items[0].name, 'changed') }) + +test('conversion awaits custom finalization with the original receiver and final bytes', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, `import index from ${JSON.stringify(baseIndex)} +import fs from 'node:fs/promises' +import {setTimeout} from 'node:timers/promises' +export default { + marker: 'custom receiver', + create(data) { data.items[0].name = 'final value' }, + async finalize(serialized, meta, uuid) { + await setTimeout(20) + if (uuid !== null || typeof serialized !== 'string' || !serialized.includes('final value')) throw new Error('wrong finalizer arguments') + await index.finalize(serialized, meta, uuid) + await fs.writeFile(meta.data + '/finalized.json', JSON.stringify({marker: this.marker, uuid})) + } +} +`) + await convert(files, indexFile) + await verifyOffsets(files) + assert.deepEqual(JSON.parse(await fs.readFile(path.join(files.dir, 'finalized.json'), 'utf8')), {marker: 'custom receiver', uuid: null}) +}) + +test('conversion propagates rejection from custom finalization', async t => { + const files = await fixture(t) + const indexFile = path.join(files.dir, 'index.mjs') + await fs.writeFile(indexFile, 'export default {create() {}, async finalize() { throw new Error("custom finalization failed") }}\n') + await assert.rejects(convert(files, indexFile), /custom finalization failed/) + await assert.rejects(fs.access(path.join(files.dir, 'index.offset.json')), /ENOENT/) +}) From 78fc5147486638e4f1c24efc19ff226e56a0fd83 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Thu, 17 Sep 2026 17:11:29 +0200 Subject: [PATCH 13/14] Record custom index finalization compatibility evidence --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 11 +++-- .spiral/evidence/EVD-20260917-TTZ7C-16.md | 55 ++++++++++++++++++++++ .spiral/evidence/EVD-20260917-TTZ7C-16.ttl | 29 ++++++++++++ 3 files changed, 92 insertions(+), 3 deletions(-) create mode 100644 .spiral/evidence/EVD-20260917-TTZ7C-16.md create mode 100644 .spiral/evidence/EVD-20260917-TTZ7C-16.ttl diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md index 2e70ff9..5b27e19 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.md +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -57,6 +57,11 @@ and commands now use the final-byte scanner/writer. The 181,724-record temporary dataset check still passes, taking about 0.23 seconds including verification with the shared scanner. EVD-20260917-TTZ7C-12 records the current evidence. -The offset correction is verified. The newly authorized index-module dispatch -refinement is in progress. Human acceptance remains pending; no merge, package -release, or downstream-file repair was performed. +The index-module dispatch refinement is implemented and verified: all 59 tests +pass, with asynchronous custom finalizers, failure propagation, and automatic +default fallback covered. A disposable fixture using byte-identical downstream +index sources and shared dependencies passed conversion and command offset +checks along with parent/root/niveau checks. EVD-20260917-TTZ7C-16 records the +evidence and the cross-checkout dependency limitation found by the first probe. +Human acceptance remains pending; no merge, package release, downstream +installation, or downstream-file repair was performed. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-16.md b/.spiral/evidence/EVD-20260917-TTZ7C-16.md new file mode 100644 index 0000000..63c90f2 --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-16.md @@ -0,0 +1,55 @@ +# Evidence: Configured Index Finalization + +Verified on 2026-09-17 with Node v24.7.0. + +## Regression And Lifecycle Checks + +Before implementation, all five focused custom-finalization tests failed: +conversion and commands ignored the configured finalizer, including rejection. +After implementation all 59 tests passed: + +- `npm test`: 33 startup, conversion, and command-offset tests. +- `npm run test:handlers`: 5 tests, including custom finalizer rejection keeping + the prior committed state visible and independently reconstructable. +- `npm run test:acid`: 5 tests. +- `npm run test:durability`: 16 crash/recovery, timeout, retry, and integrity tests. + +Focused tests verify asynchronous completion, receiver and argument preservation, +explicit delegation to default offsets, replacement without an unconditional +default write, rejection propagation, and fallback for frozen legacy wrappers. +An additional direct dispatcher probe verified an inherited finalizer on a +frozen object, preserving its receiver, default UUID, and returned promise value. + +Scoped ESLint on `src/index.mjs`, conversion and command offset tests, and handler +tests passed. Staged provenance and whitespace checks passed before committing +the implementation. Process/socket suites used sandbox escalation. + +## Curriculum-Store Compatibility + +A disposable miniature curriculum graph exercised the actual downstream +`index.mjs`, parent, root, niveau, and utility source files copied byte-for-byte +from `curriculum-restapi-dev/curriculum-store/src`. The fixture linked SimplyStore +to this checkout and shared its JSONTag/OD-JSONTag dependencies. No source edits +were made to the copied wrapper, which still has no `finalize` method. + +Conversion and a command that lengthened a child title both produced offsets +matching an independent byte-framing oracle. Conversion preserved the expected +parent, root, and niveau links; replaying the command changeset preserved the +new title and niveau index. Temporary data was removed afterward. + +The initial direct cross-checkout probe failed before finalization because the +installed old SimplyStore and this checkout resolved distinct OD-JSONTag symbol +instances: the old offset writer could not see the parsed entity's position. +The shared-dependency fixture avoids that test-environment mismatch and verifies +unchanged wrapper source against the fixed dependency; it does not prove that +mixing arbitrary installed dependency copies works. Installed downstream +dependencies and data were not changed. + +## Limits + +This refines dispatch only; the previously verified final-byte scanner remains +unchanged. Custom finalizers own their output and must delegate to retain default +offset files. Read-only finalization inputs are a documented contract, not an +enforced sandbox. Failure can leave uncommitted files, not a partial successful +command. Existing stale sidecars are not repaired automatically. No release, +downstream installation, merge, or human acceptance is claimed. diff --git a/.spiral/evidence/EVD-20260917-TTZ7C-16.ttl b/.spiral/evidence/EVD-20260917-TTZ7C-16.ttl new file mode 100644 index 0000000..e17fbdc --- /dev/null +++ b/.spiral/evidence/EVD-20260917-TTZ7C-16.ttl @@ -0,0 +1,29 @@ +@prefix sd: . +@prefix dcterms: . +@prefix project: . + +project:EVD-20260917-TTZ7C-16 + a sd:VerificationEvidence ; + dcterms:identifier "EVD-20260917-TTZ7C-16" ; + sd:repositoryPath ".spiral/evidence/EVD-20260917-TTZ7C-16.md" ; + sd:status sd:Active ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-15 ; + sd:gitCommit "6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-7 ; + sd:gitCommit "6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:IMP-20260917-TTZ7C-11 ; + sd:gitCommit "6efdf07f48c874ae3a6fea4d6ddfc87f568f4e9f" + ] ; + sd:verifies [ + a sd:ArtifactReference ; + sd:artifact project:REQ-20260917-TTZ7C-13 ; + sd:gitCommit "59214a11f730c9657024c2f338513c86b1088dad" + ] . From 975169b6d8a64de31fd00cc0997c299911560b86 Mon Sep 17 00:00:00 2001 From: Auke van Slooten Date: Fri, 18 Sep 2026 14:24:21 +0200 Subject: [PATCH 14/14] Record acceptance of persisted offset index cycle --- .spiral/cycles/CYC-20260917-TTZ7C-4.md | 17 +++++++++++++++-- .spiral/cycles/CYC-20260917-TTZ7C-4.ttl | 2 +- .spiral/project-context.md | 2 +- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.md b/.spiral/cycles/CYC-20260917-TTZ7C-4.md index 5b27e19..4e2e270 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.md +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.md @@ -63,5 +63,18 @@ default fallback covered. A disposable fixture using byte-identical downstream index sources and shared dependencies passed conversion and command offset checks along with parent/root/niveau checks. EVD-20260917-TTZ7C-16 records the evidence and the cross-checkout dependency limitation found by the first probe. -Human acceptance remains pending; no merge, package release, downstream -installation, or downstream-file repair was performed. +No merge, package release, downstream installation, or downstream-file repair +was performed in this cycle. + +## Human Acceptance + +Accepted by the maintainer on 2026-09-18: "And accept the current cycle for +simplystore." Acceptance covers final-byte offset generation for conversion +and command changesets, including optional custom finalization with the default +fallback, as evaluated above and recorded in EVD-20260917-TTZ7C-16. + +The cycle branch was rechecked as +`spiral/CYC-20260917-TTZ7C-4-conversion-offsets` when recording acceptance. +The maintainer separately requested explicit delegation in curriculum-store +and chose to retain its published SimplyStore dependency. That downstream +change is outside this cycle; using the new hook there awaits a package release. diff --git a/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl b/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl index 79daed7..8e8596e 100644 --- a/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl +++ b/.spiral/cycles/CYC-20260917-TTZ7C-4.ttl @@ -6,4 +6,4 @@ project:CYC-20260917-TTZ7C-4 a sd:Cycle ; dcterms:identifier "CYC-20260917-TTZ7C-4" ; sd:repositoryPath ".spiral/cycles/CYC-20260917-TTZ7C-4.md" ; - sd:status sd:Active . + sd:status sd:Accepted . diff --git a/.spiral/project-context.md b/.spiral/project-context.md index 63ded97..9c333d1 100644 --- a/.spiral/project-context.md +++ b/.spiral/project-context.md @@ -18,7 +18,7 @@ Project causal-graph namespace: `https://github.com/simplyedit/simplystore/spira Spiral core source: `.spiral-core/`, git submodule for `https://github.com/muze-labs/spiral-developer.git`, currently checked out at `8d4b2c738a413abd4cccca740ce958f486e5f7af`. -Current active Spiral cycle: none. Latest accepted cycle: `.spiral/cycles/CYC-20260819-09ZEF-5.md` (`Integrity Detection`). The runtime slice was accepted on 2026-08-19; the compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16. Integration must still be validated against the current target. +Current active Spiral cycle: none. Latest accepted cycle: `.spiral/cycles/CYC-20260917-TTZ7C-4.md` (`Persisted Offset Indexes`), explicitly accepted on 2026-09-18. Conversion and command changesets now finalize offsets from their serialized bytes through the configured index module, with a default fallback for existing wrappers. Integration must still be validated against the current target. The earlier integrity runtime slice was accepted on 2026-08-19; its compliance correction in `DEF-20260916-TTZ7C-1`, verified by `EVD-20260916-TTZ7C-3`, was explicitly accepted on 2026-09-16. ## Intake State