Skip to content

Commit bcd57d2

Browse files
feat: Milestone 2 — Structured Run Manifest System (#67)
* gsoc26: Layer 2 with tests initial commit * gsoc26: Refactor Layer 2 with handler architecture and improved tests * Improvements to Format layer implementation * Review comments resolved under issue #59 * #61: replacing --convert_from & --convert_to with --compression * gsoc26: layer2 complete + implementation for #61 * gsoc2026: mapping layer initial commit * fix: errors from testing fixed. * docs: update README for --format and --compression flags * fix: resolve PR review comments. * fix: remove duplicate import after merging layer2 fixes into layer3 * docs: add Layer 3 mapping conversion flags and examples * Initial commit: Manifest System * fix: move all test data to tests/resources, add round trip IR comparisons * fix: align Quad->Triple->Quad round trip test as required. * fix: fixed review comments * fix: fixed review comments(2) * Complete Implementation of Milestone 2 * edge case handled * docs: add Manifest section to README documenting --manifest flag * feat: capture operation-level errors in manifest via dbus:operationError * fix: resolve PR review comments * fix: record manifest success, capture broad exceptions, gitignore fixes * fix: minor adjustments
1 parent 1fe4817 commit bcd57d2

10 files changed

Lines changed: 1013 additions & 49 deletions

File tree

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,12 @@ share/python-wheels/
3535
*.egg
3636
MANIFEST
3737

38+
# Windows-specific: MANIFEST above matches databusclient/manifest/ case-insensitively
39+
# on Windows, preventing the manifest module from being committed.
40+
!databusclient/manifest/
41+
!databusclient/manifest/**
42+
databusclient/manifest/__pycache__/
43+
3844
# PyInstaller
3945
# Usually these files are written by a python script from a template
4046
# before PyInstaller builds the exe, so as to inject date/other infos into it.

‎README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Command-line and Python client for downloading and deploying datasets on DBpedia
1818
- [Download](#cli-download)
1919
- [Deploy](#cli-deploy)
2020
- [Delete](#cli-delete)
21+
- [Manifest](#cli-manifest)
2122
- [Module Usage](#module-usage)
2223
- [Deploy](#module-deploy)
2324
- [Development & Contributing](#development--contributing)
@@ -556,6 +557,55 @@ databusclient delete https://databus.dbpedia.org/dbpedia/collections/dbpedia-sna
556557
docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --databus-key YOUR_API_KEY
557558
```
558559
560+
<a id="cli-manifest"></a>
561+
### Manifest
562+
563+
All three commands support an optional `--manifest` flag that writes a structured JSON-LD record of the operation to disk:
564+
565+
**Download**
566+
```bash
567+
# Python
568+
databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --manifest ./manifests/download-run.jsonld
569+
# Docker
570+
docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --manifest ./manifests/download-run.jsonld
571+
```
572+
573+
**Deploy**
574+
```bash
575+
# Python
576+
databusclient deploy \
577+
--version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \
578+
--title "Client Testing" --abstract "Testing the client...." \
579+
--description "Testing the client...." \
580+
--license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \
581+
--apikey YOUR_KEY --manifest ./manifests/deploy-run.jsonld \
582+
'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger'
583+
# Docker
584+
docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy \
585+
--version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \
586+
--title "Client Testing" --abstract "Testing the client...." \
587+
--description "Testing the client...." \
588+
--license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \
589+
--apikey YOUR_KEY --manifest ./manifests/deploy-run.jsonld \
590+
'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger'
591+
```
592+
**Delete**
593+
```bash
594+
# Python
595+
databusclient delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY --manifest ./manifests/delete-run.jsonld
596+
# Docker
597+
docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY --manifest ./manifests/delete-run.jsonld
598+
```
599+
600+
The manifest records input parameters, per-file URLs, checksums, byte sizes, timestamps, and success/failure status for each file. It uses the DataID vocabulary and is versioned via `dbus:schemaVersion`.
601+
602+
- If the target path already exists, the manifest is written to an auto-suffixed path (e.g. `run_1.jsonld`) with a warning.
603+
- Sensitive fields (API keys, vault tokens) are never written.
604+
- If manifest writing fails, a warning is printed and the exit code reflects the actual operation result.
605+
- If the operation itself fails, a `dbus:operationError` block is recorded in the manifest capturing the error type, message, and traceback.
606+
607+
Refer [examples/reproducible-download.md](examples/reproducible-download.md) for a full walkthrough.
608+
559609
## Module Usage
560610
561611
<a id="module-deploy"></a>

‎databusclient/api/delete.py‎

Lines changed: 25 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,16 @@ class DeleteQueue:
2323
Allows adding multiple databus URIs to a queue and executing their deletion in batch.
2424
"""
2525

26-
def __init__(self, databus_key: str):
26+
def __init__(self, databus_key: str, manifest_context=None):
2727
"""Create a DeleteQueue bound to a given Databus API key.
2828
2929
Args:
3030
databus_key: API key used to authenticate deletion requests.
31+
manifest_context: Optional ManifestContext to record deletion
32+
outcomes into. Passed through to _delete_list on execute().
3133
"""
3234
self.databus_key = databus_key
35+
self.manifest_context = manifest_context
3336
self.queue: set[str] = set()
3437

3538
def add_uri(self, databusURI: str):
@@ -69,11 +72,13 @@ def execute(self):
6972
"""Execute all queued deletions.
7073
7174
Each queued URI will be deleted using `_delete_resource`.
75+
Passes manifest_context through so deletions are recorded.
7276
"""
7377
_delete_list(
7478
list(self.sorted_queue()),
7579
self.databus_key,
7680
force=True,
81+
manifest_context=self.manifest_context,
7782
)
7883

7984

@@ -116,6 +121,7 @@ def _delete_resource(
116121
dry_run: bool = False,
117122
force: bool = False,
118123
queue: DeleteQueue = None,
124+
manifest_context=None,
119125
):
120126
"""Delete a single Databus resource (version, artifact, group).
121127
@@ -144,6 +150,8 @@ def _delete_resource(
144150

145151
if dry_run:
146152
print(f"[DRY RUN] Would delete: {databusURI}")
153+
if manifest_context is not None:
154+
manifest_context.record_file(url=databusURI, status="dry_run")
147155
return
148156

149157
if queue is not None:
@@ -156,6 +164,8 @@ def _delete_resource(
156164

157165
if response.status_code in (200, 204):
158166
print(f"Successfully deleted: {databusURI}")
167+
if manifest_context is not None:
168+
manifest_context.record_file(url=databusURI, status="success")
159169
else:
160170
raise Exception(
161171
f"Failed to delete {databusURI}: {response.status_code} - {response.text}"
@@ -168,6 +178,7 @@ def _delete_list(
168178
dry_run: bool = False,
169179
force: bool = False,
170180
queue: DeleteQueue = None,
181+
manifest_context=None,
171182
):
172183
"""Delete a list of Databus resources.
173184
@@ -180,7 +191,7 @@ def _delete_list(
180191
"""
181192
for databusURI in databusURIs:
182193
_delete_resource(
183-
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue
194+
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
184195
)
185196

186197

@@ -190,6 +201,7 @@ def _delete_artifact(
190201
dry_run: bool = False,
191202
force: bool = False,
192203
queue: DeleteQueue = None,
204+
manifest_context=None,
193205
):
194206
"""Delete an artifact and all its versions.
195207
@@ -223,11 +235,11 @@ def _delete_artifact(
223235
else:
224236
# Delete all versions
225237
_delete_list(
226-
version_uris, databus_key, dry_run=dry_run, force=force, queue=queue
238+
version_uris, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
227239
)
228240

229241
# Finally, delete the artifact itself
230-
_delete_resource(databusURI, databus_key, dry_run=dry_run, force=force, queue=queue)
242+
_delete_resource(databusURI, databus_key, dry_run=dry_run, force=force, queue=queue,manifest_context=manifest_context)
231243

232244

233245
def _delete_group(
@@ -236,6 +248,7 @@ def _delete_group(
236248
dry_run: bool = False,
237249
force: bool = False,
238250
queue: DeleteQueue = None,
251+
manifest_context=None,
239252
):
240253
"""Delete a group and all its artifacts and versions.
241254
@@ -266,14 +279,14 @@ def _delete_group(
266279
# Delete all artifacts (which deletes their versions)
267280
for artifact_uri in artifact_uris:
268281
_delete_artifact(
269-
artifact_uri, databus_key, dry_run=dry_run, force=force, queue=queue
282+
artifact_uri, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
270283
)
271284

272285
# Finally, delete the group itself
273-
_delete_resource(databusURI, databus_key, dry_run=dry_run, force=force, queue=queue)
286+
_delete_resource(databusURI, databus_key, dry_run=dry_run, force=force, queue=queue,manifest_context=manifest_context)
274287

275288

276-
def delete(databusURIs: List[str], databus_key: str, dry_run: bool, force: bool):
289+
def delete(databusURIs: List[str], databus_key: str, dry_run: bool, force: bool, manifest_context=None):
277290
"""Delete a dataset from the databus.
278291
279292
Delete a group, artifact, or version identified by the given databus URI.
@@ -286,7 +299,7 @@ def delete(databusURIs: List[str], databus_key: str, dry_run: bool, force: bool)
286299
force: If True, skip confirmation prompt and proceed with deletion.
287300
"""
288301

289-
queue = DeleteQueue(databus_key)
302+
queue = DeleteQueue(databus_key, manifest_context=manifest_context)
290303

291304
for databusURI in databusURIs:
292305
_host, _account, group, artifact, version, file = (
@@ -296,24 +309,24 @@ def delete(databusURIs: List[str], databus_key: str, dry_run: bool, force: bool)
296309
if group == "collections" and artifact is not None:
297310
print(f"Deleting collection: {databusURI}")
298311
_delete_resource(
299-
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue
312+
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
300313
)
301314
elif file is not None:
302315
print(f"Deleting file is not supported via API: {databusURI}")
303316
elif version is not None:
304317
print(f"Deleting version: {databusURI}")
305318
_delete_resource(
306-
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue
319+
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
307320
)
308321
elif artifact is not None:
309322
print(f"Deleting artifact and all its versions: {databusURI}")
310323
_delete_artifact(
311-
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue
324+
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
312325
)
313326
elif group is not None and group != "collections":
314327
print(f"Deleting group and all its artifacts and versions: {databusURI}")
315328
_delete_group(
316-
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue
329+
databusURI, databus_key, dry_run=dry_run, force=force, queue=queue, manifest_context=manifest_context
317330
)
318331
else:
319332
print(f"Deleting {databusURI} is not supported.")

0 commit comments

Comments
 (0)