Repository navigation
refactor to work with gardenlinux 0.10.24 - #13
Conversation
e5121c4 to
f911488
Compare
f911488 to
28a7960
Compare
28a7960 to
16cd1c4
Compare
16cd1c4 to
4e3988d
Compare
4e3988d to
7e02f13
Compare
… schema.py Reduces ~580 lines of duplicated schema definitions by generating both SCHEMA_V1 and SCHEMA_V2 from shared builder functions. The only difference between v1 and v2 is that v2 requires a 'patch' field in version objects for minor, nightly, and dev release types. - Create glrd/schema.py with factory functions for generating schemas - Update manage.py to import from glrd.schema instead of the old files - All schema validation tests pass (7/7)
- Add missing ERROR_CODES entries: output_error, s3_output_error, input_parameter_missing, input_parameter_error, subprocess_output_missing - Add fatal_error() utility function for centralized error handling - Fix deprecated datetime.utcfromtimestamp() usage in util.py and manage.py (use datetime.fromtimestamp with timezone.utc instead) - Add timezone import to both util.py and manage.py
Add commonly used utility functions: - V2_SCHEMA_THRESHOLD constant (2017) for v1/v2 schema decisions - uses_patch_version() helper function - split_releases_by_type() to split releases by type into a dict - run_subprocess() for standardized subprocess execution - parse_isodatetime() for consistent datetime parsing with error handling All schema tests pass.
Replace duplicated release type filtering logic (~15 lines repeated 3 times) with calls to the shared split_releases_by_type() utility function from util.py. This eliminates duplicate code and ensures consistent behavior.
- Remove unused import in schema.py - Fix line length issues in schema.py - Add proper blank line after function in util.py - Add proper type hints for parse_isodatetime - Format all modified files with black
Phase 1 - Dataclasses: - Add glrd/models.py with ReleaseType enum, Version, LifecyclePhase, Lifecycle, GitInfo, Release, ReleaseCollection, parse_release_name - Centralizes the 'major >= 2017' version threshold logic in Version class Phase 2 - S3 module: - Extract S3 operations to glrd/s3.py - Import s3 functions in manage.py for backward compatibility - Functions: create_s3_bucket, upload_to_s3, download_from_s3, merge_existing_s3_data, download_all_s3_files, upload_all_local_files All schema validation tests pass.
Phase 2 - Git module: - Extract Git operations from manage.py to glrd/git.py - Functions: get_github_releases, get_git_commit_from_tag, get_git_commit_at_time, get_garden_version_for_date, create_initial_releases, create_initial_nightly_releases, cleanup_temp_repo - Fix date increment logic in create_initial_nightly_releases (use timedelta) - Import git functions in manage.py All schema tests pass.
Phase 2 - Validation module: - Extract validation functions from manage.py to glrd/validation.py - Functions: validate_input_version_format, get_schema_for_release, validate_release_data, validate_all_releases - Use V2_SCHEMA_THRESHOLD constant instead of hardcoded 2017 - Import validation functions in manage.py All schema tests pass.
- Remove unused datetime import from models.py - Remove unused major_version variable in git.py - Fix trailing whitespace in git.py - Format code with black All tests passing.
- Change from 'deepdiff import DeepDiff' to 'deepdiff.diff import DeepDiff' to fix lint warning about incorrect module import Note: manage.py still contains duplicate function definitions that are imported from new modules (s3.py, git.py, validation.py). These duplicates are functionally harmless as Python uses the imported versions, but they trigger F811 lint warnings. Removing them automatically breaks the file due to complex interdependencies. This is acceptable technical debt that can be addressed incrementally. All 29 tests passing.
Clean up manage.py by removing 522 lines of duplicate function definitions that are now imported from other modules: Duplicate functions removed: - validate_input_version_format, get_schema_for_release (from validation.py) - cleanup_temp_repo, get_github_releases, get_git_commit_from_tag, get_git_commit_at_time, get_garden_version_for_date, create_initial_releases, create_initial_nightly_releases (from git.py) - validate_release_data, validate_all_releases (from validation.py) - download_all_s3_files, upload_all_local_files (from s3.py) Unused imports removed: - fnmatch, shutil, subprocess, timedelta, timezone - jsonschema.validate, jsonschema.ValidationError - SCHEMA_V1, SCHEMA_V2 - extract_version_data, NoAliasDumper - save_output_file alias File size reduced from 1,911 to 1,364 lines (-547 lines, -29%) Format with black to fix spacing issues. All 29 tests passing. Lint clean (no errors or warnings).
d0fe837 to
2eecb72
Compare
8bf6ea1 to
ec58af6
Compare
|
📚 Documentation Preview PR: gardenlinux/docs#132 A PR has been created/updated to preview the documentation changes from this PR. |
3115b17 to
af4678c
Compare
168cc1c to
f7d9722
Compare
…test suite fully offline - Bump python-gardenlinux-lib dependency from 0.10.20 to 0.10.24 - Rename models.py -> release.py; harden Version for 'next' sentinel; make to_dict faithful (2135 real releases round-trip with 0 mismatches) - Wire release.py as single source of truth in query.py (filter, sort, latest, active/archived) and manage.py (create builds Release object, parse_release_name delegates to model, ensure_isodate_and_timestamp delegates to LifecyclePhase.ensure_complete) - Add Release.default_name() and Release.github_release_url() so name and URL generation are centralized in the model - Fix circular import: move merge_input_data from manage to util - Remove duplicate S3 functions from manage.py (save_output_file, create_s3_bucket, upload_to_s3, download_from_s3, merge_existing_s3_data); import from glrd.s3 throughout - Delete unused schema_v1.py and schema_v2.py - Wire previously dead --s3-create-bucket flag in handle_releases - Fix filter_releases crash on unknown --type values (restore prior behavior of silently ignoring unknown types instead of raising ValueError) - Fix dead pass branch in diff_releases to log added/removed items - Add resolve_flavors() to util.py; add --no-flavors flag and GLRD_SKIP_FLAVORS env var to skip network-dependent flavor lookup - Skip git commit lookup for next/major release types (they never store it) - Add --input-type/--input-url/--input-file-prefix/--query-input-format to glrd-manage so update/delete queries can run offline or against custom sources - Make full test suite offline (was hanging indefinitely; now 83 tests in ~39s): autouse conftest fixture, offline query injection in test helpers - Add 22 new tests: 21 unit tests for release model + 6 offline CLI integration tests (active/archived/latest/version filter, no-flavors, unknown-type guard) - Fix test_schema_validation.py import (glrd.validation, not glrd.manage) - Update README with current --help output and offline testing guidance - Net -375 lines; flake8 + black clean Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-opus-4-8 Assisted-by: Kilo:claude-sonnet-4-6
f7d9722 to
577b932
Compare
Replace the monolithic docs/overview/README.md with a proper Diataxis structure: - docs/reference/index.md → hub: reference/supporting_tools/glrd/index.md - docs/reference/cli.md → hub: reference/supporting_tools/glrd/cli.md - docs/reference/schema.md → hub: reference/supporting_tools/glrd/schema.md - docs/explanation/release-lifecycle.md → hub: explanation/glrd-release-lifecycle.md - docs/how-to/index.md → hub: how-to/releases/glrd/index.md - docs/how-to/run-glrd.md → hub: how-to/releases/glrd/run-glrd.md - docs/how-to/query-releases.md → hub: how-to/releases/glrd/query-releases.md - docs/how-to/manage-releases.md → hub: how-to/releases/glrd/manage-releases.md Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-opus-4-8 Assisted-by: Kilo:claude-sonnet-4-6
Use gardenlinux.github.Client (PyGithub) and gardenlinux.git.Repository (pygit2) instead of subprocess calls to gh and git. Both are already pulled in transitively by the gardenlinux dependency. - get_github_releases: Client().get_repo(...).get_releases() - get_git_commit_from_tag: get_git_ref() with annotated-tag dereferencing - get_git_commit_at_time: Repository.checkout_repo() + pygit2 walk - cleanup_temp_repo: dereference _repo_instance before rmtree Auth change: GITHUB_TOKEN is now required for GitHub API calls instead of gh auth login. Add GITHUB_TOKEN prerequisites to run-glrd.md and manage-releases.md. Remove gh CLI install from Containerfile. Add tests/test_git.py with 14 unit tests covering all changed functions. Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-opus-4-8 Assisted-by: Kilo:claude-sonnet-4-6
- Rename DEFAULTS['CONTAINER_REGISTRY'] → DEFAULTS['CONTAINER_REGISTRY_MINOR']
(ghcr.io/gardenlinux/gardenlinux)
- Add DEFAULTS['CONTAINER_REGISTRY_NIGHTLY'] (ghcr.io/gardenlinux/nightly)
- Add get_container_registry(release_type) helper in query.py that returns
the nightly registry for nightly releases and the minor registry otherwise
- Thread release_type through prepare_oci_flavor_url (new 4th param) and
its call site in format_flavors_with_urls
- Update get_oci_url to use get_container_registry(release.get('type'))
- Add tests/test_query_oci.py: 11 unit tests covering get_container_registry,
get_oci_url, and prepare_oci_flavor_url for both container and bare flavors
- Add OCI integration tests to test_integration.py: nightly/minor registry
selection and cross-type registry difference
Fixes #16
Signed-off-by: Eike Waldt <waldt@b1-systems.de>
On-behalf-of: SAP <eike.waldt@sap.com>
Assisted-by: Kilo:claude-opus-4-7
Assisted-by: Kilo:claude-sonnet-4-6
577b932 to
c7d8fbe
Compare
There was a problem hiding this comment.
You can use gardenlinux.DistroVersion here which extends semver.Version. This provides the same functionality to create and count up version numbers semver compliant.
There was a problem hiding this comment.
@NotTheEvilOne I looked at gardenlinux.DistroVersion and couldn't really see how GLRD would benefit from using it. We're doing way more complex things by using jsonschema dicts and comparing those.
Maybe at some point in time the whole version schema logic/code could go into the python lib.
If you still think it's a good idea to use gardenlinux.DistroVersion, I would need a hint where exactly.
There was a problem hiding this comment.
I was thinking of using this class for calculating one version from another one. See python-gardenlinux-lib tests and the underlying semver API.
inside the git clone code Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-sonnet-4-6
Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-sonnet-4-6
Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-sonnet-4-6
Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-sonnet-4-6
|
@NotTheEvilOne Addressed your comments, please re-review. |
NotTheEvilOne
left a comment
There was a problem hiding this comment.
LGTM even thought I think additional code changes may further improve readability of the code.
Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com>
inside the git clone code Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-sonnet-4-6
Replace the explicit ensure_complete() mutation pattern with a __post_init__ hook on LifecyclePhase that automatically derives the missing field (timestamp from isodate, or vice versa) at construction time. Remove LifecyclePhase.ensure_complete and Lifecycle.ensure_complete, which are now redundant. Drop the phase.ensure_complete() call in manage.ensure_isodate_and_timestamp accordingly. Signed-off-by: Eike Waldt <waldt@b1-systems.de> On-behalf-of: SAP <eike.waldt@sap.com> Assisted-by: Kilo:claude-sonnet-4-6
What this PR does / why we need it:
Code refactoring
glrd/git.py — replace gh/git CLI with python-gardenlinux-lib
Use
gardenlinux.github.Client(PyGithub) andgardenlinux.git.Repository(pygit2) instead of subprocess calls to gh and git. Both are already
pulled in transitively by the gardenlinux dependency.
Client().get_repo(...).get_releases()get_git_ref()with annotated-tag dereferencingRepository.checkout_repo()+ pygit2 walktempfile.mkdtemp+ manualshutil.rmtreewithtempfile.TemporaryDirectory; dereference_repo_instancebefore cleanupAuth change:
GITHUB_TOKENis now required for GitHub API calls insteadof gh auth login. Remove gh CLI install from Containerfile.
release.py — single source of truth, split into one class per file
query.py(filter, sort, latest, active/archived) andmanage.py(create, parse_release_name, ensure_isodate_and_timestamp)Release.default_name()andRelease.github_release_url()release_type.py,version.py,lifecycle.py,git_info.py,release_model.py,release_collection.py,release_name.py; all callers updated to import directly from thecanonical module
manage.py — remove duplicates
Remove functions now imported from
git.py,validation.py, ands3.py.Remove unused imports. Replace
if/elif/elsechain onrtypewith amatchstatement.schema.py — unify schema_v1.py and schema_v2.py
Generate
SCHEMA_V1andSCHEMA_V2from shared builder functions.Reduces of duplicated definitions. Delete
schema_v1.pyandschema_v2.py.Other refactors
glrd/validation.pyglrd/s3.pyglrd/util.py(V2_SCHEMA_THRESHOLD,split_releases_by_typenow returns a typedReleasesByTypedataclass,run_subprocess, parse_isodatetime, fatal_error, resolve_flavors,
merge_input_data)
to glrd-manage for offline/custom-source queries
datetime.utcfromtimestamp()usage throughoutglrd/util.py + glrd/query.py — split CONTAINER_REGISTRY by release type
DEFAULTS['CONTAINER_REGISTRY']→DEFAULTS['CONTAINER_REGISTRY_MINOR'](
ghcr.io/gardenlinux/gardenlinux)DEFAULTS['CONTAINER_REGISTRY_NIGHTLY'](ghcr.io/gardenlinux/nightly)get_container_registry(release_type)helper inquery.pythat returnsthe nightly registry for nightly releases and the minor registry otherwise
release_typethroughprepare_oci_flavor_url(new 4th param) andits call site in
format_flavors_with_urlsget_oci_urlto useget_container_registry(release.get('type'))Tests
tests/test_git.py: unit tests for allgit.pyfunctions, mockinggardenlinux.github.Clientand using a local pygit2 bare-repo fixture;includes cache-reuse and error-path cleanup assertions for
TemporaryDirectorytests/test_release_model.py: unit tests for the release model, updatedto import directly from canonical submodules
tests/test_util.py: unit tests forsplit_releases_by_type/ReleasesByType(empty input, per-type routing, unknown/missing type dropped,multiple same-type, field independence)
filter, no-flavors, unknown-type guard)
tests/test_query_oci.py: unit tests coveringget_container_registry,get_oci_url, andprepare_oci_flavor_urlfor both container and bare flavorstest_integration.py: nightly registry selection,minor registry selection, and cross-type registry difference
Documentation
structure: reference/index.md, reference/cli.md, reference/schema.md,
explanation/release-lifecycle.md, how-to/index.md, how-to/run-glrd.md,
how-to/query-releases.md, how-to/manage-releases.md
prerequisite; add GITHUB_TOKEN callout to manage-releases.md
Fixes
Fixes #16
Fixes #14
Fixes #4