Skip to content

Adding a sublibrary to the test data means republishing all of it #4138

Open

Description

@GuySten

The problem

tools/ci/download-xs.sh fetches the HDF5 test data as one tarball. It carries
neutron, photon, thermal and WMP, unpacked about 1.16 GB.

There is no way to add a sublibrary to it except to regenerate and rehost the
whole thing. The cost of adding data is set by the size of the existing library
rather than by the size of what is being added, and it needs whoever controls
the hosting to act before any test that depends on the new data can run in CI.

Photonuclear is the case I have run into. The library is roughly 150 MB, and
there is nowhere to put it short of a 1.3 GB republish, or committing binaries
under tests/.

Proposal

Keep download-xs.sh as the entry point. The developer workflow and
docs/source/devguide/tests.rst do not change. Split what sits behind it into
one artifact per sublibrary, each with its own URL and cache entry, so that
adding one is additive:

artifact contents size
neutron neutron + thermal + WMP ~1.12 GB
photon photoatomic 45 MB
(new) e.g. photonuclear ~150 MB

Thermal and WMP stay with neutron deliberately. Splitting them off saves 3% of
that artifact, which does not justify two more of everything. Thermal
S(alpha,beta) is useless without neutron data. WMP is a fit to the pointwise
resonance cross section rather than an independent dataset, and the transport
switches between the two representations at the edges of the multipole window,
so keeping them in one artifact means they cannot drift apart.

Questions

  1. Is this a direction the project wants, or is the single tarball deliberate?
  2. Who owns where these artifacts live?
  3. Would an additive first step be acceptable, publishing a new sublibrary as
    its own artifact alongside the existing tarball and leaving that tarball
    alone? That decouples this from any decision about the existing data.

Happy to do the script, CI and docs work if there is interest.

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions