Skip to content

ENH: Build, test, and wrap external modules against an installed ITK - #6891

Open
hjmjohnson wants to merge 5 commits into
InsightSoftwareConsortium:mainfrom
hjmjohnson:wrapping-install-optin
Open

hjmjohnson wants to merge 5 commits into
InsightSoftwareConsortium:mainfrom
hjmjohnson:wrapping-install-optin

Conversation

@hjmjohnson

@hjmjohnson hjmjohnson commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Let an external (remote) module be configured, built, and Python-wrapped against an installed ITK, not only a build tree. Opt-in via ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES (default OFF, tens of MB). Verified by building and wrapping 16 remote modules against an installed ITK 6 tree, and by configuring, building, and running module test suites (BUILD_TESTING=ON) against it; every fix the modules themselves needed is already in review upstream.

Why this matters. Today a remote module can only be wrapped from an ITK build tree. That is the reason the ITK Python wheel cache (ITKPythonBuilds) ships whole build trees, and it is why nobody can wrap a module against the ITK a distribution or a cmake --install provides. With this PR an installed prefix is a complete wrapping SDK, which is the precondition for a smaller wheel cache and for remote-module wheels built from an installed ITK.

Remote-module PRs needed for the verification below (GitHub shows each one's merge state):

None of them depends on this PR, and this PR does not depend on them: they are ITK 6 API updates and wrap files that never worked, needed to build the modules at all.

What was broken (five commits)
  1. The module build files are not installed. An external module starts with include(ITKModuleExternal); that file and its transitive includes were absent from the install tree, so configuring a module against an installed ITK failed at the first line. Installs the closure of that entry point, plus BuildHeaderTest.py, which ITKModuleHeaderTest.cmake runs by a path that only resolves in a source tree, and GetGitRevisionDescription.cmake and itkVersion.cmake, which RTK and CudaCommon include through ITK_CMAKE_DIR (about 125 KB).
  2. The wrapping infrastructure and SWIG type indices exist only in the build tree. ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES=ON installs both under the package directory, where ITK_CMAKE_DIR and ITK_DIR already point for an installed consumer; the generated indices carry no absolute paths and relocate as-is. Also installed there: each wrapped module's SWIG templates (Wrapping/Modules/<Module>/, which RTK extends from ITKBridgeNumPy) and the vendored pygccxml that igenerator.py imports from a source-tree path. Also repairs ITKModuleExternal.cmake's install-tree branch, which read EXISTS"${ITK_CMAKE_DIR}/..." with no space and therefore never evaluated true.
  3. castxml could not find ITK's headers. From a build tree the dependency targets supply ITK's include directories; from an install tree they supply nothing, so castxml failed with 'itkCommand.h' file not found on an installed header. Adds ITK_INCLUDE_DIRS, which ITKConfig.cmake sets in both trees. Also adds Python3_INCLUDE_DIRS, since ITKBridgeNumPy exports Python's include directory for the build tree only. It is not empty in a build tree, where it accumulates the include directories of every module find_package requested, so the castxml response file widens there too.
  4. The wrapping library inherits UseITK's IO factory register manager. UseITK adds ITK_<FACTORY>_FACTORY_REGISTER_MANAGER to every target in a consumer tree. A wrapped class that includes itkImageFileReader.h then references the register function of every IO factory the installed ITK enables, and the wrapping library, which links only its module's dependencies, fails to load (25 unresolved symbols in LesionSizingToolkit). Python registers factories at import (itk/support/base.py), so ITKModuleExternal.cmake sets an ITK-private ITK_WRAPPING_NO_FACTORY_REGISTER_MANAGER before adding the wrapping directory, and itk_wrap_module removes the definitions there with remove_definitions. Only the wrapping directory is affected: a module's C++ library, tests, and examples keep automatic factory registration (verified by symbol inspection of a module example executable, which still carries all 25 register references and passes its IO tests).
  5. Module tests cannot be configured or built from an install tree. With BUILD_TESTING=ON, ITK_USE_KWSTYLE defaults on and, with no KWStyle found, the module includes Utilities/KWStyle/BuildKWStyle.cmake to build one, which exists only in a source tree, so configure fails. The <Module>InDoxygenGroup test ran Utilities/Doxygen/mcdoc.py by a source-relative path. The header test and the test driver both pick up the register manager from item 4 and linked too few IO libraries (the test driver's ITKTestKernel covers the default IO modules, not the ones a maximal install enables). KWStyle is built only when the source tree provides it, mcdoc.py is installed and resolved like BuildHeaderTest.py, and both test executables link ITK_LIBRARIES for external modules.
Verification: 16 remote modules against an installed ITK 6

ITK at upstream main plus this branch (re-verified 2026-09-23 at the current head), configured with ITK_WRAP_PYTHON=ON, ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES=ON, and every EXCLUDE_FROM_DEFAULT module enabled that builds (57 modules ON, 0 failed targets), then ninja install. Each remote module was configured with ITK_DIR=<prefix>/lib/cmake/ITK-6.0, ITK_WRAP_PYTHON=ON, built, and its Python module built; LesionSizingToolkit was additionally installed and exercised from Python.

Result Modules
Pass unchanged VkFFTBackend, Cleaver, BioCell, SkullStrip, SimpleITKFilters, HASI
Pass at current upstream Shape (SlicerSALT main), TractographyTRX (tee-ar-ex/ITKTractographyTRX#30)
Pass with ITK 6 fixes now in review the eight modules listed below
Out of scope Impact (needs LibTorch)

With BUILD_TESTING=ON against the same prefix, four modules configure, build their header test and test driver, and run: Dissolve 2/2, LesionSizingToolkit 42/42, RingArtifact 1/2, Ultrasound 37/48. All four resolve the header-test script to <prefix>/lib/cmake/ITK-6.0/BuildHeaderTest.py, the install-tree branch. The remaining failures are module-side and tracked on the module PRs: RingArtifact's baseline is FFT-backend dependent, Ultrasound has one truncated data object and ten Python tests blocked by the PYTHONPATH limitation below. Running any of them needs #6894 or DYLD_LIBRARY_PATH.

Companion ITK PRs:

None of the module-side fixes is specific to an installed ITK: they are ITK 6 API changes (ITK_DISALLOW_COPY_AND_MOVE, override, the VNL FFT removal), wrap files that never worked, and examples that only compiled inside an ITK source tree. ITK's own build never includes UseITK.cmake, so in-tree wrapping is unchanged. Two things do change for an external module built against a build tree: its castxml response file gains ITK_INCLUDE_DIRS (item 3), and its wrapping directory no longer carries the IO factory register managers (item 4).

Known limitations, deliberately left for follow-up
  • Generated wrapping artifacts and an external module's executables are written into the prefix (<prefix>/lib/cmake/ITK-6.0/Wrapping/...) because the output directories come from the installed wrapping CMake. Modules therefore must be built serially against one prefix, and a read-only prefix will not work yet. Fixing this means routing the output directories through the consumer's binary dir; it is independent of the five changes here.
  • The installed itkTestDriver carries no install RPATH (ITK sets none), so on macOS it cannot load @rpath/libITKTestKernel from <prefix>/lib without DYLD_LIBRARY_PATH; every module test that runs through ITK_TEST_DRIVER aborts until it is set. Pre-existing and independent of this PR; fixed by BUG: Give the installed itkTestDriver an RPATH to ITK's libraries #6894.
  • Python tests of an external module (itk_python_add_test) do not run from an install tree: ITKModuleTest.cmake puts ${ITK_DIR}/Wrapping/Generators/Python first on PYTHONPATH, which in an install tree is the installed wrapping infrastructure (a partial itk package without ITK's own module configurations), not ITK's Python package. Same root cause as the output-directory limitation above, and the same follow-up.

@github-actions github-actions Bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Enhancement Improvement of existing methods or implementation labels Sep 22, 2026
@hjmjohnson
hjmjohnson force-pushed the wrapping-install-optin branch from 38a9615 to 6cb9163 Compare September 22, 2026 17:24
@github-actions github-actions Bot added the area:Documentation Issues affecting the Documentation module label Sep 22, 2026
@hjmjohnson
hjmjohnson force-pushed the wrapping-install-optin branch from 6cb9163 to bcf8398 Compare September 22, 2026 17:33
@dzenanz

dzenanz commented Sep 22, 2026

Copy link
Copy Markdown
Member

same ITK_WRAP_* type options

Doesn't this get taken by the remote module from the ITK_DIR it is pointed at?

@dzenanz dzenanz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for pushing this Hans.

Comment thread Documentation/docs/contributing/module_workflows.md Outdated
Comment thread CMake/ITKModuleExternal.cmake Outdated
Comment thread CMake/ITKModuleKWStyleTest.cmake Outdated
@hjmjohnson

Copy link
Copy Markdown
Member Author

Two force-pushes just now, on purpose: 72b3385da07..0e529fa590b is review fixes only (same base, so the compare link shows only the patch changes), and 0e529fa590b..5346910ab27 is a plain rebase onto main with no content change.

@blowekamp

Copy link
Copy Markdown
Member

TANGENTAL COMMENT: Keeping ITK wrapping part of the ITK build process is useful for development and maintanance. However, some packaging seniors being able to build ITK Wrapping against an installed ITK is useful. For example in the condo-forge infrastructire. IMHO it would be a good feature to add to the wrapping directory.

@hjmjohnson
hjmjohnson marked this pull request as ready for review September 23, 2026 15:55
@hjmjohnson

Copy link
Copy Markdown
Member Author

TANGENTAL COMMENT: Keeping ITK wrapping part of the ITK build process is useful for development and maintanance. However, some packaging seniors being able to build ITK Wrapping against an installed ITK is useful. For example in the condo-forge infrastructire. IMHO it would be a good feature to add to the wrapping directory.

The conda-forge infrastructure for separate packaging was the motivation for this as a first step. There is followup work that would be needed for completing this work, and hopefully we will find funding to support that larger effort :).

@greptile-apps

greptile-apps Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

Safe to merge; no blocking issues remain.

Summary

This draft makes installed ITK packages usable by external modules for configuration, testing, and optional Python wrapping development. It installs the required CMake and wrapping support files, resolves build-tree versus install-tree paths, and documents installed-package consumption.

Reviews (2) · Last reviewed commit: "COMP: Build external module tests agains..."

Comment thread Wrapping/CMakeLists.txt Outdated
@greptile-apps

This comment has been minimized.

@hjmjohnson
hjmjohnson force-pushed the wrapping-install-optin branch 2 times, most recently from 4a20b1c to 79cb0cc Compare September 23, 2026 17:35
@hjmjohnson

Copy link
Copy Markdown
Member Author

Two force-pushes again, one concern each: 5346910ab27..4a20b1c6662 is a plain rebase onto main (all five commits content-identical), and 4a20b1c6662..79cb0cc5d77 condenses the five multi-line in-source comments this PR added to one line each, per Documentation/AI/prose-budget.md.

@hjmjohnson

Copy link
Copy Markdown
Member Author

@greptileai review this draft before I make it official

@hjmjohnson

Copy link
Copy Markdown
Member Author

ARMBUILD-x86_64-rosetta failed on an ExternalData fetch, not on code: the single build error CDash counted is CMake Error at CMake/ExternalData.cmake:1169, which aborted the build before the test drivers linked, hence the 2414 "Not Run" tests. The Linux build of the same tree reported 0 errors and 0 test failures, and this job passed on the previous head, whose only difference is in-source comment text. Re-running the job.

@hjmjohnson hjmjohnson changed the title ENH: Wrap external modules against an installed ITK ENH: Build, test, and wrap external modules against an installed ITK Sep 23, 2026
@dzenanz dzenanz mentioned this pull request Sep 29, 2026
@SimonRit

Copy link
Copy Markdown

@axel-grc Can you test this PR for CudaCommon and RTK please? We have implemented quite a few subtle things in their wrappings worth testing before this is merged in my opinion.

@axel-grc

Copy link
Copy Markdown

I tested it and it seems to not work as is.

Setup: ITK at PR head 79cb0cc, -DITK_INSTALL_WRAPPING_DEVELOPMENT_FILES=ON, -DITK_WRAP_PYTHON=ON (F/D, dims 2/3/4), installed to a prefix. RTK configured/built/installed as an external module against the installed ITK_DIR.

Blockers found:

  1. GetGitRevisionDescription.cmake is not installed. RTK's CMakeLists.txt does list(APPEND CMAKE_MODULE_PATH ${ITK_CMAKE_DIR}) then include(GetGitRevisionDescription) + get_git_head_revision() (git hash in the version string) → configure fails with Unknown CMake command "get_git_head_revision" before anything else runs.
  2. Modules/Bridge/NumPy/wrapping/PyBuffer.i.{init,in} are not installed. RTK's wrapping/itkPyBufferRTK.wrap reads them from ${ITKBridgeNumPy_INCLUDE_DIR}/../wrapping → in an install tree that's /include/wrapping/, which doesn't exist.
  3. Python headers are not passed to castxml for external modules → wrapping itkPyBuffer.h (includes <Python.h>) fails with Python.h file not found. Source builds get this from include_directories("${Python3_INCLUDE_DIRS}") in Wrapping/Generators/Python/CMakeLists.txt; the install tree has no equivalent.

@hjmjohnson
hjmjohnson force-pushed the wrapping-install-optin branch from 79cb0cc to 5e059e1 Compare September 30, 2026 20:14
An external module starts with include(ITKModuleExternal), but that file
and the ones it reaches are not installed, so configuring a module against
an installed ITK fails immediately.

Adds the transitive closure of that entry point over ITK-owned CMake files,
the templates they configure, and BuildHeaderTest.py, which the header test
runs by a path that only resolves in a source tree. About 115 KB.
The wrapping infrastructure and the generated SWIG type indices exist only
in a build tree, so an installed ITK cannot wrap an external module.

ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES=ON installs both under the package
directory, where ITK_CMAKE_DIR and ITK_DIR already point for an installed
consumer. It defaults to OFF because the indices are tens of megabytes.

Also repairs the install-tree branch in ITKModuleExternal.cmake: it read
EXISTS"${ITK_CMAKE_DIR}/..." with no space, which CMake evaluates as false
even when the file is present, so the branch was unreachable.
Wrapping an external module against an installed ITK failed in castxml with
"'itkCommand.h' file not found" although the header was installed: the
generated response file carried only the module's own include directories.
In a build tree the dependency targets supply ITK's; from an install tree
they contribute nothing.

ITKConfig.cmake sets ITK_INCLUDE_DIRS in both trees, so it covers the
installed case. It is not empty in a build tree: it accumulates the include
directories of every module find_package requested, so the response file
widens there too.
A consumer tree that includes UseITK gets ITK_<FACTORY>_FACTORY_REGISTER_MANAGER
on every target, including an external module's Python wrapping library.
Wrapped classes that include itkImageFileReader.h then reference the
register function of every IO factory the installed ITK enables, and the
wrapping library, which links only its module's dependencies, fails to
load with unresolved symbols.

Python registers the factories at import, so ITKModuleExternal asks the
wrapping directory to drop them and itk_wrap_module removes the ones
ITK_FACTORY_LIST names. The request uses a variable only itk_wrap_module
reads, so a scope added after the wrapping directory, such as a module's
examples, cannot lose its factory registration.

ITK's own build never includes UseITK, so in-tree wrapping is unchanged,
and a module's library, tests, and examples keep their managers.
With BUILD_TESTING=ON, ITK_USE_KWSTYLE defaults on, and when no KWStyle is
found the module build includes Utilities/KWStyle/BuildKWStyle.cmake to
build one, which exists only in a source tree; an external module then
fails to configure against an installed ITK. Build KWStyle only when the
source tree provides it, so an install-tree consumer falls through to the
existing branch that turns ITK_USE_KWSTYLE off.

The <Module>InDoxygenGroup test ran Utilities/Doxygen/mcdoc.py by a
source-relative path. Install the script into the package directory and
resolve it the way BuildHeaderTest.py is resolved.

The header test and the test driver pick up UseITK's IO factory register
manager through itkImageFileReader.h, which references every IO factory the
ITK enables. The header test linked only the module, and the test driver
linked the module's test dependencies and ITKTestKernel, which covers the
default IO modules but not the ones an ITK built with extra IO modules
enables; both failed with unresolved symbols. Link ITK_LIBRARIES for
external modules, as any consumer executable must.
@hjmjohnson
hjmjohnson force-pushed the wrapping-install-optin branch from 5e059e1 to 17cee47 Compare September 30, 2026 20:15
@hjmjohnson

Copy link
Copy Markdown
Member Author

@axel-grc thank you for the precise report. All three blockers are fixed in 79cb0cc..5e059e1 (content only; 5e059e1..17cee47 is a plain rebase onto main), and I verified with CudaCommon and RTK themselves on a CUDA 13.3 / RTX 6000 Ada host.

  1. GetGitRevisionDescription.cmake is installed, and so is itkVersion.cmake, which RTK's applications/CMakeLists.txt includes next.
  2. Every wrapped module's wrapping/*.i, *.i.in, and *.i.init templates are installed under ${WRAP_ITK_CMAKE_DIR}/Modules/<Module>/. An install tree has no <include>/../wrapping, so RTK's itkPyBufferRTK.wrap needs a fallback; the four-line form is in Documentation/docs/contributing/module_workflows.md, and it is what I used for the RTK build below. Happy to open that RTK PR if you prefer.
  3. castxml receives Python3_INCLUDE_DIRS for an external module, because ITKBridgeNumPy exports its Python include directory for the build tree only.
  4. Found along the way: igenerator.py imports the vendored pygccxml from a source-tree path. It is now installed under ${WRAP_ITK_CMAKE_DIR}/ThirdParty/pygccxml.
Verification (cortex, Linux x86-64, conda gcc 14.3, CUDA 13.3, sm_89)

ITK at 5e059e1 with ITK_WRAP_PYTHON=ON, ITK_WRAP_double=ON, ITK_WRAP_IMAGE_DIMS=2;3;4, ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES=ON, installed to a prefix.

Module Against Result
ITKCudaCommon cf8eb2c installed ITK_DIR, ITK_WRAP_PYTHON=ON configure, build (10 castxml inputs), install: all 0
RTK 6eaffcc + wrap-file fallback installed ITK_DIR, RTK_USE_CUDA=ON, ITK_WRAP_PYTHON=ON configure, build (915 steps, 156 castxml inputs, itkPyBufferRTK wrapped), install: all 0

Runtime, from the installed site-packages: itk.CudaImage[itk.F,3].New(), rtk.CudaFDKConeBeamReconstructionFilter.New(), and itk.array_from_image on a 3D image all work.

Same series on macOS arm64 with a minimal module that includes itkPyBuffer.h and reads PyBuffer.i.init: configure, wrap, and link succeed.

CMAKE_CUDA_ARCHITECTURES=89 was needed because CUDA 13 dropped CMake's default sm_52; unrelated to this PR.

One limitation to note, not new to this PR: external-module wrapping writes its generated .idx/.mdx and the wrapped library into ${ITK_DIR}/Wrapping/..., which for an install tree means into the installed prefix. That mirrors the build-tree behavior and works for a writable prefix; a read-only prefix needs a follow-up that gives the index lookup a search path.

@hjmjohnson

Copy link
Copy Markdown
Member Author

The RTK side of item 2 is RTKConsortium/RTK#1001 (draft), and the writable-prefix limitation is now tracked in #6923.

@axel-grc

axel-grc commented Oct 1, 2026

Copy link
Copy Markdown

I tested the fix and all seems to work fine with the RTKConsortium/RTK#1001.
Thanks you !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Documentation Issues affecting the Documentation module type:Enhancement Improvement of existing methods or implementation type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants