Repository navigation
http zarr proxy #2033
Description
Activity
see also https://github.com/manzt/simple-zarr-server (cc: @manzt)
Reacted by Davis BennettThe zerve name is much better than simple-zarr-server :) Happy to chat about upstreaming some of that work. At some point I'd looked into using package entrypoints to register napari-like plugins to map file arg -> zarr python store, which could then use the rest of the serving capabilities.
# pip install my-zarr-mapper-plugin simple-zarr-server path/to/my/custom/format.foo --path=barThat would let me expose the custom stores in:
nicely over HTTP. A function that maps an arbitrary file to zarr python interface (mutable-mapping like thing), could be the entry point, since it's very straightforward to implement REST on top of the multable-mapping thing.
Reacted by Josh MooreAt some point I'd looked into using package entrypoints to register napari-like plugins to map file arg -> zarr python store, which could then use the rest of the serving capabilities
This is an awesome idea!
As for upstreaming, if we have enough developer energy I think we should have 2 instantiations of this tool, in different python packages:
- in
zarr-python: a boring, depends-on-the-python-stdlib version that mostly exists as tool for testing http-backed storage, but can also be hacked on easily/extended byzarr-pythonusers - standalone: a featureful, slick version that people could use in production, and has whatever dependencies are conducive for that.
I think
simple-zarr-serveris closer to the second option. I'd be curious to see how much code it would take to convert it into the first option.- in
I think simple-zarr-server is closer to the second option. I'd be curious to see how much code it would take to convert it into the first option.
simple-zarr-server is pretty minimal. It basically prepares an ASGI app with starlette, and then uses uvicorn to run it.
We could maybe try to define the minimum ASGI builder without deps in zarr-python, and then all the standalone stuff could include tools to actually run/extend that "core" mapper. I've never looked into writing that part from scratch with the standardlib, but I could take a look. Mapping paths to bytes shouldn't requires many bells and whistles I imagine. I chose starlette at the time because it was minimal and performant.
Reacted by Davis BennettI'd looked into using package entrypoints to register napari-like plugins to map file arg -> zarr python store, which could then use the rest of the serving capabilities.
we should totally do this btw
Reacted by Trevor Manz
I often find myself wanting a way to present non-zarr array data as if it was a zarr array, without going through the effort (and data duplication) required to convert the array to zarr. This often comes up when working with unchunked legacy file formats.
A simple way to do this would be via an HTTP server that converts incoming requests for chunk files into whatever operations are necessary to retrieve chunk data from the legacy format. In the simplest formulation, the server would need to be configured with a declaration of the data being "zarred", and runtime options like compression etc.
If the program were called
zerve(for "zarr serve"), then invocation might look like this:I copied the terminal output from the nodejs program
serve, that I often use for statically hosting zarr arrays. One can imagine more elaborate JSON configuration for the server that would declare how to embed the array (or multiple arrays) in a virtual group.The particular use case I described (wrapping a legacy format in a zarr API) is a specific instance of a more general pattern (wrapping X in a zarr API, where X is some arbitrary computation that produces an array). If designed in a modular way, the http proxy i'm proposing would support this broader usage pattern. And we can also imagine this functionality being used as a python library, a la xpublish. In fact, I think xpublish basically does what I want here, so maybe the only work is to pull the zarr proxying out of xpublish?
In terms of where this should live in
zarr-python, i think this functionality would be very useful for testing http-based storage, so on those grounds I think it's in-scope forzarr-python, but not a pressing need.cc @jhamman