Skip to content

add tooling to access and compute on large embeddings #9

@kevinyamauchi

Description

@kevinyamauchi

In order to support larger-than-memory embeddings, we need some tooling for performant lazy loading. A few features/characteristics:

  • should support most numpy style indexing
  • should allow for simple queries (e.g., ball point)
  • must support zarr both local and remote (e.g., S3)

Libraries for IO:

  • TensorStore. this seems like it's the most flexible backend-wise
  • xarray
  • dask (@kephale noted some performace limitations with large numbers of chunks in the napari tiled rendering work)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions