Skip to content

get_dataset_collection should return an iteratable #36

@koenvo

Description

@koenvo

When having lots of datasets (like 100.000+) the get_dataset_collection shouldn't return all at once. This will give memory issues. An approach with a cursor should work.

TODO:

  • make sure we have a incremental id (like uuid7) so we can sort on dataset_id to maintain order while slicing
  • Add parameter to get_dataset_collection to use pagination

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions