Skip to content

npix cache to provide page access to pixel data #1865

@abuts

Description

@abuts

Majority of slow Horace filebacked operations involves access to muliple pages of data in a loop and doing something with these data. As the data to be accessed usually known from the beginning, the code will benifit from accessing these data from a separate IO thread as the thread will read and deliver next page of data when the previous data are getting processed by an algorithm.

PixelDataBase classes already have possibility to access data by pages, but this option is limited as pages are all constant size given at initialization and assumed to be adjusent and cover the whole pixels dataset. This is not the case in many situations, as e.g. cut splits data into pages with blocks randomly distributed through whole pixels dataset and do not covering whole dataset.

The threading access is particularry important for future compressed datasets, as amout of calculations to be performed for reading these data becomes substantial.

As part of this ticket there is a need to define get_page_data method on the MultipixBase classes, and modify PageOp classes to work with paged data.

This ticket is to modify PixelData to use internal npix cache to access pixels by setting up external page number where pages would be defined by this npix cache.
In addition, PageOp class and cut algorithm should be rewritten to use PixelData cache and to be able to setup and use random PixelData pages.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions