Apply filters to non-selected columns

[hombit]

With PyArrow and Pandas, a user can select a set of columns to return and filter by a column that is not in this set. For example:

import pandas as pd

frame = pd.read_parquet('Npix=0.parquet', columns=['ra', 'dec'], filters=[('yMeanPSFMag', '<', 15)])

However, when I try to do the same with lsdb.read_hats and the HATS server, I always get an empty DataFrame.

Could we make column selection a two-step process, where we first load the columns specified both by columns and filters, apply the filter, and then remove the columns that appeared only in filters?

[Schwarzam]

Fixed the issue. lsdb-server now includes columns from the filters in the response.

[hombit]

Thank you, @Schwarzam, for the fix. I'd keep it open for now, because the standard pyarrow/pandas behavior is still different from that: they do not include columns from filters if they are not in columns