Enable filtering of specific subject session pairs

[ghisvail]

I work on pydra-bids, a set of Pydra tasks to manipulate BIDS datasets and powered by ancpbids under the hood.

The Clinica project aims at replacing their custom query logic with pydra-bids and there is an important feature currently missing. They are dealing with longitudinal datasets with a rather large range of subjects and sessions and would want to filter processing to a subset of them.

Right now, it seems that ancpbids.BIDSLayout.get only support filtering a subset of participants amongst all participants and / or a subset of sessions amongst all sessions. How about supporting a specific set of subject session pairs?

The way I see it, this feature could be introduced with a new parameter like:

class BIDSLayout:
    def get(..., subject_session_pairs: Iterable[Tuple[str, str]]):
        ...

layout = BIDSLayout("/path/to/bids/dataset")
query = {"return_type": "files", "suffix": "T1w"}

# Query session M00 of participant P01
# and session M12 of participant P02.
subject_session_pairs = [
    ("P01", "M00"),
    ("P02", "M12"),
]

files = layout.get(subject_session_pairs=subject_session_pairs, **query)

What do you think? Would it be difficult to implement given the existing design?

[erdalkaraca]

So, you are asking for ANDing the filter criteria instead of ORing.
This would already be possible by using the internal query API which is considered low level at the moment, though.

You can find an example in the test suite (adapted to your use case):

    def test_query_language(self):
        ds = <your ds> # ancpbids.load_dataset(DS005_DIR)
        schema = ds.get_schema()
        file_paths = ds.select(schema.Artifact) \
            .where(any_of(
              all_of(entity(schema, schema.EntityEnum.subject, 'P01'), entity(schema, schema.EntityEnum.session, 'M00')),\
              all_of(entity(schema, schema.EntityEnum.subject, 'P02'), entity(schema, schema.EntityEnum.session, 'M12')),\
             )).get_file_paths()
        file_paths = list(file_paths)
        # self.assertEqual(3, len(file_paths))

[ghisvail]

Of course, I could loop through the subject session pairs myself and accumulate results from BIDSLayout.get(subjects=subject, sessions=session, ...), but since the query parameters already allow to specify a list of subject labels and / or a list of session labels to filter the results with, it would make sense to add the possibility to specify a list of pairs.

It could also be expressed by an additional parameter like BIDSLayout.get(subjects: Iterable[str], sessions=sessions: Iterable[str], pair_subjects_with_sessions: bool = False) to specify whether the filter is the product of the lists (default behaviour) or the zip of both. I like this solution less because of worse type safety.

[ghisvail]

So, you are asking for ANDing the filter criteria instead of ORing. This would already be possible by using the internal query API which is considered low level at the moment, though.

You can find an example in the test suite (adapted to your use case):

    def test_query_language(self):
        ds = <your ds> # ancpbids.load_dataset(DS005_DIR)
        schema = ds.get_schema()
        file_paths = ds.select(schema.Artifact) \
            .where(any_of(
              all_of(entity(schema, schema.EntityEnum.subject, 'P01'), entity(schema, schema.EntityEnum.session, 'M00')),\
              all_of(entity(schema, schema.EntityEnum.subject, 'P02'), entity(schema, schema.EntityEnum.session, 'M12')),\
             )).get_file_paths()
        file_paths = list(file_paths)
        # self.assertEqual(3, len(file_paths))

Thanks for you reply. I suspected this was possible somehow with the low-level query engine 👍

In Clinica, these subject session pairs are provided in a dynamic fashion via a config file so constructing the low-level query could become a bit impractical. I was wondering whether the higher-level API could be extended to support this use case.

[erdalkaraca]

It could also be expressed by an additional parameter like BIDSLayout.get(subjects: Iterable[str], sessions=sessions: Iterable[str], pair_subjects_with_sessions: bool = False) to specify whether the filter is the product of the lists (default behaviour) or the zip of both. I like this solution less because of worse type safety.

Maybe, a convenience parameter could be provided to BIDSLayout.get():

additional_criteria= any_of(
              all_of(entity(schema, schema.EntityEnum.subject, 'P01'), entity(schema, schema.EntityEnum.session, 'M00')),\
              all_of(entity(schema, schema.EntityEnum.subject, 'P02'), entity(schema, schema.EntityEnum.session, 'M12')),\
             )
files = layout.get(additional_criteria=additional_criteria, **query)

[ghisvail]

The nice thing with your high-level API is that it hides very well to the user how the query is computed internally using basic Python primitives. That's good encapsulation.

In my opinion, you're leaking too much of the internals of your query engine with the additional_criteria solution.

[erdalkaraca]

The nice thing with your high-level API is that it hides very well to the user how the query is computed internally using basic Python primitives. That's good encapsulation.

In my opinion, you're leaking too much of the internals of your query engine with the additional_criteria solution.

That is true, but covering all possible custom use cases might make the implementation of BIDSLayout more complicated and not maintainable anymore.

[ghisvail]

Put differently – As a user, all I care is giving a list of subjects and sessions to filter the results with.

Now the question is how to modify the high-level API to capture zipping and products of these two lists. That's not a huge amount of additional use cases to be honest.

[erdalkaraca]

And what about someone else being interested in a pair of subject/run, run/ses, (any other combination)?
There must be a generic mechanism to handle this easily... will think about this :-)

[ghisvail]

And what about someone else being interested in a pair of subject/run, run/ses, (any other combination)?

Fair enough. However, a key difference is that entities such as run are suffix-specific and may not be present in all subject session pairs, so it would be quite difficult to generalise and capture in the high-level API.

On the other hand, combining subjects and sessions applies to the whole dataset regardless of the desired modalities being queried. Therefore, it makes sense to expose them to the high-level API as you did.

There must be a generic mechanism to handle this easily... will think about this :-)

You could support an iterable of optional entities to be filtered with, but it would probably become pretty gnarly very quickly I think.

In my opinion, dealing with custom combination of subject session pairs is common enough in longitudinal studies to warant a simple, yet not generic query API, like the two I proposed above (the dedicated Iterable[Tuple[str, str]] or the boolean).

[erdalkaraca]

Alternative solution: provide a plugin to extend BIDSLayout to cover this (general?) use case:

class BIDSLayoutLongitudinalExtensions(SchemaPlugin):
    def get_by_subject_session_pairs(subject_session_pairs: Iterable[Tuple[str, str]]):
        ...

BIDSLayout.get_by_subject_session_pairs(subject_session_pairs, **query)

This would be a good use case to show how the plugins mechanism works.

[ghisvail]

I like the extension by plugin feature of ancpbids. It allows support for future evolutions of the model in the spec or the potentially infinite ways of writing a BIDS derivative for instance. For these use cases, where all possible use cases can't be defined upfront, plugins are an elegant solution.

I don't think a plugin is the right tool to reach for this use case though. The posibilities of combining the subjects and sessions lists are limited and known a priori (zip and product) and the logic of filtering by subject and session already exist in BIDSLayout.get. The plugin would end-up reimplementing the same logic which is not good design.

It sounds to me that your answer to my request is leaning towards "go maintain your own plugin and leave me alone", am I correct? 😄

[ghisvail]

It sounds to me that your answer to my request is leaning towards "go maintain your own plugin and leave me alone", am I correct?

Which is totally acceptable 👍

[erdalkaraca]

I like the extension by plugin feature of ancpbids. It allows support for future evolutions of the model in the spec or the potentially infinite ways of writing a BIDS derivative for instance. For these use cases, where all possible use cases can't be defined upfront, plugins are an elegant solution.

I don't think a plugin is the right tool to reach for this use case though. The posibilities of combining the subjects and sessions lists are limited and known a priori (zip and product) and the logic of filtering by subject and session already exist in BIDSLayout.get. The plugin would end-up reimplementing the same logic which is not good design.

It sounds to me that your answer to my request is leaning towards "go maintain your own plugin and leave me alone", am I correct? 😄

It is about (from a technical point of view) making BIDSLayout as simple as possible and not put additional logic (that not everyone needs) into it.
Implementation-wise, the new function would internally re-use the exisiting BIDSLayout.get() function by just calling it in the way suggested in the previous comments but still hiding the low-level internals of the query engine :-)

[ghisvail]

Implementation-wise, the new function would internally re-use the exisiting BIDSLayout.get() function by just calling it in the way suggested in the previous comments but still hiding the low-level internals of the query engine

You mean using this snippet, right?

additional_criteria= any_of(
              all_of(entity(schema, schema.EntityEnum.subject, 'P01'), entity(schema, schema.EntityEnum.session, 'M00')),\
              all_of(entity(schema, schema.EntityEnum.subject, 'P02'), entity(schema, schema.EntityEnum.session, 'M12')),\
             )
files = layout.get(additional_criteria=additional_criteria, **query)

Is this feature already available?