BIDS filter not working

[bilas003]

What happened?

BIDS-filter is not working to filter out the set of files with the latest release of the build. The job ran for 4 days and then timed out . Can anyone help?

What command did you use?

Used a slurm job to run the pipeline,see the attached image.

What version of NiBabies are you using?

nibabies-24.0.0.sif

Relevant log output

Time -out

Add any additional information or context about the problem here.

[mgxd]

Can you share the output log and the contents of the filter file you are using? Also looks like your line break on --derivatives is not properly spaced.

FYI - The latest release is 24.1.0, and we just cut a release candidate (25.0.0rc1) of a new minor series slated to come out later this week.

[bilas003]

BIDS -filter json

{
"bold":{"datatype":"func","suffix":"bold","session":"V02"},
"sbref":{"datatype": "func", "suffix": "sbref","session":"V02"},
"fmap": {"datatype": "fmap","session":"V02"},
"t2w": {"datatype": "anat", "suffix": "T2w" , "run":"2", "session": "V02","SeriesDescription":"anat-t2w","ProtocolName":"anat-t2w"}

}

I corrected the derivatives flag and ran again.

[mgxd]

Were any T2w images found? I don't think --bids-filter-file will filter metadata, only bids entities directly in the filename.

[erikglee]

@mgxd, nibabies uses pybids to find files based on the bids filter file, right? @bilas003 had tried running some independent tests with pybids that involved a filter file with fields that reference the JSON metadata. In her example, it looked like pybids came up with the right files based on the filter settings.

[erikglee]

Hi @bilas003, @mgxd just informed me that another good option to test would be trying to put null (without quotations) for the t2w "acq" entity in the BIDS Filter File. This should be another strategy to avoid the localizer images (since they have "acq-mrsLoc" in their name) during processing, without Nibabies needing to search through the JSON meta-data.

Could you try running another test where you replace the t2w filter file content to be:
"t2w": {"datatype": "anat", "suffix": "T2w" , "run":"2", "session": "V02", "acq": null}

[bilas003]

Hi @mgxd @erikglee

Added the acq tag as null to bids-filter.

The new error received is
ValueError: 'acq' is not a recognized entity. If you're sure you want to impose this constraint, set invalid_filters='allow'.

[mgxd]

You'll have to use the full entity name - acquisition

[bilas003]

The job has been running for 3 days now. The error log shows FileNotFound (see screenshot)

[mgxd]

There are a few moving parts here, can you clarify the following?

1. What does your bids directory tree look like for the subject you are running?
2. What does your filter file look like now?

Additionally, some suggestions:

• --skip-bids-validation should not be used when debugging querying a BIDS dataset
• If you are just starting analysis, try using 25.0.1 as it's the most up to date version
• --participant-label <id> can be used in your command to narrow down processing for a single subject, simplifying debugging
• --age-months is deprecated and shouldn't be used

[bilas003]

Hi @mgxd

Here is the tree structure

filter like is

I followed your suggestions and made changes with latest version of sif , the job fails with below err

[mgxd]

RE: filter - Which file are you trying to isolate? I don't see anything matching your T2w criteria...you are looking for:
anat/sub-<>_ses-V02_run-2_T2w...

RE: the validation failure, I'm not quite sure what ERR_STRING_TOO_LONG is complaining about, perhaps @effigies @oesteban @rwblair have seen this before

[bilas003]

Hi Mattias
For anat scans we are trying to filter out run_2 V02 . I have attached the screenshot with files highlighted.

[mgxd]

--bids-filter is inclusive, so only the subsequent matches will be used. It seems like you are looking to use it to exclude - this behavior is not implemented. Your best bet would be to move the file to somewhere like sourcedata/raw/sub-<>/ses-V02/anat/, where it will be ignored.

FYI - if multiple files are found for t1w/t2w, they are used to form an average.

[erikglee]

Hi Mathias, thanks for the replies! That sounds good, I think we misunderstood your comment about setting "acquisition" to null. We thought you were saying that this could create exclusion like behavior.

Another way that we could use BIDS Filter Files to get the file selection we want is to have the BIDS Filter File use information from the anat JSON sidecars such as "SeriesDescription":"anat-t2w. This is what we were originally trying, and it worked as expected in pybids, but it wasn't giving the expected behavior in Nibabies.

Do you think it would be within scope for Nibabies to accept filter files that use metadata information from within JSON sidecars?