Update BidsDatasetBuilder, CreateFile and template_agnostic_files content to go towards a bids-valid output dataset

[SylvainTakerkart]

The aim of this PR: with the existing code (BidsDatasetBuilder) and our current example data available in eLab, make minimal changes so that the output dataset gets validated by the BIDS Validator...

For now, some changes are temporary-hacks, others are possibly long-term improvements...

Temporary hacks:

• modified 'CreateFile' so that only "required" BIDS components are created... this avoids generating 'genetic_info.tsv' for example! the problem with this is that is that 'participants.tsv' and 'samples.tsv' are not created anymore
• created a modified version of the BIDS schema to generate the ones that we need ('participants.tsv' and 'samples.tsv') are actually created

Definitive changes / improvements:

• renamed a few functions so that their names is more explicit (now, they are called: 'initiate_top_level_bids_files_tsv' and 'initiate_top_level_bids_files_others' (instead of 'writeheader_tsv_json_files' and 'fill_static_files')
• created 'initiate_top_level_bids_files_json' because the two existing functions didn't deal with the json files

What remains to be done: generate the proper content of the template (i.e default) files (present in 'template_agnostic_files') so that the content of all the top-level bids metadata files can be validated!

[SylvainTakerkart]

@dumaatravaie Here is a first important update on the previous code... We should meet before you start reviewing so that I can explain to you what I did and what should be tested...

[dumaatravaie]

Sure, we can have a meeting to discuss about this ..

[SylvainTakerkart]

Hello @dumaatravaie !

I've created a copy of the experiment for you to test all this... There is only one experiment, which is enough! Here is some info:

• the tag that you should use to extract experiments from elab is test_dip (there is only one experiment with this tag, you can modify this experiment as you want, it's a copy and does not interact with my work)

• I've inserted an environment variable in the data paths for this experiment: $DATA_DIR... the code will not work with $DATA_DIR in the file path, you should replace it with its value in the datafilepaths directly in elab ( if you work on niolon, it should be $DATA_DIR=/envau/work/nit/elabforms_project/)...

• the command line to use therefore is the following):
  build-bids -f ./API_elab_config/config.json -m micrs.csv -o ./output/ -t test_dip -p microscopy_confocal_v02.yml (what's important is the tag given with -t, and the project file given by -p; do not change these two!

[dumaatravaie]

Thaks @SylvainTakerkart , i will test the installation and let you know ..

[dumaatravaie]

Hello @SylvainTakerkart ,
I installed your code in my local machine, and also copied the raw data in a local folder where the file path ends with /confocal_data_spicci_CLE/celine/LSM700_250929/Lame3_S4_GLAST555_FOXA2647_ALDGFP_MOM+_40X_ZStack.czi, and then i added some code in BIDS_modality_custom.py line 165,

 if("$DATA_DIR") in (segment_info['raw_data_path']):
                    #print(segment_info['raw_data_path'])
                    segment_info['raw_data_path']=segment_info['raw_data_path'].replace("$DATA_DIR","/my/local/file/path/before/the/raw/file/path")
# after this the file path points to the full path of raw data files stored in my local disk

and it works, the code runs without error. This may be one of the solution for $DATA_DIR parameter as we discussed.

Have a nice weekend
we will discuss next week
Dipankar

[SylvainTakerkart]

Hello @SylvainTakerkart , I installed your code in my local machine, and also copied the raw data in a local folder where the file path ends with /confocal_data_spicci_CLE/celine/LSM700_250929/Lame3_S4_GLAST555_FOXA2647_ALDGFP_MOM+_40X_ZStack.czi, and then i added some code in BIDS_modality_custom.py line 165,

 if("$DATA_DIR") in (segment_info['raw_data_path']):
                    #print(segment_info['raw_data_path'])
                    segment_info['raw_data_path']=segment_info['raw_data_path'].replace("$DATA_DIR","/my/local/file/path/before/the/raw/file/path")
# after this the file path points to the full path of raw data files stored in my local disk

and it works, the code runs without error. This may be one of the solution for $DATA_DIR parameter as we discussed.

Have a nice weekend we will discuss next week Dipankar

ok, thanks!