Validation data is now available!!

[sarthakpati]

Hello All,

The training and validation data is now available! Please register to receive the download link and further instructions.

Cheers,
Organizing Team

[dskhanirfan]

Dear Organizers,
Thanks
Validation Data is available, However, is there any code available to evaluate the performance metrics on validation dataset?

[zhanghaoyue]

currently the IPP platform asks us to upload nii.gz format of validation result. Are we expected to write our own test code to perform the prediction or a standardized notebook will be provided?
Is there a ranking board at this moment?

Thank you!

[sarthakpati]

Are we expected to write our own test code to perform the prediction

Tagging @brandon-edwards @msheller @psfoley @alexey-gruzdev

Is there a ranking board at this moment?

Not at the moment, we are currently working towards this, and will update participants soon!

[brandon-edwards]

currently the IPP platform asks us to upload nii.gz format of validation result. Are we expected to write our own test code to perform the prediction or a standardized notebook will be provided?

We will provide updated code with this feature. An update will come within the next few days, and I will also post to this thread to notify you directly.

[brandon-edwards]

@zhanghaoyue sorry for the delay, validation code was just merged. The notebook has additional cells (at the bottom of the notebook) now that take you through the process to create the folder of best model inference outputs.

[Linardos]

The provided validation code seems to do inference on the validation data, but is there a script to also obtain and store Dice and Hausdorff metrics from it?

[eceisik]

Dear Organizers,
do we have a daily quota for validation job submission? I remembered that I saw three job submission limit but I couldn't find the explanation again.
Best,
Ece

[dskhanirfan]

My Jobs are 6/10, I have the same problem

[ujjwalbaid0408]

You can delete the completed jobs. We are receiving a high volume of jobs for validation and it would take some time for your job to finish. Please let us know if this is still pending for the next 48 hours.

[Linardos]

Hmm, it's been 24 hours+ without progress still. I wanted to point out that it's not "pending" it's been stuck on the "running" stage as described in the platform:

[ujjwalbaid0408]

Hi Akis,

We are looking into this and will get back to you asap. If you receive any notification from IPP please let us know.

[ujjwalbaid0408]

Hi @Linardos,

Is the issue resolved now?

[zhanghaoyue]

I uploaded validation data from the output folder in separated files, names are like this : FeTS21_Vlidation_001_seg.nii.gz..I do check the segmented results visually and they are there. However when I submitted the result to IPP I got this:
core-FeTS21eval_validationPhase.execlist.1 log:
No Matching Ground Truth Files Found! Exiting.

[dskhanirfan]

Are you using the 3rd link on ipp portal?

[zhanghaoyue]

yes

[dskhanirfan]

I can see three options on ipp portal. Can you kindly provide the link where I can upload my validation data that everyone is using? Is it the 3rd link in the pic?

[sarthakpati]

@zhanghaoyue Can you confirm that you don't have "_seg" present in the filenames during submission?

[zhanghaoyue]

the file name like this "FeTS21_Validation_001_seg.nii.gz"

[sarthakpati]

Please remove the "_seg" in the filename and resubmit.

[shouldersLL]

The readme file of task 2 requires to use "_seg" for the name of predictions. Should we keep or remove it in our submitted container? @sarthakpati

Many thanks.

[sarthakpati]

For Task-2, please keep the "_seg" nomenclature.

[Linardos]

Yes! Just saw the results. It's working now

…

On Sat, Jul 24, 2021, 12:43 AM Ujjwal Baid ***@***.***> wrote: Hi @Linardos <https://github.com/Linardos>, Is the issue resolved now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#29 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGLVHT44MAO7DCZAZNFE37LTZHO6TANCNFSM46XW35KQ> .