Question about token shifting

[fur9049-ctrl]

Hello dear author, thank you for sharing the Delphi code.
I have a question: I noticed that in the Delphi\utils.py file there is an operation like
# a technical detail: the token 0 is reserved for padding, so we shift all tokens by one tokens = tokens + 1
In this case, does the token_id corresponding to the model’s output also get shifted?

[yozhikoff]

Hi!

The shifting happens only once after we load the raw data from the .bin files. After that, in all other places (model input, model output, the .csv files with token-related information), we use the same "post-shifted" vocabulary.

[fur9049-ctrl]

Hi!

The shifting happens only once after we load the raw data from the .bin files. After that, in all other places (model input, model output, the .csv files with token-related information), we use the same "post-shifted" vocabulary.

Thank you very much for your explanation! Wishing you success in your research!

[fur9049-ctrl]

Hello, may I ask another question? When you make predictions, are the input events are the "post-shifted" vocabulary? In that case, does the first line of your "label" file still need to be “Padding”, since one token_id = 0 has already been reserved for padding? I’m not sure if my understanding is correct.

[yozhikoff]

Sure!
I understand the confusion - there's no fundamental reason for this whole shifting to exist, aside from the purely historical ones.

When you make predictions, are the input events are the "post-shifted" vocabulary?

Yes.

In that case, does the first line of your "label" file still need to be “Padding”, since one token_id = 0 has already been reserved for padding?

I'm not sure if I understand this part. The token with token_id=0 is exactly the one in the first line of the label file.
By the way, here's the more verbose version of the label file - maybe it can clarify the situation.
https://github.com/gerstung-lab/Delphi/blob/main/delphi_labels_chapters_colours_icd.csv

Either way, please feel free to ask more questions if my reply didn't fully answer them.

[fur9049-ctrl]

Thank you very much for your answer! I think I should have understood this part now. I wish you good health and success in your research!

Sure! I understand the confusion - there's no fundamental reason for this whole shifting to exist, aside from the purely historical ones.

When you make predictions, are the input events are the "post-shifted" vocabulary?

Yes.

In that case, does the first line of your "label" file still need to be “Padding”, since one token_id = 0 has already been reserved for padding?

I'm not sure if I understand this part. The token with token_id=0 is exactly the one in the first line of the label file. By the way, here's the more verbose version of the label file - maybe it can clarify the situation. https://github.com/gerstung-lab/Delphi/blob/main/delphi_labels_chapters_colours_icd.csv

Either way, please feel free to ask more questions if my reply didn't fully answer them.

[robomotic]

Hello @yozhikoff thanks for the explanation and sharing the https://github.com/gerstung-lab/Delphi/blob/main/delphi_labels_chapters_colours_icd.csv file.
Are the counts referring to the train.bin file or train.bin + val.bin file? or to your original training file that is not published?
Cheers.

[yozhikoff]

Hi @robomotic
It's the counts for the original train.bin file that is not published.

Best, Artem

[fur9049-ctrl]

Dear Authors,
Thank you very much for your previous responses and assistance.
I noticed in some of your earlier communications that you mentioned the possibility of providing an independent UKB RAP training script. I would like to inquire whether there is an available channel to obtain this script.
Specifically, we would like to confirm whether this script can be used to perform training on the same dataset and thereby reproduce or obtain the model checkpoints.
We would greatly appreciate it if you could kindly provide relevant information or guidance on this matter.
Wishing you continued success and all the best.
Sincerely,
Fur