mask_token_id is not matched with SDAR

[zaiquanyang]

In this script, the mask_toke_id is set as

DARE/scripts/run_sft_peft.sh

Line 34 in 826acb7

+data.mask_token_id=126336 \

However, the mask_token_id in SDAR is 151669 https://github.com/JetAstra/SDAR/blob/c2041a8333333d685f6e15122c84f60f51f85ebd/README.md?plain=1#L100 . Will this affect the performance of SDAR after SFT?

[yjyddq]

The mask_token_id of each model must correct.

In fact, the script in scripts/run_sft_peft.sh actually provides an example for LLaDA with a hardcoded mask_token_id. It is recommended to use the model-specific training scripts in recipe/sdar, as they all have the corresponding mask_token_id appropriately.

To prevent unnecessary confusion, we have removed the training scripts in scripts/*. Please use the training scripts in the recipe/*.

[zaiquanyang]

Thanks for your reply~

[zaiquanyang]

The implementation seems to be inaccurate for evaluation. The code in

DARE/opencompass/opencompass/models/sdar_generate.py

Line 30 in dea6a39

def sample_with_temperature_topk_topp(logits, temperature=1.0, top_k=0, top_p=1.0):

first truncates the logits when top-k=1 and then performs softmax, which leads to that probability of most tokens is up to 1.0 regardless of its position.