Question about max seq length

[yfu06]

I noticed that the max seq length of MiniCPM4.1-8B is 64k but Infllmv2 only trains with up to 32k seq length. Also the openbmb/InfLLM-V2-Long-Sparse-Base mode only has a max seq length of 32k. I wonder why there is such a difference.

[suhmily10]

MiniCPM4.1-8B is a mixed-reasoning model initialized from the MiniCPM4-8B long-decay model. Driven by the observation that longer generated sequences consistently yield superior results, we have extended the sequence length of MiniCPM4.1-8B to 64k. This model is architecturally identical to the InfLLMv2 introduced in our paper. It is worth noting that InfLLMv2 was trained with a maximum sequence length of 32k solely to facilitate efficient comparison with other baseline models.

[yfu06]

Thanks for your reply! Which model is used for evaluation in infLLMv2 paper? Is it MiniCPM4.1-8B or InfLLM-V2-Long-Sparse-Base?

[suhmily10]

InfLLM-V2-Long-Sparse-Base is used for evaluation in infLLMv2 paper.