Extremely large codebook size 2^56

[sunset-clouds]

I really appreciate your research work. I have some questions regarding Table 1. It seems that UniTok needs to expand the codebook size or vocabulary size to 2^56 to achieve an rFID of 0.33. In this case, how can the GPT model be trained? Won’t the complexity be too high?

[machuofan]

Hi there, thanks for your interest in our work. 2^56 is the number of possible combinations of multiple sub-codebooks, while each sub-codebook only has 4096 code entries. We use an autoregressive head to model the dependency among sub-codebooks.

[sunset-clouds]

Thank for your detailed explanation!