【consult🙋‍♂️】some problem about model&data

[TITC]

1. the encoder architecture used in this project is a Hybrid (ResNet&CNN) architecture, have you tried ViT without CNN backbone and how about it?
   I know you recommend CNN backbone at some suggestions #112 (comment) ,but the author suggests, the original ViT is better than Hybrid except the small model size, so I have some curiosity.

Vision Transformers generally outperform ResNets with the same computational budget. Hybrids improve upon pure Transformers for smaller model sizes, but the gap vanishes for larger models.

2. Have you experimented with some other position embedding strategy? If I haven't misunderstood your meaning at 【consult🙋‍♂️】 some confusion about CustomVisionTransformer.forward_features #130 (comment), I think the position embedding manner is different with ViT suggest 1D position embedding.

3. How many data you used for training the released model? you mentioned wikipedia, arXiv, im2latex-100k dataset, and provide a pre-processed data link. But the number of LaTex list below not equal to the number of picture, so I guess maybe your training set is larger than released?

• CROHME.zip
  10822 handwritten dataset(CROHME) and 10846 line LaTex formula.

• formula.zip
  234884 line LaTex formula, 158480 for train, 6765 for val, and 30637 for test.

I have found some other dataset, such as I2L-140K, Im2latex-90k, Marmot Dataset, have you used them?

[lukas-blecher]

1. Initially, I used a pure ViT (6ecc3f4). But the encoder was just not performing very well. The model produced latex code but it has nothing to do with the input image. Given that the model is very small, I can confirm the authors findings.
2. Well it still is a 1D embedding. I use the same strategy as the authors only in a more generalized way.
3. That is all the data I rendered out for training. The reason the latex file has more entries is that it also contains formula of equations I wasn't able to render (because they threw some errors). I had to leave them in the file because I was lazy and my dataloader would break without them. I will probably return to that problem and compile a better, bigger dataset.
   I was not familiar with the Marmot dataset. Will have a look at that.