Issue encountered while training the cord-v2

[lfy3083]

When I train cord-v2（the pre-trained model used to train cord-v2 is the donut-base model.）, the validation process shows:
：Prediction: 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, ...
Answer: <s_menu><s_nm>PREMIUM TOAST PAN BREAD</s_nm><s_cnt>1</s_cnt><s_price>24,000</s_price></s_menu><s_total><s_total_price>24,000</s_total_price><s_cashprice>24,000</s_cashprice><s_changeprice>0</s_changeprice></s_total>
Normed ED: 0.9977146588312112

After training, the saved model is test2, and when I use the test2 model for inference, there are also garbled characters, and the accuracy is 0.012....,

and I change the transformer version to 4.25.1 according to the solution proposed by previous questions and answers, and I can't solve the problem.

My Python version is 3.10. Here is a list of my dependencies：
Package Version

---

absl-py 2.1.0
aiohttp 3.8.4
aiosignal 1.3.1
async-timeout 4.0.2
asynctest 0.13.0
attrs 23.1.0
certifi 2022.12.7
charset-normalizer 3.1.0
click 8.1.3
datasets 2.12.0
dill 0.3.6
filelock 3.12.0
frozenlist 1.3.3
fsspec 2023.1.0
grpcio 1.69.0
huggingface-hub 0.14.1
idna 3.4
importlib-metadata 6.6.0
Jinja2 3.1.5
joblib 1.2.0
lightning-utilities 0.8.0
Markdown 3.7
MarkupSafe 3.0.2
mpmath 1.3.0
multidict 6.0.4
multiprocess 0.70.14
munch 3.0.0
networkx 3.4.2
nltk 3.8.1
numpy 1.21.6
nvidia-cublas-cu11 11.10.3.66
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu11 8.5.0.96
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
packaging 23.1
pandas 1.3.5
Pillow 9.5.0
pip 22.3.1
propcache 0.2.1
protobuf 5.29.3
pyarrow 12.0.0
python-dateutil 2.9.0.post0
pytorch-lightning 2.2.1
pytz 2023.3
PyYAML 6.0
readline 6.2.4.1
regex 2024.11.6
requests 2.30.0
responses 0.18.0
ruamel.yaml 0.17.26
ruamel.yaml.clib 0.2.7
safetensors 0.3.1
sconf 0.2.5
sentencepiece 0.1.99
setuptools 65.6.3
six 1.16.0
sympy 1.13.1
tensorboard 2.18.0
tensorboard-data-server 0.7.2
timm 0.9.16
tk 0.1.0
tokenizers 0.13.3
torch 2.1.2+cu118
torchaudio 2.1.2+cu118
torchmetrics 1.6.1
torchvision 0.16.2+cu118
tqdm 4.65.0
transformers 4.25.1
triton 2.1.0
typing_extensions 4.12.2
tzdata 2024.2
urllib3 2.0.2
Werkzeug 3.1.3
wheel 0.38.4
xxhash 3.2.0
yarl 1.9.2
zipp 3.15.0
zss 1.2.0

Can anyone tell me what my problem is?Thank you very much for your help.

[david0ql]

did you solved it?

[jfargus]

A little late here but installing
transformers==4.24.0
timm==0.6.13

BEFORE installing donut-python from the repo solved my problem when training in Colab

[eximius313]

I've been able to successfully train naver-clova-ix/donut-base with theese dependencies:

sconf==0.2.5
numpy==2.3.0
zss==1.2.0
tensorboard==2.19.0
transformers==4.38.0
timm==0.6.13
pytorch-lightning==1.8.6

[katarzynasornat]

Hi @eximius313

are/were you able by any chance regenerate this https://www.philschmid.de/fine-tuning-donut ???

I wanted to follow this tutorial on google colab but getting some trash. I suspect that it can be libraries update and some changes, but have no clue what and why.

[eximius313]

So far I was receiving empty response, but this is probably because of training data set being not big enough...

[katarzynasornat]

I've been able to successfully train naver-clova-ix/donut-base with theese dependencies:

sconf==0.2.5
numpy==2.3.0
zss==1.2.0
tensorboard==2.19.0
transformers==4.38.0
timm==0.6.13
pytorch-lightning==1.8.6

@eximius313 so here you meant that Trainign was fine, but inference of the fine tuned (trained) model was crappy?

[katarzynasornat]

So far I was receiving empty response, but this is probably because of training data set being not big enough...

I think it is related to newer version of transformers - check here. I downgraded transformers to 4.48.0 and it looks fine - meaning finetuning worked. Let me know if that works for you

[eximius313]

@katarzynasornat

@eximius313 so here you meant that Trainign was fine, but inference of the fine tuned (trained) model was crappy?

exactly

I think it is related to newer version of transformers - check huggingface/transformers#39473. I downgraded transformers to 4.48.0 and it looks fine - meaning finetuning worked. Let me know if that works for you

Can you please paste all the versions that worked for you?

[david0ql]

@katarzynasornat

@eximius313 so here you meant that Trainign was fine, but inference of the fine tuned (trained) model was crappy?

exactly

I think it is related to newer version of transformers - check huggingface/transformers#39473. I downgraded transformers to 4.48.0 and it looks fine - meaning finetuning worked. Let me know if that works for you

Can you please paste all the versions that worked for you?

don't use donut, it's to hard to keep all in mind and also very specific versions. i found there is a better alternative to be using unsloth

When I train cord-v2（the pre-trained model used to train cord-v2 is the donut-base model.）, the validation process shows: ：Prediction: 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, ... Answer: <s_menu><s_nm>PREMIUM TOAST PAN BREAD</s_nm><s_cnt>1</s_cnt><s_price>24,000</s_price></s_menu><s_total><s_total_price>24,000</s_total_price><s_cashprice>24,000</s_cashprice><s_changeprice>0</s_changeprice></s_total> Normed ED: 0.9977146588312112

After training, the saved model is test2, and when I use the test2 model for inference, there are also garbled characters, and the accuracy is 0.012....,

and I change the transformer version to 4.25.1 according to the solution proposed by previous questions and answers, and I can't solve the problem.

My Python version is 3.10. Here is a list of my dependencies： Package Version

absl-py 2.1.0 aiohttp 3.8.4 aiosignal 1.3.1 async-timeout 4.0.2 asynctest 0.13.0 attrs 23.1.0 certifi 2022.12.7 charset-normalizer 3.1.0 click 8.1.3 datasets 2.12.0 dill 0.3.6 filelock 3.12.0 frozenlist 1.3.3 fsspec 2023.1.0 grpcio 1.69.0 huggingface-hub 0.14.1 idna 3.4 importlib-metadata 6.6.0 Jinja2 3.1.5 joblib 1.2.0 lightning-utilities 0.8.0 Markdown 3.7 MarkupSafe 3.0.2 mpmath 1.3.0 multidict 6.0.4 multiprocess 0.70.14 munch 3.0.0 networkx 3.4.2 nltk 3.8.1 numpy 1.21.6 nvidia-cublas-cu11 11.10.3.66 nvidia-cublas-cu12 12.4.5.8 nvidia-cuda-cupti-cu12 12.4.127 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-nvrtc-cu12 12.4.127 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cuda-runtime-cu12 12.4.127 nvidia-cudnn-cu11 8.5.0.96 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.2.1.3 nvidia-curand-cu12 10.3.5.147 nvidia-cusolver-cu12 11.6.1.9 nvidia-cusparse-cu12 12.3.1.170 nvidia-nccl-cu12 2.21.5 nvidia-nvjitlink-cu12 12.4.127 nvidia-nvtx-cu12 12.4.127 packaging 23.1 pandas 1.3.5 Pillow 9.5.0 pip 22.3.1 propcache 0.2.1 protobuf 5.29.3 pyarrow 12.0.0 python-dateutil 2.9.0.post0 pytorch-lightning 2.2.1 pytz 2023.3 PyYAML 6.0 readline 6.2.4.1 regex 2024.11.6 requests 2.30.0 responses 0.18.0 ruamel.yaml 0.17.26 ruamel.yaml.clib 0.2.7 safetensors 0.3.1 sconf 0.2.5 sentencepiece 0.1.99 setuptools 65.6.3 six 1.16.0 sympy 1.13.1 tensorboard 2.18.0 tensorboard-data-server 0.7.2 timm 0.9.16 tk 0.1.0 tokenizers 0.13.3 torch 2.1.2+cu118 torchaudio 2.1.2+cu118 torchmetrics 1.6.1 torchvision 0.16.2+cu118 tqdm 4.65.0 transformers 4.25.1 triton 2.1.0 typing_extensions 4.12.2 tzdata 2024.2 urllib3 2.0.2 Werkzeug 3.1.3 wheel 0.38.4 xxhash 3.2.0 yarl 1.9.2 zipp 3.15.0 zss 1.2.0

Can anyone tell me what my problem is?Thank you very much for your help.

and now you can close the issue

https://docs.unsloth.ai/basics/vision-fine-tuning

there is a very basic explanation, let me know if someone needs a tutorial maybe i can get the time and do it for you :)