ModuleNotFoundError: No module named 'megatron.training.tokenizer'

[pspdada]

When I use the latest version of megatraon NVIDIA Megatron Core 0.15.3 to run the script, there is an error.

ModuleNotFoundError: No module named 'megatron.training.tokenizer'

The logs:

  File "/slime/slime/backends/megatron_utils/arguments.py", line 4, in <module>
    from megatron.training.tokenizer.tokenizer import _vocab_size_with_padding
ModuleNotFoundError: No module named 'megatron.training.tokenizer'

The env:

pip show megatron-core
Name: megatron-core
Version: 0.15.3
Summary: Megatron Core - a library for efficient and scalable training of transformer based models
Home-page: https://github.com/NVIDIA/Megatron-LM/megatron/core
Author: 
Author-email: NVIDIA <nemo-toolkit@nvidia.com>
License: Apache 2.0
Location: /miniconda3/envs/psp-slime/lib/python3.11/site-packages
Requires: numpy, packaging, torch
Required-by: