BUG: Multi-Node Dataset Preprocessing ValueError: invalid literal for int() with base 10: ''

[liudan193]

[rank2]: Traceback (most recent call last):
[rank2]: File "/miniconda3/envs/axolotl_202506/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank2]: return _run_code(code, main_globals, None,
[rank2]: File "/miniconda3/envs/axolotl_202506/lib/python3.10/runpy.py", line 86, in _run_code
[rank2]: exec(code, run_globals)
[rank2]: File "/src/axolotl/cli/train.py", line 131, in
[rank2]: fire.Fire(do_cli)
[rank2]: File "/miniconda3/envs/axolotl_202506/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
[rank2]: component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank2]: File "/miniconda3/envs/axolotl_202506/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
[rank2]: component, remaining_args = _CallAndUpdateTrace(
[rank2]: File "/miniconda3/envs/axolotl_202506/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank2]: component = fn(*varargs, **kwargs)
[rank2]: File "/src/axolotl/cli/train.py", line 105, in do_cli
[rank2]: return do_train(parsed_cfg, parsed_cli_args)
[rank2]: File "/src/axolotl/cli/train.py", line 59, in do_train
[rank2]: dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
[rank2]: File "/src/axolotl/common/datasets.py", line 60, in load_datasets
[rank2]: train_dataset, eval_dataset, total_num_steps, prompters = prepare_datasets(
[rank2]: File "/src/axolotl/utils/data/utils.py", line 50, in wrapper
[rank2]: return func(*args, **kwargs)
[rank2]: File "/src/axolotl/utils/data/sft.py", line 67, in prepare_datasets
[rank2]: return _prepare_standard_dataset(cfg, tokenizer, processor, preprocess_iterable)
[rank2]: File "/src/axolotl/utils/data/sft.py", line 105, in _prepare_standard_dataset
[rank2]: loader.cleanup()
[rank2]: File "/src/axolotl/utils/data/lock.py", line 57, in cleanup
[rank2]: count = int(self.counter_path.read_text().strip())
[rank2]: ValueError: invalid literal for int() with base 10: ''

[NanoCode012]

Could you provide your dataset config yaml? (feel free to redact dataset name)

Since you mentioned it was multi-node, can you ensure the path is the same and exists on each node?

[liudan193]

I'm sure the path is the same and exists on each node. I don’t encounter this problem on a single node with eight GPUs, but it does occur when using multiple nodes.

base_model: TODO
load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: TODO
    type: chat_template
    chat_template: jinja
    chat_template_jinja: "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}"
    field_messages: messages
    message_field_role: role
    message_field_content: content
    roles_to_train: [ ]
    train_on_eos: turn
    message_field_training: train
val_set_size: 0.005
output_dir: TODO
shuffle_merged_datasets: true

sequence_len: 20480
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: true

wandb_mode: offline
wandb_project: axolotl
wandb_entity:
wandb_watch:
wandb_name: TODO
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 3
optimizer: paged_adamw_32bit
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-8

lr_scheduler: cosine
learning_rate: 1e-5
cosine_min_lr_ratio: 0.1

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps:
warmup_ratio: 0.05
evals_per_epoch: 1
eval_table_size:
eval_batch_size:
eval_strategy: "epoch"
saves_per_epoch: 1
save_safetensors: true
debug:
deepspeed: TODO
weight_decay: 0.1
max_grad_norm: 1.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|endoftext|>
plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true

[winglian]

Is the dataset_prepared_path you have configured in a shared directory between nodes? My hunch is this is a race condition across nodes.

[liudan193]

Yes, my hard drive is shared. Multiple machines are accessing the same directory on the same hard drive. Is there a solution?

[NanoCode012]

@liudan193 , can you see if this fixes it #2959

[liudan193]

@liudan193 , can you see if this fixes it #2959

It prevents errors, but I doubt simply doing this might cause bad processed dataset. I hope the developers can add an option so that when multiple nodes access the same directory, only one node handles the data processing.

[NanoCode012]

@liudan193 , this is a simple lock that would prevent multiple processes overwriting. This should solve your issue.

[liudan193]

@liudan193 , this is a simple lock that would prevent multiple processes overwriting. This should solve your issue.

Wow, I see, thank you for your help.