AssertionError:Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. when finetuning #66
Description
when I set my finetuning.sh file like this:
accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 1 --num_processes 2 \
--config_file ./library/accelerate_config.yaml \
finetuning.py \
--pretrained_model_name_or_path ./step1x-edit-i1258.safetensors \
--qwen2p5vl ./Qwen2.5-VL-7B-Instruct/ \
--ae ./vae.safetensors \
--cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers \
--max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 \
--network_module library.lora_module --network_dim 64 --network_alpha 32 --network_train_unet_only \
--optimizer_type AdamW --learning_rate 1e-4 \
--cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \
--highvram --max_train_epochs 100 --save_every_n_epochs 5 --dataset_config library/data_configs/step1x_edit.toml \
--output_dir /path/to/step1x_test/ \
--output_name step1x-edit_test \
--timestep_sampling shift --discrete_flow_shift 3.1582 --model_prediction_type raw --guidance_scale 1.0
The finetuning scripts kept throwing me this error:
File "/path/to/Step1X-Edit/modules/layers.py", line 419, in apply_rope
xk_out = freqs_cis[..., 0] * xk_[..., 0] + freqs_cis[..., 1] * xk_[..., 1]
(Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:111.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
...
[rank0]: torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
[rank0]: AssertionError: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.clone.default(tensor([...], size=(16,), dtype=torch.uint8), memory_format=torch.contiguous_format)
My environments are:
absl-py==2.3.0
accelerate==0.34.2
aiofiles==24.1.0
altair==4.2.2
annotated-types==0.7.0
anyio==4.9.0
attrs==25.3.0
av==14.4.0
bitsandbytes==0.45.5
certifi==2025.4.26
charset-normalizer==3.4.2
click==8.2.1
diffusers==0.32.2
easygui==0.98.3
einops==0.8.1
entrypoints==0.4
exceptiongroup==1.3.0
fastapi==0.115.12
ffmpy==0.5.0
filelock==3.13.1
flash-attn @ file:///path/to/Step1X-Edit/flash_attn-2.7.2.post1%2Bcu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl#sha256=aaca54f8ee67507c92683e7b71e524f31d370d50ec110811e4b7492976ebe89f
fsspec==2024.6.1
ftfy==6.1.1
gradio==5.29.0
gradio_client==1.10.0
groovy==0.1.2
grpcio==1.71.0
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.30.2
idna==3.10
imagesize==1.4.1
importlib_metadata==8.7.0
Jinja2==3.1.4
jsonschema==4.24.0
jsonschema-specifications==2025.4.1
liger_kernel==0.5.4
lion-pytorch==0.0.6
Markdown==3.8
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
mpmath==1.3.0
networkx==3.3
numpy==2.1.2
nvidia-cublas-cu12==12.4.2.65
nvidia-cuda-cupti-cu12==12.4.99
nvidia-cuda-nvrtc-cu12==12.4.99
nvidia-cuda-runtime-cu12==12.4.99
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.0.44
nvidia-curand-cu12==10.3.5.119
nvidia-cusolver-cu12==11.6.0.99
nvidia-cusparse-cu12==12.3.0.142
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.4.99
nvidia-nvtx-cu12==12.4.99
opencv-python==4.8.1.78
opencv-python-headless==4.11.0.86
orjson==3.10.18
packaging==25.0
pandas==2.2.3
pillow==11.1.0
prodigy-plus-schedule-free==1.9.0
prodigyopt==1.1.2
protobuf==6.31.0
psutil==7.0.0
pydantic==2.11.5
pydantic_core==2.33.2
pydub==0.25.1
Pygments==2.19.1
python-dateutil==2.9.0.post0
python-multipart==0.0.20
pytorch_optimizer==3.5.0
pytz==2025.2
PyYAML==6.0.2
qwen-vl-utils==0.0.10
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
rich==13.9.4
rpds-py==0.25.1
ruff==0.11.11
safehttpx==0.1.6
safetensors==0.4.5
schedulefree==1.4
semantic-version==2.10.0
sentencepiece==0.2.0
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
starlette==0.46.2
sympy==1.13.3
tensorboard==2.19.0
tensorboard-data-server==0.7.2
tokenizers==0.21.1
toml==0.10.2
tomlkit==0.13.2
toolz==1.0.0
torch==2.4.0+cu124
torchvision==0.19.0+cu124
tqdm==4.67.1
transformers==4.51.3
triton==3.0.0
typer==0.16.0
typing-inspection==0.4.1
typing_extensions==4.12.2
tzdata==2025.2
urllib3==2.4.0
uvicorn==0.34.2
voluptuous==0.13.1
wcwidth==0.2.13
websockets==15.0.1
Werkzeug==3.1.3
zipp==3.22.0