Ai tools kit fail to train model

[HeinLatt]

What happened?

Can you guys suggest me how do I fix this. After the update the ai toolkit, I cannot train any model and error is happing as I attached. See the log please

Steps to reproduce

No response

Relevant logs

Running 1 job
C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py:30: UserWarning: A NumPy version >=1.22.4 and <1.29.0 is required for this version of SciPy (detected version 2.3.5)
  import scipy.stats
C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py:30: UserWarning: A NumPy version >=1.22.4 and <1.29.0 is required for this version of SciPy (detected version 2.3.5)
  import scipy.stats
Error running job: Failed to import diffusers.schedulers.scheduling_dpmsolver_multistep because of the following error (look up to see its traceback):
numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
Error running on_error: cannot access local variable 'job' where it is not associated with a value
========================================
Result:
 - 0 completed jobs
 - 1 failure
========================================
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1016, in _get_module
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1016, in _get_module
        return importlib.import_module("." + module_name, self.__name__)return importlib.import_module("." + module_name, self.__name__)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\StabilityMatrix\Data\Assets\Python\cpython-3.11.13-windows-x86_64-none\Lib\importlib\__init__.py", line 126, in import_module
  File "C:\StabilityMatrix\Data\Assets\Python\cpython-3.11.13-windows-x86_64-none\Lib\importlib\__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)return _bootstrap._gcd_import(name[level:], package, level)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py", line 30, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py", line 30, in <module>
        import scipy.statsimport scipy.stats
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\stats\__init__.py", line 605, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\stats\__init__.py", line 605, in <module>
        from ._stats_py import *from ._stats_py import *
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\stats\_stats_py.py", line 37, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\stats\_stats_py.py", line 37, in <module>
        from scipy.spatial.distance import cdistfrom scipy.spatial.distance import cdist
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\spatial\__init__.py", line 110, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\spatial\__init__.py", line 110, in <module>
        from ._kdtree import *from ._kdtree import *
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\spatial\_kdtree.py", line 4, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\scipy\spatial\_kdtree.py", line 4, in <module>
        from ._ckdtree import cKDTree, cKDTreeNodefrom ._ckdtree import cKDTree, cKDTreeNode
  File "_ckdtree.pyx", line 1, in init scipy.spatial._ckdtree
  File "_ckdtree.pyx", line 1, in init scipy.spatial._ckdtree
ValueErrorValueError: : numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObjectnumpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
The above exception was the direct cause of the following exception:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\run.py", line 120, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\run.py", line 120, in <module>
        main()main()
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\run.py", line 108, in main
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\run.py", line 108, in main
        raise eraise e
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\run.py", line 95, in main
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\run.py", line 95, in main
        job = get_job(config_file, args.name)job = get_job(config_file, args.name)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\toolkit\job.py", line 28, in get_job
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\toolkit\job.py", line 28, in get_job
        from jobs import ExtensionJobfrom jobs import ExtensionJob
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\__init__.py", line 1, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\__init__.py", line 1, in <module>
        from .BaseJob import BaseJobfrom .BaseJob import BaseJob
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\BaseJob.py", line 5, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\BaseJob.py", line 5, in <module>
        from jobs.process import BaseProcessfrom jobs.process import BaseProcess
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\process\__init__.py", line 1, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\process\__init__.py", line 1, in <module>
        from .BaseExtractProcess import BaseExtractProcessfrom .BaseExtractProcess import BaseExtractProcess
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\process\BaseExtractProcess.py", line 7, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\jobs\process\BaseExtractProcess.py", line 7, in <module>
        from toolkit.metadata import get_meta_for_safetensorsfrom toolkit.metadata import get_meta_for_safetensors
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\toolkit\metadata.py", line 9, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\toolkit\metadata.py", line 9, in <module>
        from toolkit.train_tools import addnet_hash_legacyfrom toolkit.train_tools import addnet_hash_legacy
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\toolkit\train_tools.py", line 10, in <module>
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\toolkit\train_tools.py", line 10, in <module>
        from diffusers import (from diffusers import (
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1007, in __getattr__
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1007, in __getattr__
        value = getattr(module, name)value = getattr(module, name)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1006, in __getattr__
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1006, in __getattr__
        module = self._get_module(self._class_to_module[name])module = self._get_module(self._class_to_module[name])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1018, in _get_module
  File "C:\StabilityMatrix\Data\Packages\AI-Toolkit\venv\Lib\site-packages\diffusers\utils\import_utils.py", line 1018, in _get_module
        raise RuntimeError(raise RuntimeError(
RuntimeErrorRuntimeError: : Failed to import diffusers.schedulers.scheduling_dpmsolver_multistep because of the following error (look up to see its traceback):
numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObjectFailed to import diffusers.schedulers.scheduling_dpmsolver_multistep because of the following error (look up to see its traceback):
numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Version

2.15.4

What Operating System are you using?

Windows

[mcDandy]

Also happens in 2.16.1-rc1. Can be worked around by going back in time to commit 87edca1.

[NeuralFault]

Downgrade your numpy to 1.26.4.

SM Packages > 3dot menu for AI Toolkit > Python Packages

Scroll down to "numpy" and click it then click the version dropdown and select 1.26.4 and then the downgrade icon that appears next to the dropdown.

Another module installation likely upgraded your numpy as a dependency, version 2.x of numpy made changes that break compatibility of package's functions and extensions that still rely on numpy 1.x's version. (numpy.dtype size for instance).

It can happen from time to time, updating pytorch modules can cause this also for instance.

[mcDandy]

stability matrix will update it to the wrong version on each update...