Report about clusterscope, or slurm related error when fine-tuning uma: keyword 'scontrol'

[ytkkk]

---

This is an error report and how to evade when following recently updated fine-tuning tutorial

Tutorial page:
https://fair-chem.github.io/core/common_tasks/fine_tuning.html

Always thanks for creating well-behaved MLIP, and diverse datasets for chemistry.
A tutorial for fine-tuning uma model is updated. It helps me a lot to know how-to-play with uma.
There was an error in following tutorial, so I write this page.

Generating toy dataset has no problem, it works fine!
But running a fine-tuning with the prepared dataset, same as ($ fairchem -c uma_sm_finetune_template.yaml)
It gives not a directory error.

  import pkg_resources
Traceback (most recent call last):
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/bin/fairchem", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/work/ytk/uma/fairchem/src/fairchem/core/_cli.py", line 421, in main
    cfg = get_hydra_config_from_yaml(args.config, override_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/uma/fairchem/src/fairchem/core/_cli.py", line 404, in get_hydra_config_from_yaml
    return get_canonical_config(cfg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/uma/fairchem/src/fairchem/core/_cli.py", line 363, in get_canonical_config
    job = OmegaConf.to_object(config.job)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/omegaconf/omegaconf.py", line 594, in to_object
    return OmegaConf.to_container(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/omegaconf/omegaconf.py", line 573, in to_container
    return BaseContainer._to_content(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/omegaconf/basecontainer.py", line 288, in _to_content
    return conf._to_object()
           ^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/omegaconf/dictconfig.py", line 765, in _to_object
    result = object_type(**init_field_items)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 15, in __init__
  File "/home/work/ytk/uma/fairchem/src/fairchem/core/_cli.py", line 171, in __post_init__
    cluster = clusterscope.cluster()
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/clusterscope/lib.py", line 14, in cluster
    return cluster_info.get_cluster_name()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/clusterscope/cache.py", line 50, in wrapper
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/clusterscope/cluster_info.py", line 74, in get_cluster_name
    result = subprocess.run(
             ^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'scontrol'

As the error looks like to originated from clusterscope, I tried to run clusterscope in cli,
$ cscope info

And almost same error found.

Traceback (most recent call last):
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/bin/cscope", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/clusterscope/cli.py", line 63, in main
    cluster_name = cluster_info.get_cluster_name()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/clusterscope/cache.py", line 50, in wrapper
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/site-packages/clusterscope/cluster_info.py", line 74, in get_cluster_name
    result = subprocess.run(
             ^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/work/ytk/pkgs/mambaforge/envs/fair2/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'scontrol'

As 'slurm' is not installed in my linux account and 'scontrol' or 's-something' is a word usually used in slurm, It might be related with slurm.

For evading this error, I editted a 171 line on fairchem/src/fairchem/core/_cli.py
Commenting out clusterscope.cluster() and defining cluster to any string solves the error, successfully run fine-tuning.

168     def __post_init__(self) -> None:
169         self.run_dir = os.path.abspath(self.run_dir)
170         try:
171             #cluster = clusterscope.cluster()
172             cluster = "ytk"
173         except RuntimeError:
174             cluster = ""

I think it is not the best method, or I could be wrong at following the fine-tuning tutorial.
But if there is anyone who have trouble in 'scontrol', wish it could help.

Thanks again for passion and contribution in chemistry.

[zulissimeta]

Thanks for the note; made an issue (#1342) to track this. I'm going to close the discussion here, please follow up on the issue.