'Submission' object has no attribute 'remote_root'

[zhao-w-en]

Hello, Developers:
Recently I encountered an error in my dpgen process. Everytime I use dpgen run para.json machine.json to submit jobs, I will receive the prompts like below:
Traceback (most recent call last): File "/home/zhaow/.local/bin/dpgen", line 8, in <module> sys.exit(main()) File "/home/zhaow/.local/lib/python3.9/site-packages/dpgen/main.py", line 175, in main args.func(args) File "/home/zhaow/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 2997, in gen_run run_iter (args.PARAM, args.MACHINE) File "/home/zhaow/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 2973, in run_iter run_model_devi (ii, jdata, mdata) File "/home/zhaow/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 1328, in run_model_devi submission.run_submission() File "/home/zhaow/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 182, in run_submission self.handle_unexpected_submission_state() File "/home/zhaow/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 220, in handle_unexpected_submission_state f"Debug information: remote_root=={self.remote_root}.\n" AttributeError: 'Submission' object has no attribute 'remote_root'
It seems dpgen goes wrong and exit, but my jobs which already submitted to pbs system are still running.
I checked the source code and found when JobStatus is unknown the prompt will appear.
However I have no idea why the jobstatus is unknown while the submitted jobs are still running.
How can I solve this problem?

[AnguseZhang]

Please provide your machine.json to help locate the problem.

[AnguseZhang]

@felix5572 Please have a look. Thanks

[zhao-w-en]

Looking forward for your reply

  "api_version": "1.0",
  "deepmd_version": "2.0.3",
  "train" :[
    {
      "command": "/home/zhaow/anaconda3/envs/deepmd/bin/dp",
      "machine": {
        "batch_type": "PBS",
        "context_type": "SSHContext",
        "local_root": "./",
        "remote_root": "/home/zhaow/WORK_SPACE/dpgen-test",
        "remote_profile": {
        "hostname": "172.19.191.54",
        "username": "zhaow",
        "passphrase": "111111",
        "port": 22
        }
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 20,
        "gpu_per_node": 0,
        "queue_name": "batch",
        "group_size": 1,
        "mem_limit": false,
        "source_list": [
        ]
      }
    }
  ],

  "model_devi": [
    {
      "machine": {
        "batch_type": "PBS",
        "context_type": "SSHContext",
        "local_root": "./",
        "remote_root": "/home/zhaow/WORK_SPACE/dpgen-test",
        "remote_profile": {
        "hostname": "172.19.191.54",
        "username": "zhaow",
        "passphrase": "111111",
        "port": 22
        }
      },
      "resources": {
        "number_node": 1,
        "gpu_per_node": 1,
        "cpu_per_node": 10,
        "queue_name": "batch",
        "mem_limit": false,
        "source_list": [],
        "module_list": [],
        "group_size": 1
      },
        "command": "mpirun -np 20 ~/anaconda3/envs/deepmd/bin/lmp"
    }
  ], 
  "fp":
  [
    {
      "machine": {
        "batch_type": "PBS",
        "context_type": "SSHContext",
        "local_root": "./",
        "remote_root": "/home/zhaow/WORK-SPACE/dpgen",
        "remote_profile": {
        "hostname": "172.19.191.51",
        "username": "zhaow",
        "passphrase": "111111",
        "port": 22
        }
      },
      "resources": {
        "number_node": 1,      
        "cpu_per_node": 28,
        "gpu_per_node": 0,
        "exclude_list": [
        ],
        "with_mpi": false,
        "source_list": [
        ],
        "module_list": [
        ],
        "time_limit": "4:0:0",
        "queue_name": "short",
        "group_size": 30,
        "_comment": "that's all"
      },
      "command": "runvasp"

    }
  ]
}
`

[felix5572]

could you please check the version of dpdispatcher？

[zhao-w-en]

deepmd-kit 2.0.3
dpdata 0.2.4
dpdispatcher 0.3.39
dpgen 0.10.4

[felix5572]

It might be due to a known bug that has been fixed now.
could you please update dpdispatcher.
pip uninstall dpdispatcher
pip install dpdispatcher

[zhao-w-en]

I have updated dpdispatcher. The error appears again. I wonder if dpgen needs root permission to get job status？

[felix5572]

could you please provide the new error message？

[zhao-w-en]

I have used python to import paramiko and tried to connect hostname "172.19.191.51" using above settings. It went very smoothly thus I don't understand why dpGen reported an error.

[felix5572]

maybe due to connection error . could you ssh to this ip ？

[njzjz]

The key is not passphrase but password. Refer https://docs.deepmodeling.org/projects/dpdispatcher/en/latest/machine.html

[njzjz]

I would suggest to add keys to the remote machine. Refer https://linux.die.net/man/1/ssh-copy-id

[zhao-w-en]

yes, your're right. The jobs submitted to another cluster successfully after I changed the passphrase to password. Actually I'm not sure about the difference between these two keyword. It seems that passphrase means a file and password is a string of Numbers.
By the way, dpgen iterated twice and finished with '2 2' in recoed.dpgen. However, I didn't get a frozen model in iter.000002/00.train/， only soft connections in the directory

Part of massage is list below

2022-01-06 23:28:41,689 - INFO : remote path: /home/zhaow/WORK_SPACE/dpgen-test/ac/f9bbb9371cb9343541627adb84cd888198349ba7
2022-01-06 23:28:49,601 - INFO : job: 5c8182750d6368b4e370b22f0d18e7111d796b89 submit; job_id is 88510.admin
INFO:dpgen:-------------------------iter.000001 task 05--------------------------
INFO:dpgen:-------------------------iter.000001 task 06--------------------------
INFO:dpgen:system 001 candidate :      0 in    301   0.00 %
**INFO:dpgen:system 001 failed    :    301 in    301 100.00 %**
INFO:dpgen:system 001 accurate  :      0 in    301   0.00 %
INFO:dpgen:system 001 accurate_ratio:   0.0000    thresholds: 1.0000 and 1.0000   eff. task min and max   -1   20   number of fp tasks:      0
INFO:dpgen:-------------------------iter.000001 task 07--------------------------
INFO:dpgen:-------------------------iter.000001 task 08--------------------------
INFO:dpgen:=============================iter.000002==============================
INFO:dpgen:-------------------------iter.000002 task 00--------------------------
INFO:dpgen:-------------------------iter.000002 task 01--------------------------
INFO:dpgen:-------------------------iter.000002 task 02--------------------------
INFO:dpgen:-------------------------iter.000002 task 03--------------------------
INFO:dpgen:finished