ncluster ssh failure

[rahimentezari]

Hello and thanks for your great work.
I tried to rerun your code on 2xP3.16xlarge. Everything seems fine up to creating machines. They run but ncluster cannot ssh to both machines:
1.imagenet: Exception connecting to "IP" via ssh (could be a timeout): [Errno None] Unable to connect to port 22 on "IP".

Is there any specific details that I needed to take care of?
I checked the created VMs and both allow SSH on port 22.
It is worth mentioning that I copied the key from the following path in the machine I am running your code /home/ubuntu/.ncluster/ncluster-ubuntu-831839378763-us-east-2.pem and could not ssh to both machines too. connection refused!

[yaroslavvb]

Does any ncluster task work? IE, something like https://github.com/cybertronai/ncluster/blob/master/examples/simple_task.py?

Wondering if it's problem with ncluster, or with imagenet18 repo

[rahimentezari]

I guess this is working. This gives me the following:

ubuntu@ip-172-31-24-237:~/ncluster/examples$ python simple_task.py 
ncluster version 0.1.91
('Found local keypair /home/ubuntu/.ssh/id_rsa',) {}

.
.
.

2020-09-13 21:46:27.124084 unnamedtask-0-fw2qy: tmux> sudo chmod 777 /ncluster
2020-09-13 21:46:27.219833 unnamedtask-0-fw2qy: tmux> export PYTHONUNBUFFERED=1
2020-09-13 21:46:27.303014 unnamedtask-0-fw2qy: Initialize complete
2020-09-13 21:46:27.303064 unnamedtask-0-fw2qy: To connect to unnamedtask-0-fw2qy do "ncluster connect unnamedtask-0-fw2qy" or
    ssh ec2-user@3.131.133.85
    tmux a
2020-09-13 21:46:27.306676 unnamedtask-0-fw2qy: tmux> ifconfig
Task ifconfig returned None

I wonder if sth is wrong with Imagenet18 repo or I configured sth wrong along with Imagenet18 instructions. I guess I did right, as the only thing is to set AWS key, etc and everything else is running only codes.