fatal: [controller0]: FAILED! => { "msg": "Timeout (12s) waiting for privilege escalation prompt: " }

[karimbzu]

Hi,
The below error appears when i try to inspect the overloud hardware. Kindly help me to resolve this issue. Many Thanks

==============================
(kayobe-venv) [centos@localhost kayobe]$ kayobe overcloud hardware inspect -vvvv
initialize_app
prepare_to_run_command OvercloudHardwareInspect
Inspecting overcloud
Running command: ansible-playbook -vvvv --inventory /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/bifrost.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/bmc.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ceph.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/compute.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/controllers.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/dell-switch-bmp.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/dns.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/docker-registry.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/docker.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/globals.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/grafana.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/idrac.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inspector.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ipa.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ironic.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/kolla.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/monitoring.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/network-allocation.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/networks.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/neutron.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/nova.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ntp.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/opensm.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/openstack.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/overcloud.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/pip.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/seed-hypervisor.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/seed-vm.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/seed.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ssh.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/storage.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/swift.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/users.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/yum-cron.yml -e @/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/yum.yml /home/centos/kayobe-venv/share/kayobe/ansible/kolla-bifrost-hostvars.yml /home/centos/kayobe-venv/share/kayobe/ansible/overcloud-hardware-inspect.yml
/home/centos/kayobe-venv/lib/python2.7/site-packages/cryptography/init.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
CryptographyDeprecationWarning,
ansible-playbook 2.8.13
config file = None
configured module search path = [u'/home/centos/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /home/centos/kayobe-venv/lib/python2.7/site-packages/ansible
executable location = /home/centos/kayobe-venv/bin/ansible-playbook
python version = 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/groups as it did not pass it's verify_file() method
script declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/groups as it did not pass it's verify_file() method
auto declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/groups as it did not pass it's verify_file() method
Not replacing invalid character(s) "set([u'-'])" in group name (seed-hypervisor)
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

Not replacing invalid character(s) "set([u'-'])" in group name (seed-hypervisor)
Not replacing invalid character(s) "set([u'-'])" in group name (container-image-builders)
Not replacing invalid character(s) "set([u'-'])" in group name (container-image-builders)
Not replacing invalid character(s) "set([u'-'])" in group name (docker-registry)
Not replacing invalid character(s) "set([u'-'])" in group name (docker-registry)
Not replacing invalid character(s) "set([u'-'])" in group name (baremetal-compute)
Not replacing invalid character(s) "set([u'-'])" in group name (baremetal-compute)
Not replacing invalid character(s) "set([u'-'])" in group name (mgmt-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (mgmt-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (ctl-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (ctl-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (hs-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (hs-switches)
Parsed /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/groups inventory source with ini plugin
setting up inventory plugins
host_list declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/hosts as it did not pass it's verify_file() method
script declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/hosts as it did not pass it's verify_file() method
auto declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/hosts as it did not pass it's verify_file() method
Set default localhost to localhost
Not replacing invalid character(s) "set([u'-'])" in group name (seed-hypervisor)
Not replacing invalid character(s) "set([u'-'])" in group name (baremetal-compute)
Not replacing invalid character(s) "set([u'-'])" in group name (mgmt-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (ctl-switches)
Not replacing invalid character(s) "set([u'-'])" in group name (hs-switches)
Parsed /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/hosts inventory source with ini plugin
setting up inventory plugins
host_list declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/overcloud as it did not pass it's verify_file() method
script declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/overcloud as it did not pass it's verify_file() method
auto declined parsing /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/overcloud as it did not pass it's verify_file() method
Parsed /home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory/overcloud inventory source with ini plugin
[WARNING]: Found both group and host with same name: seed

[WARNING]: Found both group and host with same name: seed-hypervisor

Loading callback plugin default of type stdout, v2.0 from /home/centos/kayobe-venv/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: kolla-bifrost-hostvars.yml ***********************************************************************************Positional arguments: /home/centos/kayobe-venv/share/kayobe/ansible/kolla-bifrost-hostvars.yml /home/centos/kayobe-venv/share/kayobe/ansible/overcloud-hardware-inspect.yml
become_method: sudo
inventory: (u'/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inventory',)
forks: 5
tags: (u'all',)
extra_vars: (u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/bifrost.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/bmc.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ceph.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/compute.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/controllers.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/dell-switch-bmp.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/dns.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/docker-registry.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/docker.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/globals.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/grafana.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/idrac.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/inspector.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ipa.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ironic.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/kolla.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/monitoring.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/network-allocation.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/networks.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/neutron.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/nova.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ntp.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/opensm.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/openstack.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/overcloud.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/pip.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/seed-hypervisor.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/seed-vm.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/seed.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/ssh.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/storage.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/swift.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/users.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/yum-cron.yml', u'@/home/centos/kayobe/config/src/kayobe-config/etc/kayobe/yum.yml')
verbosity: 4
connection: smart
timeout: 10
1 plays in /home/centos/kayobe-venv/share/kayobe/ansible/kolla-bifrost-hostvars.yml

PLAY [Ensure the Bifrost overcloud inventory is populated] *************************************************************META: ran handlers

TASK [Ensure the Bifrost host variables directory exists] **************************************************************task path: /home/centos/kayobe-venv/share/kayobe/ansible/kolla-bifrost-hostvars.yml:29
<192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
<192.168.33.5> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stack"' -o ConnectTimeout=10 -o ControlPath=/home/centos/.ansible/cp/35beaf5ddd 192.168.33.5 '/bin/sh -c '"'"'echo ~stack && sleep 0'"'"''

<192.168.33.5> (0, '/home/stack\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for \r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/home/centos/.ansible/cp/35beaf5ddd" does not exist\r\ndebug2: resolving "192.168.33.5" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to 192.168.33.5 [192.168.33.5] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9998 ms remain after connect\r\ndebug1: identity file /home/centos/.ssh/id_rsa type 1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_dsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_dsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_ecdsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_ecdsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_ed25519 type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/centos/.ssh/id_ed25519-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_7.4\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_7.4\r\ndebug1: match: OpenSSH_7.4 pat OpenSSH compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 192.168.33.5:22 as 'stack'\r\ndebug3: hostkeys_foreach: reading file "/home/centos/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type ECDSA in file /home/centos/.ssh/known_hosts:11\r\ndebug3: record_hostkey: found key type RSA in file /home/centos/.ssh/known_hosts:12\r\ndebug3: record_hostkey: found key type ED25519 in file /home/centos/.ssh/known_hosts:13\r\ndebug3: load_hostkeys: loaded 3 keys from 192.168.33.5\r\ndebug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa\r\ndebug3: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c\r\ndebug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss-cert-v01@openssh.com,ssh-dss\r\ndebug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc\r\ndebug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc\r\ndebug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: compression ctos: zlib@openssh.com,zlib,none\r\ndebug2: compression stoc: zlib@openssh.com,zlib,none\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519\r\ndebug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc\r\ndebug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc\r\ndebug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: compression ctos: none,zlib@openssh.com\r\ndebug2: compression stoc: none,zlib@openssh.com\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug1: kex: algorithm: curve25519-sha256\r\ndebug1: kex: host key algorithm: ecdsa-sha2-nistp256\r\ndebug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: zlib@openssh.com\r\ndebug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: zlib@openssh.com\r\ndebug1: kex: curve25519-sha256 need=64 dh_need=64\r\ndebug1: kex: curve25519-sha256 need=64 dh_need=64\r\ndebug3: send packet: type 30\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1: Server host key: ecdsa-sha2-nistp256 SHA256:Law9MRHgw1XlfbIx463QCtnqr2KcZ7KQqNkM+gbB3oc\r\ndebug3: hostkeys_foreach: reading file "/home/centos/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type ECDSA in file /home/centos/.ssh/known_hosts:11\r\ndebug3: record_hostkey: found key type RSA in file /home/centos/.ssh/known_hosts:12\r\ndebug3: record_hostkey: found key type ED25519 in file /home/centos/.ssh/known_hosts:13\r\ndebug3: load_hostkeys: loaded 3 keys from 192.168.33.5\r\ndebug1: Host '192.168.33.5' is known and matches the ECDSA host key.\r\ndebug1: Found key in /home/centos/.ssh/known_hosts:11\r\ndebug3: send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey after 134217728 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug3: receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1: rekey after 134217728 blocks\r\ndebug2: key: /home/centos/.ssh/id_rsa (0x55a30597b470)\r\ndebug2: key: /home/centos/.ssh/id_dsa ((nil))\r\ndebug2: key: /home/centos/.ssh/id_ecdsa ((nil))\r\ndebug2: key: /home/centos/.ssh/id_ed25519 ((nil))\r\ndebug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>\r\ndebug3: receive packet: type 6\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic\r\ndebug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup gssapi-with-mic\r\ndebug3: remaining preferred: gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled gssapi-with-mic\r\ndebug1: Next authentication method: gssapi-with-mic\r\ndebug1: Unspecified GSS failure. Minor code may provide more information\nNo Kerberos credentials available (default cache: KEYRING:persistent:1000)\n\r\ndebug1: Unspecified GSS failure. Minor code may provide more information\nNo Kerberos credentials available (default cache: KEYRING:persistent:1000)\n\r\ndebug2: we did not send a packet, disable method\r\ndebug3: authmethod_lookup gssapi-keyex\r\ndebug3: remaining preferred: hostbased,publickey\r\ndebug3: authmethod_is_enabled gssapi-keyex\r\ndebug1: Next authentication method: gssapi-keyex\r\ndebug1: No valid Key exchange context\r\ndebug2: we did not send a packet, disable method\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: /home/centos/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 60\r\ndebug1: Server accepts key: pkalg rsa-sha2-512 blen 279\r\ndebug2: input_userauth_pk_ok: fp SHA256:sqJ2TYjSf2MvrkYnJHQcJ6MGh47wD4rQ9h/JT5yWzeU\r\ndebug3: sign_and_send_pubkey: RSA SHA256:sqJ2TYjSf2MvrkYnJHQcJ6MGh47wD4rQ9h/JT5yWzeU\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 52\r\ndebug1: Enabling compression at level 6.\r\ndebug1: Authentication succeeded (publickey).\r\nAuthenticated to 192.168.33.5 ([192.168.33.5]:22).\r\ndebug1: setting up multiplex master socket\r\ndebug3: muxserver_listen: temporary control path /home/centos/.ansible/cp/35beaf5ddd.ojUOwIBJKqSiOd8t\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug3: fd 4 is O_NONBLOCK\r\ndebug3: fd 4 is O_NONBLOCK\r\ndebug1: channel 0: new [/home/centos/.ansible/cp/35beaf5ddd]\r\ndebug3: muxserver_listen: mux listener channel 0 fd 4\r\ndebug2: fd 3 setting TCP_NODELAY\r\ndebug3: ssh_packet_set_tos: set IP_TOS 0x08\r\ndebug1: control_persist_detach: backgrounding master process\r\ndebug2: control_persist_detach: background process is 4610\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug1: forking to background\r\ndebug1: Entering interactive session.\r\ndebug1: pledge: id\r\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\r\ndebug1: multiplexing control connection\r\ndebug2: fd 5 setting O_NONBLOCK\r\ndebug3: fd 5 is O_NONBLOCK\r\ndebug1: channel 1: new [mux-control]\r\ndebug3: channel_post_mux_listener: new mux channel 1 fd 5\r\ndebug3: mux_master_read_cb: channel 1: hello sent\r\ndebug2: set_control_persist_exit_time: cancel scheduled exit\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4\r\ndebug2: process_mux_master_hello: channel 1 slave version 4\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4\r\ndebug2: process_mux_alive_check: channel 1: alive check\r\ndebug3: mux_client_request_alive: done pid = 4612\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 105\r\ndebug2: process_mux_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "xterm-256color", cmd "/bin/sh -c 'echo ~stack && sleep 0'", env 1\r\ndebug3: process_mux_new_session: got fds stdin 6, stdout 7, stderr 8\r\ndebug2: fd 7 setting O_NONBLOCK\r\ndebug2: fd 8 setting O_NONBLOCK\r\ndebug1: channel 2: new [client-session]\r\ndebug2: process_mux_new_session: channel_new: 2 linked to control channel 1\r\ndebug2: channel 2: send open\r\ndebug3: send packet: type 90\r\ndebug3: receive packet: type 80\r\ndebug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0\r\ndebug3: receive packet: type 91\r\ndebug2: callback start\r\ndebug2: client_session2_setup: id 2\r\ndebug1: Sending environment.\r\ndebug1: Sending env LANG = en_US.UTF-8\r\ndebug2: channel 2: request env confirm 0\r\ndebug3: send packet: type 98\r\ndebug1: Sending command: /bin/sh -c 'echo ~stack && sleep 0'\r\ndebug2: channel 2: request exec confirm 1\r\ndebug3: send packet: type 98\r\ndebug3: mux_session_confirm: sending success reply\r\ndebug2: callback done\r\ndebug2: channel 2: open confirm rwindow 0 rmax 32768\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: client_check_window_change: changed\r\ndebug2: client_check_window_change: changed\r\ndebug2: client_check_window_change: changed\r\ndebug2: channel 2: rcvd adjust 2097152\r\ndebug3: receive packet: type 99\r\ndebug2: channel_input_status_confirm: type 99 id 2\r\ndebug2: exec request accepted on channel 2\r\ndebug3: receive packet: type 98\r\ndebug1: client_input_channel_req: channel 2 rtype exit-status reply 0\r\ndebug3: mux_exit_message: channel 2: exit message, exitval 0\r\ndebug3: receive packet: type 98\r\ndebug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0\r\ndebug2: channel 2: rcvd eow\r\ndebug2: channel 2: close_read\r\ndebug2: channel 2: input open -> closed\r\ndebug3: receive packet: type 96\r\ndebug2: channel 2: rcvd eof\r\ndebug2: channel 2: output open -> drain\r\ndebug2: channel 2: obuf empty\r\ndebug2: channel 2: close_write\r\ndebug2: channel 2: output drain -> closed\r\ndebug3: receive packet: type 97\r\ndebug2: channel 2: rcvd close\r\ndebug3: channel 2: will not send data after close\r\ndebug2: channel 2: send close\r\ndebug3: send packet: type 97\r\ndebug2: channel 2: is dead\r\ndebug2: channel 2: gc: notify user\r\ndebug3: mux_master_session_cleanup_cb: entering for channel 2\r\ndebug2: channel 1: rcvd close\r\ndebug2: channel 1: output open -> drain\r\ndebug2: channel 1: close_read\r\ndebug2: channel 1: input open -> closed\r\ndebug2: channel 2: gc: user detached\r\ndebug2: channel 2: is dead\r\ndebug2: channel 2: garbage collecting\r\ndebug1: channel 2: free: client-session, nchannels 3\r\ndebug3: channel 2: status: The following connections are open:\r\n #1 mux-control (t16 r-1 i3/0 o1/16 fd 5/5 cc -1)\r\n #2 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)\r\n\r\ndebug2: channel 1: obuf empty\r\ndebug2: channel 1: close_write\r\ndebug2: channel 1: output drain -> closed\r\ndebug2: channel 1: is dead (local)\r\ndebug2: channel 1: gc: notify user\r\ndebug3: mux_master_control_cleanup_cb: entering for channel 1\r\ndebug2: channel 1: gc: user detached\r\ndebug2: channel 1: is dead (local)\r\ndebug2: channel 1: garbage collecting\r\ndebug1: channel 1: free: mux-control, nchannels 2\r\ndebug3: channel 1: status: The following connections are open:\r\n #1 mux-control (t16 r-1 i3/0 o3/0 fd 5/5 cc -1)\r\n\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\r\ndebug2: Received exit status from master 0\r\n')
<192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
<192.168.33.5> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stack"' -o ConnectTimeout=10 -o ControlPath=/home/centos/.ansible/cp/35beaf5ddd 192.168.33.5 '/bin/sh -c '"'"'( umask 77 && mkdir -p "echo /home/stack/.ansible/tmp"&& mkdir /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836 && echo ansible-tmp-1597518342.25-4549-187346835488836="echo /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836" ) && sleep 0'"'"''
<192.168.33.5> (0, 'ansible-tmp-1597518342.25-4549-187346835488836=/home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 4612\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /home/centos/kayobe-venv/lib/python2.7/site-packages/ansible/modules/files/file.py
<192.168.33.5> PUT /home/centos/.ansible/tmp/ansible-local-4535g2t2MZ/tmp4Bha6u TO /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836/AnsiballZ_file.py
<192.168.33.5> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stack"' -o ConnectTimeout=10 -o ControlPath=/home/centos/.ansible/cp/35beaf5ddd '[192.168.33.5]'
<192.168.33.5> (0, 'sftp> put /home/centos/.ansible/tmp/ansible-local-4535g2t2MZ/tmp4Bha6u /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836/AnsiballZ_file.py\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 4612\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/stack size 0\r\ndebug3: Looking up /home/centos/.ansible/tmp/ansible-local-4535g2t2MZ/tmp4Bha6u\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn't stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836/AnsiballZ_file.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:13384\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 13384 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
<192.168.33.5> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stack"' -o ConnectTimeout=10 -o ControlPath=/home/centos/.ansible/cp/35beaf5ddd 192.168.33.5 '/bin/sh -c '"'"'chmod u+x /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836/ /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836/AnsiballZ_file.py && sleep 0'"'"''

<192.168.33.5> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 4612\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.33.5> ESTABLISH SSH CONNECTION FOR USER: stack
<192.168.33.5> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="stack"' -o ConnectTimeout=10 -o ControlPath=/home/centos/.ansible/cp/35beaf5ddd -tt 192.168.33.5 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-xfwttybrjqfytpbwgkpsbjbjfbhcuxlz ; /opt/kayobe/venvs/kayobe/bin/python /home/stack/.ansible/tmp/ansible-tmp-1597518342.25-4549-187346835488836/AnsiballZ_file.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
fatal: [controller0]: FAILED! => {
"msg": "Timeout (12s) waiting for privilege escalation prompt: "
}

NO MORE HOSTS LEFT *****************************************************************************************************

PLAY RECAP *************************************************************************************************************
controller0 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[markgoddard]

Hi. It looks like you are unable to access the seed machine via SSH, or possibly having problems with sudo. Have you already deployed the seed? Could you try the following:

ssh stack@192.168.33.5 sudo echo hi

Sometimes we see that error on slower hardware. You could try setting export ANSIBLE_TIMEOUT=30.

[karimbzu]

Thank you for your response, i have resolved the above mentioned issue with your help i.e export ANSIBLE_TIMEOUT=30 However, still i am struggling to complete this project. I am using centos 7.7 VM with 128GB RAM and 500 GB storage, during ./dev/seed-deploy.sh command the process hangs at *TASK [baremetal] synchronizing time one-time. *When i ssh centos@192.168.33.5, I saw ntpd service is failed to start. I tried $sudo kill -HUP {process id of ntpd| and the process resumes again. The seed successfully deployed after that, however later these commands: kayobe overcloud hardware inspect, kayobe overcloud provision were not successful causing the 3 VMs in pause state here is the error screenshots: [image: image.png] [image: image.png] Thanks and really appreciate your assistance in this regard.

…

On Mon, Aug 17, 2020 at 12:54 PM Mark Goddard ***@***.***> wrote: Hi. It looks like you are unable to access the seed machine via SSH, or possibly having problems with sudo. Have you already deployed the seed? Could you try the following: ssh ***@***.*** sudo echo hi Sometimes we see that error on slower hardware. You could try setting export ANSIBLE_TIMEOUT=30. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#48 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIHBQKTNNBGMBD46VK6XRBTSBDO5FANCNFSM4QANLTSA> .

[karimbzu]

[markgoddard]

Which branch of the a-universe-from-nothing repo are you using?

I'm not sure what the NTP problem is. Does the seed have access to the configured NTP servers?

Console logs for the overcloud hosts are in /var/log/tenks. These might help to diagnose any boot failures.

[priteau]

I'm not sure what the NTP problem is. Does the seed have access to the configured NTP servers?

I've seen the same NTP issue in several other environments. I assume this is caused by firewall restrictions.

[karimbzu]

Previously, I was using stable/train branch. Now i am testing stable/ussuri branch. I will update you if the issues resolved. Currently on stable/train the state of each vm is "paused".

Another thing i want to ask as a newbie in openstack (only experienced with VIO), i have enough storage and compute resources available on a vsphere datacenter. I need to ask can i deploy a production level HA openstack hosting maybe 200 to 300 VMs with kayobe? If so, how can i replicate the whole infrastructure i.e 3 controllers, 3 compute and 3 storage nodes with HA supported? Is kayobe supports NSX-T networks? I shall really appreciate your kind response.

[markgoddard]

For Ussuri you will need a CentOS 8 host.

I haven't seen the VMs go paused like that.

Kayobe does support HA deployment. You can include multiple hosts in each group in your inventory. NSX-T does claim to integrate with OpenStack (https://docs.vmware.com/en/VMware-Integrated-OpenStack/5.1/com.vmware.openstack.install.doc/GUID-D9BD61E1-E444-441E-90E7-4921A5014CA3.html), although I have not tried it. Typically people use Open vSwitch or Linux bridge as the networking driver.

Are you suggesting to run 200-300 VMs on 3 compute nodes?

[karimbzu]

Thanks for your valuable insights
Actually i am not sure, how much workload a compute node can accommodate. Therefore, i am asking you about the best use case scenario in my requirements (200 to 300 VMs). I am well aware of PAAS (Openshift) which can supports 250 pods per compute node, so we can add compute nodes if it is needed. Obviously this type of restriction will also be applied to Openstack. Further i also need to set storage nodes (block, object) how can i do this from your workshop guide? can i use your workshop as a baseline to setup production grade clusters? Thanks once again

[markgoddard]

VMs have a larger overhead than containers, so while you might be able to squeeze many tens of them onto a compute node, they will probably not be very efficient. It really depends on what you are using them for.

Typically we use Ceph storage with OpenStack. Up to the Nautilus release we typically use ceph-ansible to deploy it. Then follow this guide to connect it to OpenStack. From the latest Ceph Octopus release you can use the new Ceph Orchestrator to deploy it.

This workshop is really designed to be a tool to learn about using Kayobe. To create a production cluster, you should refer to the kayobe documentation for more information. You can refer back to this set of configuration for a minimal example. What the documentation lacks at the moment however is a complete end-to-end deployment and configuration guide. You can join the #openstack-kolla IRC channel on freenode if you have questions.

[rplazevedo]

Was the pause issue ever fixes? The same thing is occurring for me?