Skip to content
This repository was archived by the owner on Mar 23, 2020. It is now read-only.

master don't come back online after reboot in 99_post_install - etcd member pod not running #100

@sreichar

Description

@sreichar

[kni@worker-0 logs]$ cat 99_post_install-2019-09-07-190210.log

  • source common.sh
    ++++ dirname common.sh
    +++ cd .
    +++ pwd
    ++ SCRIPTDIR=/home/kni/install-scripts/OpenShift
    +++ whoami
    ++ USER=kni
    ++ '[' -z '' ']'
    ++ '[' -f /home/kni/install-scripts/OpenShift/config_kni.sh ']'
    ++ echo 'Using CONFIG /home/kni/install-scripts/OpenShift/config_kni.sh'
    Using CONFIG /home/kni/install-scripts/OpenShift/config_kni.sh
    ++ CONFIG=/home/kni/install-scripts/OpenShift/config_kni.sh
    ++ source /home/kni/install-scripts/OpenShift/config_kni.sh
    +++ set +x
    +++ export INT_IF=eno2
    +++ INT_IF=eno2
    +++ export PRO_IF=eno1
    +++ PRO_IF=eno1
    ++ export LIBVIRT_DEFAULT_URI=qemu:///system
    ++ LIBVIRT_DEFAULT_URI=qemu:///system
    ++ '[' kni '!=' root -a /run/user/1000 == /run/user/0 ']'
    ++ sudo -n uptime
    +++ awk -F= '/^VERSION_ID=/ { print $2 }' /etc/os-release
    +++ cut -f1 -d.
    +++ tr -d '"'
    ++ VER=8
    +++ tr -d '"'
    +++ awk -F= '/^ID=/ { print $2 }' /etc/os-release
    ++ [[ rhel != \r\h\e\l ]]
    ++ [[ 8 -ne 8 ]]
    ++ '[' 3940 = 0 ']'
  • export KUBECONFIG=ocp/auth/kubeconfig
  • KUBECONFIG=ocp/auth/kubeconfig
  • POSTINSTALL_ASSETS_DIR=./assets/post-install
  • IFCFG_INTERFACE=./assets/post-install/ifcfg-interface.template
  • IFCFG_BRIDGE=./assets/post-install/ifcfg-bridge.template
  • BREXT_FILE=./assets/post-install/99-brext-master.yaml
  • export bridge=brext
  • bridge=brext
  • create_bridge
  • echo 'Deploying Bridge brext...'
    Deploying Bridge brext...
    ++ head -1
    ++ oc get node -o 'custom-columns=IP:.status.addresses[0].address' --no-headers
  • FIRST_MASTER=10.19.1.231
    ++ ssh -q -o StrictHostKeyChecking=no core@10.19.1.231 'ip r | grep default | grep -Po '''(?<=dev )(\S+)''''
  • export interface=eno2
  • interface=eno2
  • '[' eno2 == '' ']'
  • '[' eno2 '!=' brext ']'
  • echo 'Using interface eno2'
    Using interface eno2
    ++ envsubst
    ++ base64 -w0
  • export interface_content=REVWSUNFPWVubzIKQlJJREdFPWJyZXh0Ck9OQk9PVD15ZXMKTk1fQ09OVFJPTExFRD15ZXMKQk9PVFBST1RPPW5vbmUK
  • interface_content=REVWSUNFPWVubzIKQlJJREdFPWJyZXh0Ck9OQk9PVD15ZXMKTk1fQ09OVFJPTExFRD15ZXMKQk9PVFBST1RPPW5vbmUK
    ++ envsubst
    ++ base64 -w0
  • export bridge_content=REVWSUNFPWJyZXh0Ck5BTUU9YnJleHQKVFlQRT1CcmlkZ2UKT05CT09UPXllcwpOTV9DT05UUk9MTEVEPXllcwpCT09UUFJPVE89ZGhjcApCUklER0lOR19PUFRTPXZsYW5fZmlsdGVyaW5nPTEKQlJJREdFX1ZMQU5TPSIxIHB2aWQgdW50YWdnZWQsMjAsMzAwLTQwMCB1bnRhZ2dlZCIK
  • bridge_content=REVWSUNFPWJyZXh0Ck5BTUU9YnJleHQKVFlQRT1CcmlkZ2UKT05CT09UPXllcwpOTV9DT05UUk9MTEVEPXllcwpCT09UUFJPVE89ZGhjcApCUklER0lOR19PUFRTPXZsYW5fZmlsdGVyaW5nPTEKQlJJREdFX1ZMQU5TPSIxIHB2aWQgdW50YWdnZWQsMjAsMzAwLTQwMCB1bnRhZ2dlZCIK
  • envsubst
  • echo 'Done creating bridge definition'
    Done creating bridge definition
  • apply_mc
  • for node_type in master worker
  • oc patch --type=merge '--patch={"spec":{"paused":true}}' machineconfigpool/master
    machineconfigpool.machineconfiguration.openshift.io/master patched
  • for node_type in master worker
  • oc patch --type=merge '--patch={"spec":{"paused":true}}' machineconfigpool/worker
    machineconfigpool.machineconfiguration.openshift.io/worker patched
  • '[' '' '!=' '' ']'
  • for node_type in master worker
    ++ find ./assets/post-install -iname '*-master.yaml' -type f
  • test ./assets/post-install/99-brext-master.yaml
  • echo 'Applying machine configs...'
    Applying machine configs...
  • oc create -f ./assets/post-install/99-brext-master.yaml
    machineconfig.machineconfiguration.openshift.io/99-brext-master created
  • oc patch --type=merge '--patch={"spec":{"paused":false}}' machineconfigpool/master
    machineconfigpool.machineconfiguration.openshift.io/master patched
  • echo 'Rebooting nodes...'
    Rebooting nodes...
  • sleep 30
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Unable to connect to the server: unexpected EOF
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    Error from server (NotFound): the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io master)
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
  • sleep 1
  • oc wait mcp/master --for condition=updated --timeout 600s
    error: timed out waiting for the condition on machineconfigpools/master
    .
    .
    .

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions