Controlling installation result of osde2e with exit code is not enough #73
Description
Installing with a wrong ocm token results on thread exiting with exit code 0
2021-03-10 10:51:26.081 INFO osde2e-wrapper - main: Attempting to start 10 clusters with 2 batch size
2021-03-10 10:51:26.089 INFO osde2e-wrapper - _build_cluster: Started cluster osde2e-o3z-0001 (1 of 10)
2021-03-10 10:51:26.092 INFO osde2e-wrapper - _build_cluster: Started cluster osde2e-o3z-0002 (2 of 10)
2021-03-10 10:51:26.544 INFO osde2e-wrapper - _watcher: Requested Clusters for test 22b46690-9244-4a5d-935e-3c48f27a6338: 0
2021-03-10 10:51:31.888 INFO osde2e-wrapper - _build_cluster: Attempting to load metadata json
2021-03-10 10:51:31.888 ERROR osde2e-wrapper - _build_cluster: [Errno 2] No such file or directory: '/tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0002/metadata.json'
2021-03-10 10:51:31.889 ERROR osde2e-wrapper - _build_cluster: Failed to load metadata.json file located /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0002
2021-03-10 10:51:31.889 INFO osde2e-wrapper - _download_kubeconfig: Attempting to load metadata json
2021-03-10 10:51:31.890 ERROR osde2e-wrapper - _download_kubeconfig: Failed to load cluster-id from metadata.json file located on /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0002, kubeconfig file wont be downloaded
Installation logs:
2021/03/10 10:51:26 addon timeout is 30
2021/03/10 10:51:26 load.go:57: Custom YAML config provided, loading from cluster_account.yaml
2021/03/10 10:51:26 e2e.go:260: Could not create reporter directory: mkdir /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0001: file exists
2021/03/10 10:51:26 e2e.go:303: Outputting log to build log at /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0001/test_output.log
2021/03/10 10:51:26 versions.go:34: Querying cluster versions endpoint.
2021/03/10 10:51:26 go_logger.go:152: OCM auth: failed to get tokens, got http code 400, will not attempt to retry. err: invalid_grant: Offline user session not found
2021/03/10 10:51:26 retryer.go:23: error during OCM attempt: can't get access token: invalid_grant: Offline user session not found
2021/03/10 10:51:30 go_logger.go:152: OCM auth: failed to get tokens, got http code 400, will not attempt to retry. err: invalid_grant: Offline user session not found
2021/03/10 10:51:30 retryer.go:23: error during OCM attempt: can't get access token: invalid_grant: Offline user session not found
2021/03/10 10:51:31 go_logger.go:152: OCM auth: failed to get tokens, got http code 400, will not attempt to retry. err: invalid_grant: Offline user session not found
2021/03/10 10:51:31 retryer.go:23: error during OCM attempt: can't get access token: invalid_grant: Offline user session not found
This conditional is not being executed, as we dont see the "Failed to build cluster" logging.
We need to fix the failed cluster installation detection but also assigning a default cluster-id as "not generated" or something if we detect that metadata.json file is not generated or generated without cluster-id