stf_collect_logs leaves logs in the git path

[leifmadsen]

When running stf-run-ci with ansible-playbook build/run-ci.yml the result is various logs dumped into the default git path.

Would suggest either changing the default logging directory, adding a new logging directory in the build/ directory and writing there, along with some update to .gitignore so we don't accidentally see logs committed to the repo.

	build/describe_sto.log
	build/post_oc_describe_build_prometheus-webhook-snmp-2.log
	build/post_oc_describe_build_service-telemetry-operator-2.log
	build/post_oc_describe_build_sg-bridge-2.log
	build/post_oc_describe_build_sg-core-2.log
	build/post_oc_describe_build_smart-gateway-operator-2.log
	build/post_oc_describe_pod_prometheus-default-0.log
	build/post_oc_describe_pod_prometheus-webhook-snmp-2-build.log
	build/post_oc_describe_pod_service-telemetry-operator-2-build.log
	build/post_oc_describe_pod_sg-bridge-2-build.log
	build/post_oc_describe_pod_sg-core-2-build.log
	build/post_oc_describe_pod_smart-gateway-operator-2-build.log
	build/post_oc_get_builds.log
	build/post_oc_get_images.log
	build/post_oc_get_subscriptions.log
	build/post_question_deployment.log
	build/sgo_gen_bundle.log
	build/sto_gen_bundle.log
	build/validate_deployment.log

[elfiesmelfie]

The default logging dir can be under build/working, which we create, and hopefully already ignore.
Or the default logging dir (or *.log) can be added to .gitignore

[leifmadsen]

cat .gitignore 
watches.local
.venv
build/working/
bundle.Dockerfile
build/stf-run-ci/filter_plugins/__pycache__/
roles/servicetelemetry/filter_plugins/__pycache__/

I would probably create a new build/working/logs/ directory and store things there by default.

[csibbitt]

Adding things to the build directory (possibly outside of buid/working/?) slows down iterative development given our current layout. What I mean is when I do this (found in our wiki):

oc start-build service-telemetry-operator --wait --from-dir . 

This is currently the fastest way to iterate on operator code (actually a fully local build where the local operator connects to the cluster API is faster, but quite different approach). The speed of it is affected (sometimes dominated) by how long it takes to upload all the files from the local directory into the cluster - now including all the logs from your last round of testing, multiple copies of the operator-sdk, and checkouts of other projects in build/working, etc. Fine if your lab machine is in the same datacenter as your cluster, but otherwise it can be long. I usually work around the biggest offenders by deleting build/working/, so agree that putting all the logs under there would be helpful.

I also noted a very large must-gather.local directory in my workspace when I started troubleshooting my slow build. Is this something that was recently added (I'm having trouble putting my finger on it if so) or a case of me shooting myself in the foot with something with something else I did? :)

All this to say that it's not just .gitignore that is required here, we want to consider how we can implement this in a way that doesn't make development builds more painful.