(enhancement): gatecheck - Coredump/Crash related enhancements

[Manimaran-MM]

Changes

• enable coredump for both ceph and gpfs workflow
• collect coredump in case of failure
• export coredump as artifacts for better visibility
• parallel thread on observing ganesha pid

Logs

• Run with crash Logs: https://jenkins-nfs-ganesha.apps.ocp.cloud.ci.centos.org/view/WIP%20(Test%20only)/job/test-mani-1/16/
• Run without crash Logs: https://jenkins-nfs-ganesha.apps.ocp.cloud.ci.centos.org/view/WIP%20(Test%20only)/job/test-mani-1/14/

[kalebskeithley]

lgtm

[diederichm]

I wonder if we need to preserve more data (maybe its there and I do not see it)
if there is a crash -
we would want the coredump, meta data for the coredump -
but also
the ganesha binary/build along with debuginfo and debugsource rpms - so that an analysis outside of the CI environment can be performed..

[Manimaran-MM]

I wonder if we need to preserve more data (maybe its there and I do not see it) if there is a crash - we would want the coredump, meta data for the coredump - but also the ganesha binary/build along with debuginfo and debugsource rpms - so that an analysis outside of the CI environment can be performed..

@diederichm This is one such job where we capture backtrace after enabling debuginfo with the same set of rpms and capturing details of "bt full" and display it as artifacts for the individual tests

https://jenkins-nfs-ganesha.apps.ocp.cloud.ci.centos.org/view/Gatecheck%20Tests/job/gatecheck-trigger-gerrithub/861/
(Please check on the artifacts that ends with "*_backtrace.txt")

I believe this should be helpful to start with initial triage... And since this crash will be specific to the individual patchset changes(which is not merged to next branch yet), we may not require to preserve the coredump/binaries to identify the root cause 🙂

[diederichm]

right - I understand we will have the backtrace with symbols.
That is certainly better then to not have it.
Not all analysis can be done from backtrace alone - it is sometimes helpful to understand the state of Ganesha (ie mdcache content, idmap cache content) - and that can be inspected if we have the core, the binaries and deuginfo/source of the build.

[Manimaran-MM]

Noted.. Thanks for the feedback @diederichm .. We will review and see how we can accomodate the same on gatecheck