Logging on ephemeral runners

[shankarRaman]

Hello,
Hoping someone has encountered a similar problem and can provide some direction. When using ephemeral runners, how does one persist the diagnostic logs so that it can be used for troubleshooting? We are encountering a weird behavior which is not 100% reproducible and we want to look in to the logs to do RCA. However the runner pod is destroyed and there's no way we can find out what caused this.
So the question is how does one solve this problem?

[kevholmes]

@shankarRaman could you add a sleep 99999 step or similar to your workflow experiencing the issue, and then drop into the pod/container with a shell, or use the newer kubectl debug -it ... cmd and debug from there?

[shankarRaman]

Thanks @kevholmes
As a follow up I am consistently seeing these messages
An error occurred: Access denied. System:ServiceIdentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD needs View permissions to perform the action.
Runner listener exit with retryable error, re-launch runner in 5 seconds.
Failed to create a session. The runner registration has been deleted from the server, please re-configure.
Runner listener exit with terminated error, stop the service, no retry needed.

What would cause this behavior? This happens when a job fails and a job succeeds and I'd like to understand where these errors are coming from.