Prefect flow runs lose automatic `opentelemetry-instrument` configuration

[chrisguidry]

Bug summary

For users who want to use OpenTelemetry's automatic instrumentation so that all of their flow runs export OTEL traces and metrics without requiring in-code manual instrumentation setup with, they would configure their work pools to use the command opentelemetry-instrument prefect flow-run execute. This is the standard approach for Python automatic instrumentation, but it has a known limitation that it doesn't carry through to child processes started by spawn (it works with some caveats with fork).

Since the prefect flow-run execute command starts a Runner that spawns a subprocess, any automatic instrumentation that was setup in the parent process's interpreter isn't carried through. So while the OTEL environment variables will be there, the actual setup of the providers and exporters won't happen.

Version info

3.6.11 (all versions)

Additional context

https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html
open-telemetry/opentelemetry-python#2765
open-telemetry/opentelemetry-python-contrib#385

In particular there's a possible workaround here that may help:

open-telemetry/opentelemetry-python-contrib#385 (comment)

import opentelemetry.instrumentation.auto_instrumentation.sitecustomize

I believe this would end up triggering the same full setup that opentelemetry-instrument uses, which would read envvars and scan site-packages to look for instrumentation libraries, etc. We wouldn't want to automatically do this for users, because that may not actually be what they want.

Some ideas:

• Detect any wrapper scripts that are wrapping prefect flow-run execute on the command line and use those for spawning the subprocesses. So if we see the command line being some-wrapper another-wrapper ddtrace opentelemetry-instrument prefect execute flow-run we use some-wrapper another-wrapper ddtrace opentelemetry-instrument python -m prefect.engine as the subprocess.
• Do that ^ but just to detect opentelemetry-instrument specifically
• Create a user setting for a default entrypoint besides get_sys_executable()