How do you log panics and stdout/stderr?

[jayd3e]

Thank you for putting this package together. I think it's a really valuable addition to the ecosystem's support of OpenTelemetry. One gap, however, is that there is no way to easily redirect stdout/stderr to the Otel collector as well. One vital part of maintaining a production application is seeing any unintended errors or panics that occur.

Third party libraries also often use fmt.Println, etc which prints to stderr. This is resulting in my application sending nicely formatted logs to the collector, but then occasionally key details on an unexpected error get lost. In the case of a panic, it is easy enough to recover(); however, not having third party logs get sent to my logging platform has the potentialy to create confusion. Any ideas on how this will likely be handled by the Go community and this library?

We are currently doing something like this:

// logging package
func RedirectStdOut(writer io.Writer) func() {
	old := os.Stdout

	r, w, err := os.Pipe()
	if err != nil {
		panic(err)
	}

	os.Stdout = w

	go func() {
		if _, err := io.Copy(writer, r); err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			panic(err)
		}
	}()

	return func() {
		w.Close()
		os.Stdout = old
	}
}

func RedirectStdErr(writer io.Writer) func() {
	old := os.Stderr

	r, w, err := os.Pipe()
	if err != nil {
		panic(err)
	}

	os.Stderr = w

	go func() {
		if _, err := io.Copy(writer, r); err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			panic(err)
		}
	}()

	return func() {
		w.Close()
		os.Stderr = old
	}
}

// When instantiating the logger
func redirectStdLog() func() {
	// redirect std out and std err to the new slog logger
	sw := logging.NewSlogWriter(slog.Default())

	restoreErr := logging.RedirectStdErr(sw)
	restoreOut := logging.RedirectStdOut(sw)

	return func() {
		sw.Close()
		restoreErr()
		restoreOut()

		if otelShutdown != nil {
			err := otelShutdown(context.Background())
			if err != nil {
				panic(err)
			}
		}
	}
}

We have also thought about wrapping the execution of our application in another Go program that reads all of the stdout/stdin from the running application and pipes it to OpenTelemetry. Any thoughts?

[chameleon82]

Thanks for the catch. It definitely can improve panic.

otelShutdown will shutdown exporters so no logs can be sent after and panic can cause last batch will not be sent as otelShutdown wasn't called - this should be considered.

I'm not sure about best practice here, but it will be nice if you can create MR we can discuss options