[Feature Request][Go SDK]: Rewrite the `dot` runner in terms of a Portable pipeline.

[lostluck]

What would you like to happen?

The Go SDK's dot runner current generates a pipeline graph based on the Go SDK internal structures.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/util/dot/dot.go#L89

If it were rewritten in terms of the portable pipeline proto (https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L91) then it could be used to render Cross Language pipelines and be used as a basis for the stand alone Prism Runner's web UI.

That would sufficiently allow #25896 to be closed.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[mls3odp]

.take-issue

[mls3odp]

Does this mean to use Pipeline proto as the input of Execute function in dot.go?

Or modify the Render function to transforming edges and nodes into the Pipeline proto and writing the corresponding information (e.g. Pipeline::Components containing transforms and pcolllections) as texts? CMIIW thanks!

[lostluck]

In this case it means rendering the pipeline structure from the Pipeline Proto instead of from the raw Go structures.

This would let it take any pipeline proto graph and produce a dot graph for it.

So this means building the pipeline proto with the existing code to do so (see how the Universal runner is doing it) and then building the visual graph based on the information only from the proto.

[mls3odp]

Gotcha! So we need to convert beam.Pipeline to Pipeline proto like https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/universal/universal.go#L64-L91 in https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/dot/dot.go#L40.

[mohamedawnallah]

It looks like this issue has been inactive for a while, and the submitted PR was automatically closed due to inactivity. I'd like to take it on! 🙏

[mohamedawnallah]

.take-issue

[mohamedawnallah]

Freeing up my assignment to this good-first-issue so other contributors can jump in if interested! If no one picks it up by then, I'm happy to submit a PR to close it out

[mohamedawnallah]

.free-issue