[Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows

[BI-ErickTrindade]

What would you like to happen?

Hi team,

It would be extremely helpful to have a feature that allows users to "enter" or open a running pipeline or workflow to see its current execution status — similar to how it's possible in Pentaho.

This would make it easier to troubleshoot long-running jobs or monitor processes without having to wait for them to finish or relying only on logs. Ideally, we’d be able to see which step is currently being executed and the progress so far.

Issue Priority

Priority: 3

Issue Component

Component: Hop Gui

[steewel]

+1

[mattcasters]

That's exactly what an execution information location can do for you.

https://hop.apache.org//manual/latest/hop-gui/perspective-execution-information.html

I can recommend sending the execution information to local files, Neo4j or an Elastic server.

[hansva]

.take-issue

[hansva]

I think both can live next to each other, but it has to be bettern then what it was in said previous platform.
We can use live drill down for development and the execution information for history and runs using the CLI or other platforms.

[mattcasters]

Indeed @hansva. The Previous Devepment Instance only tracked things in the GUI and not very well. I think that it should be possible to implement something that links a running sub-pipeline or sub-workflow to the executing transform or action.
You can store this information in IExtensionData.getExtensionDataMap().
The main concern would be leaking memory. We'd probably need options to "forget" completed work ... or not. Visual indicators on the actions and transforms like running, failed or completed icons are already present. Perhaps we can implement click-through options for them.

[hansva]

I've been drafting something that I'll bring forward for discussion. But I am not unhappy with the progress I have made; the main issue I am pondering about is how to integrate it cleanly into the UI.

Currently, I have a dialog popping up where you can select which instance of a loop you want to open and what its current state and parameters were. But it might be cleaner to also implement the Info tab the execution information perspective has.

I would like the result to be something actionable. If you have a loop and it fails on iteration x, we would want to be able to restart a pipeline on that iteration with the state it had to make debugging even more powerful, or at least have great insights on what and why it is going wrong.

Also wondering if we should include/store the data sniffers as it would be great information to have for each iteration but it might blow up memory consumption during development.

[mattcasters]

At first all it needs to be is practical. We can make it beautiful later. If you go down the route of storing the information, you might as well get it from an Execution Information Location.