[WIP][DSIP-99][TaskPlugin] Save task output to a separate file #17791
Description
Search before asking
- I had searched in the DSIP and found no similar DSIP.
Motivation
Currently, we store the task log and task output in the same file.
The problem is that the task log is generated by DolphinScheduler, the end users don't actually understand the underlying logic. They are more care about the output of their tasks.
Therefore, they need to locate their task outputs within the vast array of task instances.
Additionally, since we often use separate threads to collect task outputs, this can also lead to issues where task outputs and task log entries become interspersed and out of order.
This DSIP aims to separate the task log and task output and use an extra file to store the task output.
Design Detail
todo:
Compatibility, Deprecation, and Migration Plan
todo:
Test Plan
todo:
Code of Conduct
- I agree to follow this project's Code of Conduct