[Bug]: compass did not parse to completion dolphinscheduler log #240
Description
Contact Details
No response
Is there an existing issue for this?
- I have searched all the existing issues
Priority
high
Current Behavior
I used ds2hdfs.py provided by the author to upload logs.
It is currently found that compass is inaccurate when reading kafak data to parse the dolphinscheduler log uploaded to hdfs by flume. When the log generates multiple hdfs files, sometimes the last file is not read.
After investigation, it was found that the last tmp file was not generated when compass parsed the log. The delay in flume generating files caused the last file to be missing when parsing the log.
Expected Behavior
No response
Steps To Reproduce
No response
Compass Log
No response
Anything else? (Additional Context)
No response