Add periodic updates to file transfer plugin output/error

[brianhlin]

Pelican Service:

• Client
• Plugin
• Registry
• Director
• Origin
• Cache
• Other (please give the detail)

Is your feature request related to a problem? Please describe.
We are troubleshooting potentially hung input file transfer Pelican processes in the CHTC and all the logging is eerily quiet. It's unclear whether the Pelican process is hung or transfers are just very slow.

Describe the solution you'd like

1. Heartbeat messages dumped to stderr. On the order of every 5-10 min based on the HTCSS dev team recommendation
2. Flushing messages to the -outfile upon completion of each file

[brianaydemir]

I wonder if d6fae18 affects the file transfer plugin.

[bbockelm]

Note -- deferred this to v7.20 because it's going to require HTCSS-side changes to be of any use.

In the meantime, d6fae18 will allow anyone to see partial progress if the plugin's stdout is being streamed to the starter logs (unfortunately, I think this may not be true in current releases?).

[brianhlin]

AIUI, HTCondor doesn't stream FT plugin stdout/stderr into the StarterLog. It's all stored in a buffer that gets flushed, I think upon FT plugin exit. The message-per-minute in d6fae18 may actually fill up said buffer so we may want to turn the frequency down or turn this feature off completely in the plugin case.