Waiting for job output - no live job output stdout streaming?

[arminus]

I have an event with a shell script calling a python script which prints to stdout, first print right when the script starts, then as it progresses. I've also set the "Data Passthrough" flag set. However, all I see in the Live Job Output is "Waiting for job output...", the full log only appears once the entire script has finished.

[jhuckaby]

Ah, this is a very common issue! I really should just write a wiki article about this... 😝

Many languages, Python included, will BUFFER standard output until either the buffer entirely fills up, or the process exits. You will need to bypass stdout buffering in your script for things like live progress updates. There are a number of ways to do this with Python:

Flush in code (most common)

Explicit flush:

print("Hello", flush=True)

Manual flush:

import sys

print("Hello")
sys.stdout.flush()

Disable buffering for the entire process (CLI)

Enable -u (unbuffered mode):

python -u script.py

Control buffering via environment variables

PYTHONUNBUFFERED=1 python script.py

Change buffering behavior in code (process-wide)

Reconfigure stdout (Python 3.7+):

import sys

sys.stdout.reconfigure(line_buffering=True)

or fully unbuffered:

sys.stdout.reconfigure(write_through=True)

Hope this helps!

[arminus]

My bad, somewhere from the dusty attic of my brain I could have remembered this :-/ Thanks for the thorough explanation!