Question - how to stream shell pipes into helix?

[varon]

Is there any way to stream shell output instead of buffering the entire thing?
Say I wanted to run the command:
| tail -f /var/syslog - it won't terminate.

Is there a mechanism to do this via streaming until user cancels, adding the output as it arrives?

[the-mikedavis]

It isn't possible to interrupt the pipe for | tail currently. IMO streaming functionality as you say is a pager's responsibility rather than an editor. If we implement a way to cancel pipes, I think that the pipe should be like a transaction and no text should be inserted.

[varon]

Thank you for starting this discussion.

There are a number of use cases where we simply can't approach things transactionally, or rather it serves to be highly advantageous to stream information as it arrives.

Imagine a long-running computation that is gradually returning results. Those results are incrementally useful, and the decision to continue the computation depends on the results themselves.

Perhaps a more topical example would be streaming in text via one of the OpenAI APIs such as GPT. Without streaming the interaction breaks down horribly to a very boring, flow-breaking waiting process.

[varon]

This might also help with #1927