How to use the subprocess processor to invoke a script for each message

[mihaitodor]

UPDATE: A new command processor has been introduced in v4.21.0 which can be used instead of this workaround. The instructions below can still be leveraged if you need to do something like this for the subprocess input or output.

While the subprocess processor is meant to start long-running commands, in some cases it's desirable to have it call a simple script which starts up, prints its output to stdout and exits for each message which flows through this processor. To achieve this, one can use a wrapper shell script like so:

#!/bin/sh

while read -r line;
do
    your_simple_script "${line}" | base64 | tr -d '\n\r' && echo;
done

You'll also need to configure the subprocess processor along with a downstream mapping processor like so (where wrapper.sh is the wrapper script above):

- subprocess:
    name: ./wrapper.sh
    max_buffer: 65536
    codec_send: lines
    codec_recv: lines
- mapping: |
    root = content().decode("base64")

Note: While encoding the data to base64 isn't mandatory, if your script emits any newline characters, then those have to be escaped somehow and this is a simple way of doing so. The base64 command behaves somewhat differently on various platforms and the above script is meant to be platform-independent. However, the | base64 | tr -d '\n\r' && echo part can be replaced with | base64 on OSX and | base64 -w 0 && echo on Linux.

[sharkfin009]

Very helpful, thanks Mihai!