State being maintained between calls to python

[cgubbin]

I am trying to execute a Tensorflow Transform pipeline implemented in Apache Beam from Rust. If I call the algorithm once everything works as expected, however if I try to call it a second time without restarting the Rust program an exception is returned.

I fixed the issue in the end by reloading both the logging and apache beam modules at the beginning of my Python entrypoint.

import apache_beam
from importlib import reload
import logging

fn run():
    reload(logging)
    reload(apache_beam)
    ...
 

It seems that some state from the first call to python is being maintained, which is then seen by any subsequent call. Is this a bug or a feature? If the latter is there a way to clear the state from rust, without having to reload individual modules?

[davidhewitt]

@cgubbin can you share some details on the exceptions which are returned? Without more information we will struggle to offer insight.

[cgubbin]

Before I reloaded the logging module it returned a recursion error, from a call to 'basicConfig()' made by Apache Beam. After resetting the logger Apache Beam returned a broken pipe error [32].

[davidhewitt]

It sounds like this is an error in Apache Beam's logging integration. You might want to report an issue with them and share a repro.