What is the inspiration behind this change, or can you please provide more context or use case? I'm thinking about the corner case of a Python class as an argument, which AFAIK isn't JSON-serializable or -deserializable out of the box with the json module.

Can you also add tests to make sure this proposed feature works?

If you want to support this feature you also need to add a flag somewhere around here to serialize Python objects by JSON too, as arguments via "infile" are always serialized with cloudpickle at the moment (see

What is the inspiration behind this change, or can you please provide more context or use case? I'm thinking about the corner case of a Python class as an argument, which AFAIK isn't JSON-serializable or -deserializable out of the box with the json module.

Can you also add tests to make sure this proposed feature works?

This feature enables the design of a new graph executor, where the entire graph is installed within the library. Each function call carries the key that identifies the function to execute, whose format is string. Since the task key is a string, we can pass the text directly instead of serializing it via cloudpickle.

For example, a task graph can be defined as:

def add(x, y):
    return x + y

graph = {
    "a": (add, 1, 2),
    "b": (add, "a", 3),
    "c": (add, "a", "b"),
}

Here, graph is installed in the library, the executor takes a key, retrieves the corresponding task definition, and executes it. Thus, the infile for the three tasks would look like:

{args=("a"), kwargs={}}
{args=("b"), kwargs={}}
{args=("c"), kwargs={}}

Although a single argument could suffice for representing the key, json is used for better flexibility.

If you want to support this feature you also need to add a flag somewhere around here to serialize Python objects by JSON too, as arguments via "infile" are always serialized with cloudpickle at the moment (see

cctools/taskvine/src/bindings/python3/ndcctools/taskvine/task.py

Lines 1113 to 1125 in 2c8f80d

	name = os.path.join(self.manager.staging_directory, "arguments", self._id)
	with open(name, "wb") as wf:
	cloudpickle.dump(self._event, wf)
	self._input_file = self.manager.declare_file(name, unlink_when_done=True, cache=False, peer_transfer=True)
	if self._tmp_output_enabled:
	self._output_file = self.manager.declare_temp()
	else:
	name = os.path.join(self.manager.staging_directory, "outputs", self._id)
	self._output_file = self.manager.declare_file(name, cache=self._cache_output, unlink_when_done=False)
	self._event = None # free args memory. Once in a file they are not needed anymore.
	self.add_input(self._input_file, "infile")

)

The new feature doesn't go through task.py, instead it creates tasks directly on the C side, I will create some examples later in a later PR!