Add support for custom serialization of observation fields

[ashamlian-roadway]

Describe the feature or potential improvement

Currently (and if I'm not mistaken), observation data is serialized with EventSerializer. This class supports relatively robust serialization out of the box, but I think it would be nice to allow users to specify how to serialize objects within the observation. As an example, an observation may have an input with args: (str, pd.DataFrame) object as function input or output.

The default logic here assumes that this input is serializable, but in fact we need to specify how to serialize the DataFrame. Additionally, it is possible that the order of checks in the EventSerializer.default() method should be altered such that serialization of Sequences comes before that of lists or tuples (though I can appreciate there being a LOT of edge cases here...hence the request for general, custom serialization).

A CustomEventSerializer could be injected as an arg to Langfuse() to allow clients to specify how to handle arbitrary types, which is extremely useful when tracking observation inputs and outputs from span to span in conjunction with generation observations (practically speaking, I'm referring to agentic tool-calling scenarios)

Additional information

No response

[stepanogil]

The input and and output of each observation are completely customizable. Just set the capture_input and capture_output args in the observe() decorator to False. Then specify the the desired format of your observation using the 'input' and 'output' params of langfuse_context.update_current_observation().

[ashamlian-roadway]

Is there a way to do this without calling update_current_observation()/update_current_trace() within every desired func? If not, this kinda defeats the purpose of the observe() decorator.

Either way, I think injecting a custom serializer would be a better way to handle this, as it minimizes the "leakage" of Langfuse. The more Langfuse code you see, the harder it is to focus on what you really care about - your application logic.

[marcklingen]

Agree that a custom serializer would be helpful here as overriding the input/output for each and every function adds a lot of instrumentation code which is not necessary. @hassiebp what do you think about this?

[hassiebp]

Agree this would be an elegant solution to allow passing a custom serializer. Added to our backlog of SDK improvements and will update this discussion once released 👍🏾