Field-level selection for input/output filtering in decorator instead of using update_current_observation/trace

[amir-mic-lin]

Describe the feature or potential improvement

This feature lets you pick specific fields from the input and output of a function by using a decorator. It allows you to control which parts of the data you want to keep or exclude, without changing how the function works.
I want to avoid calling the update function inside the function.

Option 1:

@observe(input_to_capture=["content.messages"], output_to_capture=["data.0"])
def run_agent(content):
     ....
    return data

Another approach can be to send instead of the field hierarchy a func that extracts the data required for input and output

Option 2:

@observe(input_to_capture=self.extract_relevant_input)
def run_agent(content):
    ....
    return data

Additional information

No response

[marcklingen]

Thanks for sharing this idea! Agree that both would be a nice alternatives to the current explicit overrides.

[amir-mic-lin]

@marcklingen can I open issue and create a PR for it?

[hassiebp]

@micky-amir Thanks a lot for raising this! While I agree that removing instrumentation code from the observed function body via some form of field-level selection would be good, I see significant implementation + maintenance complexity in covering all possible data structures and edge cases which such a generic field access via dot notation requires.

I think though there is some elegance in providing an interface for some selector function which would not only allow select but also transform while keeping the decorator interface + implementation clean. Important here would be to have appropriate exception handling. I'm thinking along these lines:

@observe(
    input_transform=lambda args, kwargs: {'messages': args[0]['messages']},
    output_transform=lambda result: result[0]
)
def main(content):
    ...
    return data

[marcklingen]

Seems like this could be related to #3518

A transform or mask that can be configured globally or overridden for a single decorated function.

[amir-mic-lin]

@hassiebp regarding the first option I suggested, I completely agree—it could lead to many future issues.
What you're suggesting aligns with what I intended in option 2, but you've expressed it more clearly :). The title of the discussion primarily refers to option 1, which includes a selector.

@marcklingen I do believe that #3518 could be implemented with the suggestion above. However, I’m wondering if, in addition to input and output transformations, it might be beneficial to include middleware for masking secrets or similar elements in the input and output (since the transformers are intended to modify the format and select specific fields). My LLM secret was sent more than once to the server which is no good.

[marcklingen]

Agree, that these masks/transformations could stack if they are set on the function and global level.

My LLM secret was sent more than once to the server which is no good.

Can you open an issue for this? This should not happen with any of the Langfuse integrations

[amir-mic-lin]

@marcklingen it happens when I can use a Langfuse llm, but the vanilla one. That is by design as I see it, no?

What should be the next step with this with this idea?

[marcklingen]

I do not understand the issue, can you open an issue and help with some steps to reproduce this?

[amir-mic-lin]

One of the params of the method (which was decorated with @observe) was a llm client which stored the azure open ai deployment secret.
It is pretty easy to reproduce, one just need to put the azure llm client as a param and as a result it will be logged to the server.
As the whole object is serialized and sent to the server, the secret was sent too. I think that it is the responsibility of the one writing the code to make sure no secret is being sent. Therefore, not sure that there is anything to do here.