Inspect/intercept traces before they are sent to the Langfuse backend

[yuchsiao]

I have a scenario that I cannot log data (traces) with PII into the langfuse backend so I will need to get PII redacted via a separate service. Yet, I want to evaluate the trace language generation performance using the original traces.

One possibility: If I can inspect/intercept the trace object for post processing before sending it out to Langfuse, that will make it work. Even more, if I can post-process it and sent it to a different project_id to the same or a different Langfuse backend, that will make things a lot flexible.

Any guidance?

[marcklingen]

This is an interesting requirement, thanks for sharing!

There are some options today:

1. Data masking
2. Using the low-level SDKs, you can connect to multiple Langfuse projects simultaneously by creating multiple SDK Clients (Langfuse(<keys>)). This approach helps dynamically route traces to projects based on user inputs. Currently, Langfuse does not support middleware or callbacks for dynamic routing based on inspecting the trace, so you must use the appropriate Langfuse SDK Client for each trace before creating it. This pattern is implemented in all multi-tenancy applications that have a Langfuse integration, e.g. Litellm and Dify.

Is (1) enough for your use case or could you use (2)?

[yuchsiao]

Thank you Marc for the quick response.

For 1, the doc of Data masking reads like the supplied masking function is called individually for each observe. If we want to redact the same person's name in the same way (e.g., replace name John with <client_1>) across the whole trace (or even a session, but we can discuss session later), how do we do it? I guess we will need to present some whole trace and do the redaction in one shot but the current design seems not allowing to do so easily. And a side question, how does the latency of the masking function contributes to the user experience? Will it be applied right before flush to the backend or will it be computed along the line of application computation? The latter will impact the user experienced latency, right? Also, once data masking is applied, the Langfuse backend won't receive the original trace, hence the evaluation is not truthful, correct?

For 2, it seems possible to send 1) the trace with data masking and 2) without data masking to two separate backends (e.g., via two project ids). Yet, we will still need to manipulate 2) the trace without data masking by a) extract inputs/outputs and b) apply an evaluation functions, and c) remove all inputs/outputs and attach the evaluation scores, before sending the trace to the backend. Any guidance on how to accomplish this?

I found that langfuse_context.client_instance.client.trace.list() can grab the traces so far. But it is not clear which has been sent to the backend and which has not. And all data objects are not immutable: "TraceWithDetails" is immutable and does not support item assignment.

Any idea to proceed?

Thanks a lot in advance!

[marcklingen]

the supplied masking function is called individually for each observe

that's correct

If we want to redact the same person's name in the same way (e.g., replace name John with <client_1>) across the whole trace (or even a session, but we can discuss session later), how do we do it? I guess we will need to present some whole trace and do the redaction in one shot but the current design seems not allowing to do so easily.

That's currently not possible as the traces are sent before they are completed to be able to view in-progress long-running executions in Langfuse. Open for suggestions on this and contributions to the masking implementation. I understand the requirement and why this would be useful in this scenario.

Alternatively, you could switch to the low-level Langfuse SDK and create traces/observations manually. Thereby you have full control what kind of events are created and tracked in Langfuse.

how does the latency of the masking function contributes to the user experience? Will it be applied right before flush to the backend or will it be computed along the line of application computation?

yes, the mask is applied in the background thread and does not add latency to the execution of the application.

Also, once data masking is applied, the Langfuse backend won't receive the original trace, hence the evaluation is not truthful, correct?

This depends. he unmasked data is not available in Langfuse (that's the whole point of the mask) but that does not impact many evals. But this depends on the eval that you might want to run – i.e. "does the firstname start with A" would obviously be broken if you redact names

For 2, it seems possible to send 1) the trace with data masking and 2) without data masking to two separate backends (e.g., via two project ids). Yet, we will still need to manipulate 2) the trace without data masking by a) extract inputs/outputs and b) apply an evaluation functions, and c) remove all inputs/outputs and attach the evaluation scores, before sending the trace to the backend. Any guidance on how to accomplish this?

I'd suggest to switch to the low-level SDK if you want to have full control. Have a look at the documentation of the low-level SDK and let me know if you have any follow up questions.