Support for more modalities and base64 content

[marliessophie]

Describe the feature or potential improvement

After our latest release supporting images (passed via url) in Langfuse, we're looking to support other modalities soon.

We're committed to supporting base64 images in the near term. We're further planning support for audio, video, and attachments (e.g. PDFs).

We'd love to gauge interest across the modalities to help with most efficient prioritisation.

[chai3]

+1 base64 image (most wanted)
+1 audio
+1 attachments(pdf, xlsx, html)

I dont need to support video in my use-case

[badbye]

Great! The dataset also need to support Multi-Modal input. Looking forward!

[Alpacolypse]

Eagerly awaiting base64 support, especially in markdown

[clemra]

A few double clicks on base64 as reactions to our most recent newsletter

[Hallicopter]

base64 images for sure.

[bioshazard]

+1 b64 images. Seems like the very next thing to support. Schema seems well established at this point.

[bioshazard]

Update: So I actually wound up writing a span intercept to strip b64 data and replace with filename. I plan on having a batch operation and worried it would balloon my DB. While looking at supporting other modalities, consider a way to decouple attachment reference so it can maybe resolve from a single source than to risk duplicating into the traces.

[tcbonds]

Base64 images for multimodal models is the missing feature that is holding us back from using Langfuse. PDF attachments would also be very helpful.

[pleomax0730]

@tcbonds What alternative solution are you currently using? Our team is planning to add an AI observability tool, and your experience and insights would be greatly appreciated.

[tcbonds]

Arize Phoenix. So far we have had a pretty good experience with it.

[profikid]

Bas64 or image url would be great to support

[wbreilly]

Audio is mission critical for us

[zxkxyz]

just here to add my support for base64 (because claude)

[marcklingen]

this is on the near-term roadmap, @hassiebp is working on it and specs look great, can't wait for the release!

[pleomax0730]

Here to support Audio, PDF source.

[darylsew]

+1 base64 (because claude)

[alexdanilowicz]

+1 base64 image (because claude)

[hassiebp]

Good news - we have just launched multimodal support in beta! 🎨

• Changelog post
• Docs
• JS SDK release
• Python SDK release

Please let us know your feedback here or create an issue if you encounter any bugs. Thanks! 🙏🏾

[marcklingen]

huge! looking forward to the feedback

[bioshazard]

DUDE YES! Y'all decoupled it to a bucket!! This is HUGE!! LANGFUSE_S3_MEDIA_UPLOAD_, now I can undo my customer span processor I was using to prevent the b64 from going into my DB.

[mlcocdav]

Hi, I've tried the suggested ways to upload images using the API directly from https://langfuse.com/docs/tracing-features/multi-modality

But my images do not render correctly in the UI. I tried both Markdown with the image link and sending base64 encoded data URIs passing into the trace input field.

This is the UI:

I want to see the images rendered.

Please help 🙏

[AviPeltz]

the ability to embed an iframe -> we would want to visualized 3d objects so either native support for a simple three.js renderer would be pretty easy to implement and nice, alternatively an iframe where we could add our own 3d renderer

[tom-diacono]

It would be really cool if this supported OTEL data. From from what I'm seeing so far it doesn't. If I could send my traces with the images in base64 (which I'm currently doing anyway), it would be great if Langfuse converted them to it's object storage, and have a link to them. Unfortunately for now, it's looking like I'm going to have to call an API to do this.

[tom-diacono]

Hi Marc,

Thanks for the reply, that's really good to know! Both that the new versions will be OTEL Native, and that images will be taken into account! (Personally a big OTEL fan).

Our current implementation is using the Golang OTEL SDK, which as far as I can see, is a couple of versions behind the new OTEL semconv (meaning no event bodies, and slightly older gen_ai attributes, but seems to be working ok with Langfuse so far, I guess it still supports them).

So even when you guys get onto OTEL native, we'd probably have to roll our own implementation for images, given there's no Go SDK yet, and you're looking to handle images client side. I'm guessing we can still do that via API?

So far, though I may have mentioned this in a separate thread, I've not had much luck getting my implementation to work. I'm unsure whether it's just frontend rendering though. We've got multiple "chat" spans per trace, but our root span is not a "gen ai" one, it's a standard app trace. Is it the case that the frontend is only looking for images to render in the root span? I've noticed that in spans where I can see images (single call spans) I can only click on them when a top level spal is selected.

Can grab a screen grab tomorrow if you're interested

[marcklingen]

Can you share your current go implementation as a gist together with any errors that you get? Generally the media api returns presigned s3 upload links

[tom-diacono]

I've just re-read this and my messages are confusing even me, so first let me clarify:

• In the first message I was referring to hoping that I could send the full base64 encoded image data in the trace, and have Langfuse convert it for me (which, given you explanation, is going to be done client side with the new SDK's, so I'm guessing probably won't end up being supported- all good, I think this one is answered)

• Second message is once I'd moved to doing the following:

  • Extract the image data from the traces client side
  • Send it to the Langfuse with the API call (which is successful)
  • Finally replace the image data in the trace (again, client side) with the Langfuse media token

So- what I'm trying to get to work is the images in the frontend. Below is an example trace of what it's currently looking like. You can see the media link, but there's no image link within the trace window.

I know the code works, as when I use it with a span of which the openai call is the root span, I can see the image.

The only thing I'm wondering- is will these images only show if the metadata containing them is in the root span, as opposed to being a child span? In the span in my second screenshot. I can only see the image when I'm focusing on the root one. If I focus on either child span, openai_request or openai_completion, there's again no image link. Given this last point, I'm wondering if this is by design?

[tom-diacono]

So to update- I finally got this to somewhat work.

The issue I was having was due to sending the wrong Trace ID. I was sending the Span ID instead 🙄 .

Something it does highlight though (and I think I was alluding to this in my previous message, just not that I really knew it) is that it would be really cool if the media was stored per span that it was created in, as opposed to at the "Trace" level.

This would help as if there were many spans per trace, it would allow you to attribute the data to the correct span, as opposed to having all images at the top level, with no real order, or way to attribute it to a particular call

[git-hulk]

@tom-diacono You can have a try at https://github.com/git-hulk/langfuse-go, which supports most Langfuse APIs.

[jingtra]

Video would be great!

[barseghyanartur]

Please, add DOCX support.

[thomasdussaut]

As stated in this issue : #7319, in the meanwhile it would be a nice workaround to work with the dataset item POST endpoint. As for now, this route doesn't accept payloads bigger than 1MB. Increasing this limit to 10MB would allow to work with most PDF documents and images

[RuidongLiu]

need video support.

[Janscode]

same video support is probably a blocker for my organization

[Namir0806]

Please also support .flac file format for audio files. Thank you!

[Namir0806]

Please also support .ogg file format for audio files. Thank you!

[rileymiller]

csvs and xlsx would be great!!!

[marin1994]

html, markdown, but most important (as said above) video

[iyansr]

The base64 somehow not working (?)

[hassiebp]

Could you please create an issue with a trace link and repro example?

[iyansr]

@hassiebp I create issue here #10495

[JoostvDoorn]

For litellm it seems this works only when using the langfuse v2 integration as explained here https://docs.litellm.ai/docs/observability/langfuse_integration. With the v3 integration using litellm.callbacks = ["langfuse_otel"] it will just store the base64 data and not display it in the interface.

[alexbensavanta]

Super tool out there! would love to see video rendering as well!