v1.60.0 removes support for log_stream_event and async_log_stream_event

[VyValence]

I currently use async_log_stream_event in my app to parse each chunk returned by litellm. The removal of this callback support for custom loggers breaks this functionality. I was wondering if any here has used this callback to do something similar and has found a workaround for this. I'm writing my own handler now but this was a surprising breaking change to me and a valuable feature that was removed.

[krrishdholakia]

Hey @VyValence we discovered a memory leak caused by firing a thread on each chunk, so i believe this might've been impacted as a result of that.

Do you just need a list of chunks? if so, can we attach this on the success event?

I agree the change, if intentional, could've been better handled.

[VyValence]

Hey- thanks for the background on this issue. Really I need each chunk and the timestamp in which it was received- This was originally provided in the async_log_stream_event callback since the callback was called whenever we received the next chunk. This allows me to find the time between specific chunks in the response.

[krrishdholakia]

maybe we can retain the event hook, just not fire a new thread for each chunk

cc: @ishaan-jaff - since i think you worked on the original fix

[VyValence]

Will keep my eyes on this issue 👀 @ishaan-jaff if this sounds possible that would be great.

[krrishdholakia]

Hey @VyValence i don't see this behaviour - i'm running streaming, and i can see both events being logged

[VyValence]

I don't see this with my custom logger. This release also indicates it was deprecated: https://github.com/BerriAI/litellm/releases/tag/v1.60.0

These events definitely aren't being called after I upgraded from 15x.x -> 16x.x with no other changes.

[VyValence]

Are you seeing these events fired for each chunk being returned? If so, what version are you on? @krrishdholakia

[krrishdholakia]

i see it being fired on each chunk -

litellm/litellm/litellm_core_utils/streaming_handler.py

Line 1558 in 41d1a88

executor.submit(

How do i repro the error you're seeing?

[VyValence]

This is what I use for my custom logger configuration. I'm not seeing the stream callbacks hit.

class MyCustomLogger(CustomLogger):
    def _set_first_visible_token_time(self, kwargs, response_obj, end_time, is_async=False):
        """
        Checks the streaming response chunk for a user-visible token, and if there is one, sets the
        `first_visible_token_time` timestamp on the call's `kwargs`.  
        """

    def log_stream_event(self, kwargs, response_obj, start_time, end_time):
        self._set_first_visible_token_time(kwargs, response_obj, end_time)

    async def async_log_stream_event(self, kwargs, response_obj, start_time, end_time):
        self._set_first_visible_token_time(kwargs, response_obj, end_time, is_async=True)

litellm.callbacks = [MyCustomLogger()]

[VyValence]

Been trying on v1.61.19