[bug] #1099
Description
Describe the bug
Using await on an AsyncGuard with the callable litellm.acompletiion results in following error when using guardrails==0.5.11
async for chunk in resp_gen:
File "/opt/miniconda3/envs/GUARDENV/lib/python3.9/site-packages/guardrails/telemetry/guard_tracing.py", line 197, in trace_async_stream_guard
res = await anext(result) # type: ignore
NameError: name 'anext' is not defined
To Reproduce
- Install guardrails-ai==0.5.11
- Install guardrails hub install hub://guardrails/profanity_free
- Set below env variables
export OPENAI_ENDPOINT_URL=https://api.openai.com/v1/chat/completions
export OPENAI_API_KEY=sk-...
- Copy below script to demo.py and run as
python demo.py
import asyncio
import json
import time
import guardrails as gd
import litellm
from guardrails.hub import ProfanityFree
guard = gd.AsyncGuard().use(ProfanityFree, on_fail="exception")
def outcome_to_stream_response(validation_outcome):
stream_chunk_template = {
"choices": [
{
"delta": {
"content": validation_outcome.validated_output,
},
}
],
"guardrails": {
"reask": validation_outcome.reask or None,
"validation_passed": validation_outcome.validation_passed,
"error": validation_outcome.error or None,
},
}
stream_chunk = stream_chunk_template
stream_chunk["choices"][0]["delta"]["content"] = validation_outcome.validated_output
return stream_chunk
async def guarded(messages):
# Call the OpenAI API through litellm and guard
fragment_generator = await guard(
litellm.acompletion,
model="gpt-4o-mini",
messages=messages,
max_tokens=1024,
temperature=0,
stream=True,
)
return fragment_generator
async def process_chunks(resp_gen):
async for chunk in resp_gen:
chunk_string = f"data: {json.dumps(outcome_to_stream_response(chunk))}\n\n"
print(chunk_string)
user_message = "tell me about singapore"
messages = [{"content": user_message, "role": "user"}]
resp_gen = asyncio.run(guarded(messages))
print(resp_gen)
asyncio.run(process_chunks(resp_gen))
Above code is adapted form
- https://www.guardrailsai.com/docs/concepts/async_streaming
- https://github.com/guardrails-ai/guardrails-api/blob/main/guardrails_api%2Futils%2Fopenai.py
Expected behavior
Code should not error out
Library version:
Version (e.g. 0.5.11)
Additional context
Works fine on guardrails-ai==0.5.1