[bug] 500 status code for Llamaguard-7B

[iamvineettiwari]

Describe the bug
Getting ValueError: Failed to get valid response from Llamaguard-7B model. Status: None. Detail: Unknown error on Llama Guard.

Following are the configurations I am using:

remote-inferencing is enabled

config.py for guard:

llama_guard = Guard(
    name="llama_guard"
).use(
    LlamaGuard7B,
    policies=[
        LlamaGuard7B.POLICY__NO_ILLEGAL_DRUGS,
        LlamaGuard7B.POLICY__NO_VIOLENCE_HATE,
        LlamaGuard7B.POLICY__NO_SEXUAL_CONTENT,
        LlamaGuard7B.POLICY__NO_CRIMINAL_PLANNING,
        LlamaGuard7B.POLICY__NO_GUNS_AND_ILLEGAL_WEAPONS,
        LlamaGuard7B.POLICY__NO_ENOURAGE_SELF_HARM,
    ],
    on_fail=OnFailAction.EXCEPTION,
)

To Reproduce
Steps to reproduce the behavior:

1. Run with the provided config

Expected behavior
A valid response for the llmOutput message

Library version:
Version - 0.6.6

[ls-omar-nbou]

Hi there, just want to mention that I'm having the exact same issue with remote inference enabled on version 0.6.6. The same issue seems to be also present on another remote inferencing model https://hub.guardrailsai.com/validator/guardrails/shieldgemma_2b . Hence, I'm wondering if this is a general problem with remote inferencing based guardrails/validators; or maybe there something I'm not configuring properly.

Relevant piece of code (I'm implement Guards on both input and output):
**input_guard = Guard()
input_guard.name = 'ChatBotGuard - inputs'
input_guard.use_many(ShieldGemma2B(), DetectPII())

output_guard = Guard()
output_guard.name = 'ChatBotGuard - inputs'
output_guard.use_many(ShieldGemma2B(), DetectPII())**

When executing my test script, the DetectPII() guardrails is working properly except for the remote-inference based on which returns a 500:

🔍 Running test: safe_emoji
🔸 Input: Here are some fun emojis 😊👍🎉
ERROR:root:500
❌ BLOCKED: Failed to get valid response from ShieldGemma-2B model. Status: None. Detail: Unknown error

🔍 Running test: contains_pii
🔸 Input: My phone number is 555-123-4567
❌ BLOCKED: Validation failed for field with errors: The following text in your response contains PII:
My phone number is 555-123-4567

🔍 Running test: violent_instruction
🔸 Input: Here's how to make a Molotov cocktail
ERROR:root:500
ERROR:root:500
❌ BLOCKED: Failed to get valid response from ShieldGemma-2B model. Status: None. Detail: Unknown error

🔍 Running test: normal_greeting
🔸 Input: Hi there! How can I help you?
ERROR:root:500
❌ BLOCKED: Failed to get valid response from ShieldGemma-2B model. Status: None. Detail: Unknown error

As an FYI: I'm a newbie here :) , I started to use guardrails for the past 24 hours; hence, sorry in advance if I missed something obvious in my setup.

Txs in advance for the help.
Cheers.

[ls-omar-nbou]

Hello @iamvineettiwari , it seems that the Llama Guard (remote inference validator) is working again.
I did a couple of tests this afternoon and it looks like I'm not encountering the same issue again.

Let me know if you see the same behaviour on your end.

Cheers.

[iamvineettiwari]

Thanks @ls-omar-nbou for the update !