How to run only self cehck facts rail ?

[VireshAmbardar]

So i am trying to implement Nemoguard frame work into my project..

I am calling my main LLM (chat completion) separately as i have included fallback mechanism for the mail LLM .. (Note : this is a separate functionality which we have included before including nemoguard)

Now i want to wrap Input and output rails around my functionality.

Now where i am stuck is i want to implement self check fact and extract score from it..

when i visit
https://docs.nvidia.com/nemo/guardrails/latest/user-guides/guardrails-library.html#fact-checking

there is a github link
https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/self_check/output_check/actions.py

which indicate to self_check_output

Q1 why is that ??

---

As i want to only include only self check fact in my optput rail.. and i already have llm response

and accordng to documentation

this is my prompt
my_context = "My_context"
llm_response = "response_from_main_llm"

prompt = f"""
prompts:
  - task: self_check_facts
    content: |-
      You are given a task to identify if the hypothesis is grounded and entailed to the evidence.
      You will only use the contents of the evidence and not rely on external knowledge.
      Answer with yes/no. "evidence": {my_context} "hypothesis": {llm_response } "entails":
"""

and my config contains

config_dict = {
    "models": [
        {
            "type": "main",
            "engine": "google_genai",         # or your LLM engine
            "model": "gemini-2.0-flash",
            "parameters": {
                            "api_key":"<API_KEY>"
                        }
        }
    ],
    "rails": {
        "output": {
            "flows": ["self check facts"]
        }
    }
}

and in my colang

define flow
  $check_facts = True
  $accuracy = execute self_check_facts
  if $accuracy < 0.5
    bot say "Blocked"
    stop
  else
    bot say "Correct"

and according to my workflow this is how i am making my rails

from nemoguardrails import LLMRails, RailsConfig
    
config = RailsConfig.from_content(
    colang_content=empty_colang,
    yaml_content=prompts,
    config=config_dict
)
rails = LLMRails(config)

how can i just run the rails on this output ???

[VireshAmbardar]

@drazvan @Pouyanpi Please help me with this. if you can

i am using nemoguard 0.17 version.

[Pouyanpi]

Hi @VireshAmbardar

re the documentation it is clearly wrong, the correct url should be: https://github.com/NVIDIA-NeMo/Guardrails/tree/develop/nemoguardrails/library/self_check/facts/actions.py

to run input rails or output rails only you can use GenerationOptions see ( to only run input rails, output rails, and both input and output rails)

you can use following util:

def determine_options_from_messages(messages: List[dict]) -> Optional[dict]:
    if not messages:
        raise ValueError("messages list cannot be empty")

    roles = {msg.get("role") for msg in reversed(messages)}
    has_user = "user" in roles
    has_assistant = "assistant" in roles

    if not has_user and not has_assistant:
        raise ValueError("not possible, right?") 

    if has_user and has_assistant:
        return {"rails": ["input", "output"]}
    if has_user:
        return {"rails": ["input"]}
    return {"rails": ["output"]}

then pass the options to the generate_async method:

options = determine_options_from_messages(messages)
response = await generate_async(messages=messages, options=options)

pay close attention to how messages look like for each of those options, for example when you want to run output rails only you must set :

{
    "role": "user",
    "content": ""
}

I hope it helps. BTW, I recommend you to use 0.19.0 version where a bug related to this feature was fixed.

In future releaes we will introduce a check_async method which will only run input/output rails without all these tweaks.

[VireshAmbardar]

@Pouyanpi

Yes i updated to version 0.19 as of today..

although generate is been mentioned in official document..

but i am getting this error

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[15], line 16
     10 messages = [
     11     {"role": "assistant", "content": llm_response_1}
     12 ]
     13 state = {"context": {"evidence": context}}
---> 16 response = await app.generate_async(
     17     messages=messages,
     18     state=state,
     19     options={"rails": ["output"]}
     20 )
     24 print(response)

File ~/LLM_Gateway/axoma-llm-gateway/.venv/lib/python3.11/site-packages/nemoguardrails/rails/llm/llmrails.py:884, in LLMRails.generate_async(self, prompt, messages, options, state, streaming_handler)
    882 if state:
    883     assert isinstance(state, dict)
--> 884     state_events = state["events"]
    886 new_events = []
    887 # Compute the new events.

KeyError: 'events'

this is my code block

from nemoguardrails import RailsConfig, LLMRails

prompt = """
prompts:
  - task: self_check_facts
    content: |-
      You are a fact-checking model.
      Evaluate the factual accuracy of the following answer.
      Return ONLY a JSON object with:
      - "score": number between 0 and 1

      Evidence:
      {{ evidence }}

      Answer:
      {{ response }}

"""

config_dict = {
    "rails": {
        "output": {
            "flows": [
                "self check facts"
            ]
        }
    },
    "models": [
        {
            "type": "main",
            "engine": "google_genai",
            "model": "gemini-2.0-flash",
            "parameters": {
                "api_key": "<API_KEY>"
            }
        }
    ]
}

config = RailsConfig.from_content(
    colang_content="", 
    yaml_content=prompt,   
    config=config_dict
)
app = LLMRails(config)

context = "The Treaty of Versailles was signed on June 28, 1919, officially ending World War I. The principal parties involved were the Allied and Associated Powers (including France, Britain, and the United States) and Germany. A key provision required Germany to accept sole responsibility for causing the war and mandated significant reparations payments."

# Correct response
llm_response_1 = "The Treaty of Versailles, signed in 1919, officially brought an end to World War I. Among its most significant terms was the requirement for Germany to accept responsibility for the conflict and pay substantial reparations to the Allied nations. This treaty took place in France."

# Hallucinated response
llm_response_2 = "The treaty that ended World War I was the Treaty of Ghent, which was signed in 1945. This agreement was notable for establishing the League of Nations as a security council based in New York and mainly addressed territorial disputes between Russia and the United States."

messages = [
    {"role": "assistant", "content": llm_response_1}
]

messages = [
    {"role": "assistant", "content": llm_response_1}
]
state = {"context": {"evidence": context}}


response = await app.generate_async(
    messages=messages,
    state=state,
    options={"rails": ["output"]}
)

print(response)

although when i open LLMRails i can see options generate()
which accepts GenerationRailsOptions / "output" as a single field

also i am not sure do i need to add model in my config or not.. can you clearify that also.

i am still struggling to pass the context and last llm response to check for facts_check

[VireshAmbardar]

@Pouyanpi @drazvan Can you please help

[VireshAmbardar]

@Pouyanpi @drazvan

i also explored the way to directly call the action

from nemoguardrails.library.self_check.facts.actions import self_check_facts

config = RailsConfig.from_content(
    colang_content="",
    yaml_content="",
    config=config_dict
)

# 3) Create rails (this builds the internal task manager)
rails = LLMRails(config)

result = await self_check_facts(
    llm_task_manager=rails.llm,  # internal, private, but needed
    context=context,
    llm=llm,
    config=config
)

here also i faced problem . How to pass BaseLLM

i dont it be a provider specific.. i want it to extract from config and decide internally which provider to use.

[VireshAmbardar]

also it is mentioned in documnetation that

but is there a way around ?

[Pouyanpi]

@VireshAmbardar the error shows the issue is related to how you are using the fact checking not input/output rails only options. Use following:

1. Pass context via {"role": "context"} message, not the state parameter
2. Use relevant_chunks as the context variable (not evidence)
3. Set check_facts: True to trigger the rail
4. Include a user message before the assistant message (required for output-only mode)

The action maps context variables to prompt variables:

  relevant_chunks (context) → {{ evidence }} (prompt)
  bot_message (context)     → {{ response }} (prompt)

and try out following (note that I have included the user message because fact checking requires the user message) It is using openai engine, change it to google_genai etc:

import asyncio

from nemoguardrails import LLMRails, RailsConfig

YAML_CONFIG = """
models:
  - type: main
    engine: openai
    model: gpt-4

rails:
  output:
    flows:
      - self check facts

prompts:
  - task: self_check_facts
    content: |-
      You are given a task to identify if the hypothesis is grounded and entailed to the evidence.
      You will only use the contents of the evidence and not rely on external knowledge.
      Answer with yes/no. "evidence": {{ evidence }} "hypothesis": {{ response }} "entails":
"""


config = RailsConfig.from_content(yaml_content=YAML_CONFIG)
app = LLMRails(config)

relevant_chunks = """The Treaty of Versailles was signed on June 28, 1919, officially ending
World War I. The principal parties involved were the Allied and Associated Powers
(including France, Britain, and the United States) and Germany. A key provision
required Germany to accept sole responsibility for causing the war and mandated
significant reparations payments."""

factual_response = """The Treaty of Versailles, signed in 1919, officially brought an end
to World War I. Among its most significant terms was the requirement for Germany to accept
responsibility for the conflict and pay substantial reparations to the Allied nations."""

hallucinated_response = """The treaty that ended World War I was the Treaty of Ghent,
which was signed in 1945. This agreement was notable for establishing the League of Nations
as a security council based in New York and mainly addressed territorial disputes between
Russia and the United States."""

messages = [
    {
        "role": "context",
        "content": {
            "relevant_chunks": relevant_chunks,
            "check_facts": True,
        },
    },
    {"role": "user", "content": "What was the Treaty of Versailles?"},
    {"role": "assistant", "content": factual_response},
]

response = await app.generate_async(
    messages=messages,
    options={"rails": ["output"]},
)
print("Factual response result:")
print(response)
print()

messages = [
    {
        "role": "context",
        "content": {
            "relevant_chunks": relevant_chunks,
            "check_facts": True,
        },
    },
    {"role": "user", "content": "What was the Treaty of Versailles?"},
    {"role": "assistant", "content": hallucinated_response},
]

response = await app.generate_async(messages=messages, options={"rails": ["output"]})
print("Hallucinated response result:")
print(response)