Langserve RemoteRunnable Error

[conaku]

I am invoking RemoteRunnable to remote langserve instance that hosts a chain_with_guardrail and facing error. When I call the invoke method on chain without guardrail then it works fine.

Server Side

chain = (
{"context": reranking_retriever, "question": RunnablePassthrough()}
| prompt_template
| llm
| StrOutputParser()
)

chain_with_guardrails = guardrails | chain

add_routes(
app,
chain_with_guardrails,
path="/conversational_ai_with_rag_text_chat"
)

Client Side

llm = RemoteRunnable("http://127.0.0.1:90000") | StrOutputParser()
llm.invoke("Tell me something about Langchain")

this invoke method is erroring out if the langserve chain is having "chain_with_guardrails" but not erroring with just chain

[dioschuarz]

Hi,

I am with same error here, but from my Server Side I also tested with a LangGraph Runnable.

Here is the error:

{'messages': [HumanMessage(content='Olá', id='008831e6-0197-4d6e-96e4-773438719bd6'), HumanMessage(content='Olá', id='040e06b5-72b2-4ee9-9d88-50eb20471455'), HumanMessage(content='Desculpe não consigo responder essa pergunta. Posso te ajudar em algo mais?', id='013c48ef-a832-42b3-8548-99980cef55f2')], 'question': 'Olá'}
INFO:     127.0.0.1:40170 - "POST /api/v1/ai_agent/invoke HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/routing.py", line 754, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/routing.py", line 774, in app
    await route.handle(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/routing.py", line 295, in handle
    await self.app(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/fastapi/routing.py", line 297, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/fastapi/routing.py", line 210, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/langserve/server.py", line 530, in invoke
    return await api_handler.invoke(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wsl/git/references/playground/playground/lib/python3.11/site-packages/langserve/api_handler.py", line 848, in invoke
    self._InvokeResponse(
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ai_agentInvokeResponse
output
  value is not a valid dict (type=type_error.dict)

It looks like an error inside of module api_handler.py, I tried a lot of ways to isolate it and solve, but when I isolate it, it works:

from langchain.load.dump import dumps
from langchain_core.messages import HumanMessage
from langserve.api_handler import _json_encode_response, _resolve_model
from langserve.validation import create_invoke_response_model
from langserve.schema import InvokeResponseMetadata
from langserve import RemoteRunnable
import uuid
import os

AI_AGENT_ENDPOINT = os.getenv("AI_AGENT_ENDPOINT")
BACKEND_URL = os.getenv("BACKEND_URL")
CHAT_URI = BACKEND_URL + AI_AGENT_ENDPOINT

app = RemoteRunnable(CHAT_URI)

d = {'messages': [HumanMessage(content='Olá', id='8071c888-393b-4de8-9027-03919a3fbe27'), 
                  HumanMessage(content='Olá', id='ebaf4f44-6922-44ce-8d70-49359f91b0ff'),
                  HumanMessage(content='Desculpe não consigo responder essa pergunta. Posso te ajudar em algo mais?', id='cf51f97d-df1f-4d1e-a3f1-5baa5d42ad86')],
     'question': 'Olá'
    }

dd = dumps(d)

output_type_ = _resolve_model(
            app.get_output_schema(),
            "Output",
            app.get_name(),
        )

ivk = create_invoke_response_model(
    app.get_name(),
    output_type_,
    False
    )

res = _json_encode_response(ivk(output=dd,
                                metadata=InvokeResponseMetadata(run_id=uuid.uuid1(),
                                                                feedback_tokens=[])))

res.body

Output:
b'{"output":"{\"messages\": [{\"lc\": 1, \"type\": \"constructor\", \"id\": [\"langchain\", \"schema\", \"messages\", \"HumanMessage\"], \"kwargs\": {\"content\": \"Ol\\u00e1\", \"type\": \"human\", \"id\": \"8071c888-393b-4de8-9027-03919a3fbe27\"}}, {\"lc\": 1, \"type\": \"constructor\", \"id\": [\"langchain\", \"schema\", \"messages\", \"HumanMessage\"], \"kwargs\": {\"content\": \"Ol\\u00e1\", \"type\": \"human\", \"id\": \"ebaf4f44-6922-44ce-8d70-49359f91b0ff\"}}, {\"lc\": 1, \"type\": \"constructor\", \"id\": [\"langchain\", \"schema\", \"messages\", \"HumanMessage\"], \"kwargs\": {\"content\": \"Desculpe n\\u00e3o consigo responder essa pergunta. Posso te ajudar em algo mais?\", \"type\": \"human\", \"id\": \"cf51f97d-df1f-4d1e-a3f1-5baa5d42ad86\"}}], \"question\": \"Ol\\u00e1\"}","metadata":{"run_id":"3f67bafa-6bce-11ef-98c8-9b7daa0c7572","feedback_tokens":[]}}'

Does anyone knows how to solve it in LangServe?

Why we have an endpoint like invoke if it don't works?