[ModelProvider] error. Reason: Empty or invalid response in batch

[Hankpipi]

I encountered the same issue as #184. Below is the script I used:

import os
import duckdb

if __name__ == "__main__":
    print("DuckDB:", duckdb.__version__)

    con = duckdb.connect(database=":memory:")
    con.execute("INSTALL flock FROM community;")
    con.execute("LOAD flock;")

    print(
        con.execute("""
                        SELECT *
                        FROM duckdb_extensions() 
                        WHERE extension_name = 'flock';""").fetchall()
    )

    openai_api_key = os.getenv("OPENAI_API_KEY")
    con.execute("CREATE SECRET (TYPE OPENAI, API_KEY ?);", [openai_api_key])

    model_alias = "test"
    base_model  = "gpt-4o"

    con.execute(f"""
        CREATE MODEL(
            '{model_alias}',
            '{base_model}',
            'openai',
            {{
                "tuple_format": "json",
                "batch_size": 32,
                "model_parameters": {{"temperature": 0.7}}
            }}
        );
    """)

    result_df = con.execute(f"""
        SELECT llm_complete(
            {{'model_name': '{model_alias}'}},
            {{'prompt': 'Talk like a duck and write a poem about a database'}}
            );
    """).fetchdf()

    print(result_df)

The output is:

DuckDB: 1.4.0
[('flock', True, True, '/home/-/.duckdb/extensions/v1.4.0/linux_amd64/flock.duckdb_extension', '', [], '7f1c36a', 'REPOSITORY', 'community')]
Traceback (most recent call last):
  File "/home/-/flock/test.py", line 37, in <module>
    result_df = con.execute(f"""
_duckdb.Error: [ModelProvider] error. Reason: Empty or invalid response in batch

I am sure that the API key is valid, as the API key works with older versions of Flock (e.g., DuckDB 1.3.0 + Flockmtl 3de4a71). Could you please help me double-check this issue?

[ju-klein]

I have the same problem (albeit using C++, not Python). I'll investigate! ( @queryproc: Assign this to me, please, if that is okay! :) )

[anasdorbani]

@Hankpipi , I tried reproducing the issue with the code you provided, but I wasn’t able to. I tested it on both osx_arm64 and linux_amd64 for versions 1.4.0, 1.4.1, and 1.4.2 using Python. Could you please check the DuckDB CLI and let me know if you see the same issue?

@ju-klein , definitely, I’ll assign it to you for now. Could you also share any reproduction environment or code that I can use to verify the issue on my end?

[Hankpipi]

@anasdorbani Yes, I tried using the DuckDB CLI, but encountered the same error.

[grbinho]

I get the same error with DuckDB 1.4.4.

[anasdorbani]

@grbinho @Hankpipi @ju-klein We’ve completed a major refactor to address the related issues. Could you verify that the problem no longer occurs by running the following?

pip install duckdb==1.4.4

import duckdb

conn = duckdb.connect(":memory:")

conn.execute("FORCE INSTALL flock FROM community")
conn.execute("LOAD flock")

conn.execute("GET PROMPTS;")
print(conn.fetchall())

conn.execute("select * from duckdb_extensions() where extension_name = 'flock'")
print(conn.fetchall())

Please let me know if you run into any problems so we can address them.

[grbinho]

@anasdorbani I was able to execute your snipped without a problem, but if after that I add an OPENAI secret and try to execute a query that uses that hello-world prompt SELECT llm_complete({'model_name': 'default'}, {'prompt_name': 'hello-world'}); i get

_duckdb.Error: [ModelProvider] error. Reason: Invalid JSON response:

[anasdorbani]

@grbinho, to help verify that everything is working correctly on your side, since I’m unable to reproduce the issue, could you please share the following details?

• Your OS
• The steps you used to create the secret in Flock
• The response you get when running this command:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Tell me hello world"
      }
    ]
  }'

[grbinho]

Sure.

OS: Linux (Ubuntu 24.04)
I created the secret like this

CREATE SECRET IF NOT EXISTS (TYPE OPENAI, API_KEY 'my_secret_key');

This is the response

curl https://api.openai.com/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer $OPENAI_API_KEY"   -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Tell me hello world"
      }
    ]
  }'
{
  "id": "chatcmpl-DFJi8KXrJv80aPzfnrAtR4X1OzSUf",
  "object": "chat.completion",
  "created": 1772543060,
  "model": "gpt-4o-mini-2024-07-18",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello, World!",
        "refusal": null,
        "annotations": []
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 4,
    "total_tokens": 15,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": "fp_373a14eb6f"
}

[anasdorbani]

@grbinho, thank you for the follow-up. Are you building from the source code or installing the community version? I haven't been able to reproduce this on my end yet, but we’re about to push a new release with enhanced logging to help us debug the issue more effectively.

[grbinho]

@anasdorbani I just installed the community version. I did it in Python with the example you shared and just added a few commands after it.
If you need me to, I'm happy to try and build from source and then run that to get some better logging.
Just let me know what to pull.

[anasdorbani]

@grbinho, as the release will take some time, it would be better if you pull the repo and try to build it. Here's the instruction:

# 1) Clone Flock
git clone --recursive https://github.com/dais-polymtl/flock.git
cd flock

# 2) Set up vcpkg toolchain (one-time)
source ./scripts/setup_vcpkg.sh

# 3) Build Flock
make -j"$(nproc)"

#4) Run duckdb with Flock loaded
./build/release/duckdb

Then run your queries and let me know what error you are getting.

Thanks!