`PluginConfig` object has no attribute `_paged_kv_cache`

[Yoloex]

System Info

I tried to run the following sample code but got the error saying PluginConfig object has no attribute _paged_kv_cache when initializing LLM.

Here is the detailed info.

tensorrt_llm.version 0.19.0
OS: WSL - Ubuntu 22.04
Host CPU: Intel Core Ultra 9 185H
GPU: Nvidia RTX 4070

from tensorrt_llm import LLM, SamplingParams


def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

    # Print the outputs.
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


# The entry point of the program need to be protected for spawning processes.
if __name__ == '__main__':
    main()

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Install tensorrt_llm==0.19.0 on WSL - Ubuntu 22.04 with Python 3.10.0
2. Run the above code

Expected behavior

The code should run smoothly.

actual behavior

Got an error when initializing LLM instance.

additional notes

Could be the issue with WSL?

[hchings]

Hi @Yoloex. _paged_kv_cache is defined here in v0.19 and this quickstart example has been tested before v0.19 release and is in our CI. I haven't seen yet how this would be WSL dependent.

Could you try it with main as well and see if you still have issue? I can run through your script.

[Kane-Wong]

I encountered the same problem. Removing "(slots=True)" in plugin.py L140 can avoid this error.

[karljang]

I don’t have a WSL environment, but the sample code runs without any errors on my Ubuntu machine, at least.
I’m closing this issue due to its prolonged inactivity. However, if the problem persists in the latest release, please feel free to open a new issue.
Thanks!

[skiloop]

I encountered the same problem. Removing "(slots=True)" in plugin.py L140 can avoid this error.

not sure if this could be a solution, solts=True enable dynamic field to the class, some attributes (such as _paged_kv_cache) may not be able to add in runtime if set to false. The problem seems to be mixed using dynamic lib files (so files on linux)

[Kane-Wong]

This error occurs in python==3.10.0, and not in upgraded python, such as 3.10.18, 3.12 etc.