unable to use VllmServer without CUDA

[mkhludnev]

I need to use vLLM via REAST API. but llama-index-llms-vllm hooks up vllm core, which looks for GPU, which is obviously absent in my macbook. Are there alternative modules? or known workaround?

pip show -f llama-index-llms-vllm
Name: llama-index-llms-vllm
Version: 0.1.6

...
>>> from llama_index.llms.vllm import VllmServer
>>> llm = VllmServer(api_url="http://192.168.1.1:8000/v1/", max_new_tokens=100, temperature=0)
..
ModuleNotFoundError: No module named 'vllm'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../.venv/lib/python3.11/site-packages/llama_index/llms/vllm/base.py", line 326, in __init__
    super().__init__(
  File "/.../.venv/lib/python3.11/site-packages/llama_index/llms/vllm/base.py", line 148, in __init__
    raise ImportError(

...
ImportError: Could not import vllm python package. Please install it with `pip install vllm`.
...
$pip install vllm
          raise OSError('CUDA_HOME environment variable is not set. '
      OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

[dosubot[bot]]

Hey @mkhludnev! Great to see you back, navigating the adventurous world of LlamaIndex with us. Hope your code journey's been more of a smooth sail than a wild llama chase lately! 🦙💻

Based on the information provided, it appears that the llama-index-llms-vllm module you're trying to use has dependencies that require CUDA, specifically PyTorch and possibly NVIDIA TensorRT. These dependencies are used for GPU acceleration, which is why the module is looking for a GPU on your MacBook.

Unfortunately, there doesn't seem to be a version of the llama-index-llms-vllm module that doesn't require CUDA according to the information available in the LlamaIndex repository.

However, you might be able to work around this issue by using a version of PyTorch that doesn't require CUDA. PyTorch offers CPU-only versions of their package, which you can install with the following command:

pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

Please replace the version numbers with the ones that match your requirements. This command will install PyTorch, torchvision, and torchaudio without CUDA support.

After installing the CPU-only version of PyTorch, you should be able to install the vllm package without encountering the CUDA_HOME environment variable is not set error.

Please note that this workaround might not work if the llama-index-llms-vllm module or any of its other dependencies require CUDA. In that case, you might need to look for alternative modules or solutions that don't require CUDA.

For more information, you can refer to the PyTorch website and the LlamaIndex repository.

If you encounter any issues or need further assistance, please provide more details about your setup and the exact error messages you're seeing.

Sources

• llama-index-integrations/llms/llama-index-llms-nvidia-tensorrt/pyproject.toml

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mkhludnev]

Here's a workaround

$pip install  llama-index-llms-openai-like

from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(api_base="http://192.168.1.1:8000/v1/",model="mistral", api_key="bogus", max_new_tokens=100, temperature=0)
llm.complete("Paul Graham is?")

I'll try to come up with the fix for VllmServer in a few days.

[mkhludnev]

here we go. There a thing which worth to realise in the beginning. Vllm has two REST API:

1. vllm.entrypoints.api_server the client for it is llama-index-llms-vllm but it requires some fixes llms-vllm: fix VllmServer to work without CUDA-required vllm core. #12003
2. vllm.entrypoints.openai.api_server which is provided via Docker the client for it is llama-index-llms-openai-like