change model

[slowmagic10]

can I use other nim models like llama-3.1 8b instead of 70b?

[omihub777]

@slowmagic10
Yes, you can

In case of using nvidia-hosted LLMs, you just need to set an environment variable by export APP_LLM_MODELNAME=meta/llama-3.1-8b-instruct before run docker compose up

For the self-hosted deployment, you additionally need to change the image on deploy/compose/nims.yaml:

services:
  nemollm-inference:
    container_name: nemollm-inference-microservice
    # image: nvcr.io/nim/meta/llama-3.1-70b-instruct:1.1 # comment out
   image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest # use 8b instead

Be aware that llama-3.1-8b might not be performant enough to run this blueprint successfully