Can we deploy at Jetson Orin? #6
-
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi @notelainemade, thanks for the question! NeMo Guardrails itself doesn't require a GPU to run, it's currently all in CPU. So you could definitely try it in a Jetson, it just won't use the GPU directly. All of the LLM calls are to external hosted services. The one thing that might cause a problem on "slimmer" devices is that it uses SentenceTransformers to create an embedding index. Again, that's currently done in CPU. We're looking at other options for that embedding process, including using a local GPU if one is available. By the way, we've now enabled GitHub Discussions as a good place to ask questions like this. I'm going to close the issue, but I'll convert it to a Discussion in case you have any other questions. |
Beta Was this translation helpful? Give feedback.
Hi @notelainemade, thanks for the question!
NeMo Guardrails itself doesn't require a GPU to run, it's currently all in CPU. So you could definitely try it in a Jetson, it just won't use the GPU directly. All of the LLM calls are to external hosted services.
The one thing that might cause a problem on "slimmer" devices is that it uses SentenceTransformers to create an embedding index. Again, that's currently done in CPU. We're looking at other options for that embedding process, including using a local GPU if one is available.
By the way, we've now enabled GitHub Discussions as a good place to ask questions like this. I'm going to close the issue, but I'll convert it to a Discussion in cas…