Update LLM section for winter kickstart#761
Conversation
|
TODO: Add a brief description (or link to instructions) how to get a Jupyter session on Triton. |
ruokolt
changed the title
triton/apps/llms.rst
Outdated
|
|
||
|
|
||
| Here is a Python script using huggingface model. | ||
| Below is a Python script using a HuggingFace model (``mistralai/Mistral-7B-v0.1``) for text generation. |
There was a problem hiding this comment.
We can use this one: https://github.com/AaltoSciComp/llm-examples/blob/main/huggingface-models/your_script.py, with a newer model, explicit dtype specification and device mapping.
There was a problem hiding this comment.
Looks good, let's use that script.
| - ``MODEL_WEIGHTS`` - Path to the model weights in GGUF file format. | ||
|
|
||
| This Python code snippet is part of a 'Chat with Your PDF Documents' example, utilizing LangChain and leveraging model weights stored in a .gguf file. For detailed environment setting up and Python code, please check out `this repo <https://github.com/AaltoSciComp/llm-examples/tree/main/chat-with-pdf>`__. | ||
| NOTE: this example repo is mainly meant to run on CPUs, if you want to run on GPUs, you can checkout a branch "llamacpp-gpu" of this repo for details. |
There was a problem hiding this comment.
Agree with the deletion, too old models. But maybe add a line: "If you need assistance running LLMs in formats other than HuggingFace's, please contact the RSEs.
triton/apps/llms.rst
Outdated
| .. group-tab:: jupyter notebook | ||
| .. group-tab:: Jupyter notebook | ||
|
|
||
| TODO: How to get Jupyter session with GPU on Triton? |
There was a problem hiding this comment.
Not sure if we recommend this, I had an example before: https://github.com/AaltoSciComp/triton-llm-tutorial/tree/main/triton-examples though.
triton/apps/llms.rst
Outdated
| Here is a Python script using huggingface model. | ||
| Below is a Python script using a HuggingFace model (``mistralai/Mistral-7B-v0.1``) for text generation. | ||
|
|
||
| Use it as the content of ``your_script.py`` in the Slurm script example above or a Jupyter notebook cell. |
There was a problem hiding this comment.
If we recommend the same content for jupyter notebook, a gpu session is needed.
2 participants
WIP.
Suggestion to make the LLM page more concise, removing obsolete sections (llama checkpoint sections).