The HuggingFace Transformers library supports a wide variety of NLP and vision models with a convenient API, and is used by many of the other LLM packages. There are a large number of models that it's compatible with on HuggingFace Hub.
Note
If you wish to use Transformer's integrated bitsandbytes quantization (load_in_8bit/load_in_4bit) or AutoGPTQ quantization, run these containers instead which include those respective libraries installed on top of Transformers:
auto_gptq(depends on Transformers)bitsandbytes(depends on Transformers)
Substitute the text-generation model that you want to run (it should be a CausalLM model like GPT, Llama, ect)
./run.sh $(./autotag transformers) \
huggingface-benchmark.py --model=gpt2If the model repository is private or requires authentication, add
--env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN>
By default, the performance is measured for generating 128 new output tokens (this can be set with --tokens=N)
The prompt can be changed with --prompt='your prompt here'
Use the --precision argument to enable quantization (options are: fp32 fp16 fp4 int8, default is: fp16)
If you're using fp4 or int8, run the bitsandbytes container as noted above, so that bitsandbytes package is installed to do the quantization. It's expected that 4-bit/8-bit quantization is slower through Transformers than FP16 (while consuming less memory) - see here for more info.
Other libraries like exllama, awq, and AutoGPTQ have custom CUDA kernels and more efficient quantized performance.
- First request access from https://ai.meta.com/llama/
- Then create a HuggingFace account, and request access to one of the Llama2 models there like https://huggingface.co/meta-llama/Llama-2-7b-hf (doing this will get you access to all the Llama2 models)
- Get a User Access Token from https://huggingface.co/settings/tokens
./run.sh --env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN> $(./autotag transformers) \
huggingface-benchmark.py --model=meta-llama/Llama-2-7b-hfCONTAINERS
transformers:git |
|
|---|---|
| Builds |  |
| Requires | L4T ['>=32.6'] |
| Dependencies | build-essential cuda cudnn python numpy cmake onnx pytorch:2.2 torchvision huggingface_hub rust |
| Dockerfile | Dockerfile |
| Images | dustynv/transformers:git-r35.2.1 (2023-12-15, 5.9GB)dustynv/transformers:git-r35.3.1 (2023-12-12, 5.9GB)dustynv/transformers:git-r35.4.1 (2023-12-11, 5.9GB) |
| Notes | bitsandbytes and auto_gptq dependencies added on JetPack5 for 4-bit/8-bit quantization |
transformers:nvgpt |
|
|---|---|
| Builds |  |
| Requires | L4T ['>=32.6'] |
| Dependencies | build-essential cuda cudnn python numpy cmake onnx pytorch:2.2 torchvision huggingface_hub rust |
| Dockerfile | Dockerfile |
| Images | dustynv/transformers:nvgpt-r35.2.1 (2023-12-05, 5.9GB)dustynv/transformers:nvgpt-r35.3.1 (2023-12-15, 5.9GB)dustynv/transformers:nvgpt-r35.4.1 (2023-12-14, 5.9GB) |
| Notes | bitsandbytes and auto_gptq dependencies added on JetPack5 for 4-bit/8-bit quantization |
CONTAINER IMAGES
| Repository/Tag | Date | Arch | Size |
|---|---|---|---|
dustynv/transformers:git-r35.2.1 |
2023-12-15 |
arm64 |
5.9GB |
dustynv/transformers:git-r35.3.1 |
2023-12-12 |
arm64 |
5.9GB |
dustynv/transformers:git-r35.4.1 |
2023-12-11 |
arm64 |
5.9GB |
dustynv/transformers:nvgpt-r35.2.1 |
2023-12-05 |
arm64 |
5.9GB |
dustynv/transformers:nvgpt-r35.3.1 |
2023-12-15 |
arm64 |
5.9GB |
dustynv/transformers:nvgpt-r35.4.1 |
2023-12-14 |
arm64 |
5.9GB |
dustynv/transformers:r32.7.1 |
2023-12-15 |
arm64 |
1.5GB |
dustynv/transformers:r35.2.1 |
2023-12-11 |
arm64 |
5.9GB |
dustynv/transformers:r35.3.1 |
2023-12-12 |
arm64 |
5.9GB |
dustynv/transformers:r35.4.1 |
2023-12-15 |
arm64 |
5.9GB |
dustynv/transformers:r36.2.0 |
2023-12-15 |
arm64 |
7.6GB |
Container images are compatible with other minor versions of JetPack/L4T:
• L4T R32.7 containers can run on other versions of L4T R32.7 (JetPack 4.6+)
• L4T R35.x containers can run on other versions of L4T R35.x (JetPack 5.1+)
RUN CONTAINER
To start the container, you can use jetson-containers run and autotag, or manually put together a docker run command:
# automatically pull or build a compatible container image
jetson-containers run $(autotag transformers)
# or explicitly specify one of the container images above
jetson-containers run dustynv/transformers:nvgpt-r35.3.1
# or if using 'docker run' (specify image and mounts/ect)
sudo docker run --runtime nvidia -it --rm --network=host dustynv/transformers:nvgpt-r35.3.1
jetson-containers runforwards arguments todocker runwith some defaults added (like--runtime nvidia, mounts a/datacache, and detects devices)
autotagfinds a container image that's compatible with your version of JetPack/L4T - either locally, pulled from a registry, or by building it.
To mount your own directories into the container, use the -v or --volume flags:
jetson-containers run -v /path/on/host:/path/in/container $(autotag transformers)To launch the container running a command, as opposed to an interactive shell:
jetson-containers run $(autotag transformers) my_app --abc xyzYou can pass any options to it that you would to docker run, and it'll print out the full command that it constructs before executing it.
BUILD CONTAINER
If you use autotag as shown above, it'll ask to build the container for you if needed. To manually build it, first do the system setup, then run:
jetson-containers build transformersThe dependencies from above will be built into the container, and it'll be tested during. Run it with --help for build options.