Failed to build docker image due to missing cutlass/cutlass.h

[CoolFish88]

System Info

Hello,

I am trying to build the TEI 1.6 docker image on a GPU machine (A10G), whose OS is describe by the following parameters:

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Rust version: 1.85.1

The reason I am trying to built in from scratch, instead of fetching it using:
docker pull ghcr.io/huggingface/text-embeddings-inference:1.6, is to benefit from the improvements brought by this recent PR: #559

As I intend to deploy the TEI container on a Sagemaker endpoint, I am using Dockerfile-cuda-all, which references a sagemaker entry point script, as opposed to Dockerfile-cuda.

When I run: docker build . -f Dockerfile-cuda-all --build-arg CUDA_COMPUTE_CAP=86
I get the error:

error: failed to run custom build command for candle-flash-attn-v1 v0.0.1 (/usr/src/candle-extensions/candle-flash-attn-v1) Caused by: process didn't exit successfully: /usr/src/target/release/build/candle-flash-attn-v1-74cefb017284d1d2/build-script-build` (exit status: 1)

With the stderr printing:

2.920 --- stderr
2.920 #$ NVVM_BRANCH=nvvm
2.920 #$ SPACE=
2.920 #$ CUDART=cudart
2.920 #$ HERE=/usr/local/cuda/bin
2.920 #$ THERE=/usr/local/cuda/bin
2.920 #$ TARGET_SIZE=
2.920 #$ TARGET_DIR=
2.920 #$ TARGET_DIR=targets/x86_64-linux
2.920 #$ TOP=/usr/local/cuda/bin/..
2.920 #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
2.920 #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/src/target/release/deps:/usr/src/target/release:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2.920 #$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/root/.cargo/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2.920 #$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"
2.920 #$ NVVM_BRANCH=nvvm
2.920 #$ LIBRARIES= "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
2.920 #$ SPACE=
2.920 #$ CUDAFE_FLAGS=
2.920 #$ CUDART=cudart
2.920 #$ PTXAS_FLAGS=
2.920 #$ HERE=/usr/local/cuda/bin
2.920 #$ THERE=/usr/local/cuda/bin
2.920 #$ TARGET_SIZE=
2.920 #$ TARGET_DIR=
2.920 #$ TARGET_DIR=targets/x86_64-linux
2.920 #$ gcc -std=c++17 -D__CUDA_ARCH_LIST__=750 -E -x c++ -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__ -O3 -I"cutlass/include" "-I/usr/local/cuda/bin/../targets/x86_64-linux/include" -U "CUDA_NO_HALF_OPERATORS" -U "CUDA_NO_HALF_CONVERSIONS" -U "CUDA_NO_HALF2_OPERATORS" -U "CUDA_NO_BFLOAT16_CONVERSIONS" -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=2 -D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12 -D__CUDA_API_VER_MINOR__=2 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "kernels/flash_api.cu" -o "/tmp/tmpxft_00000507_00000000-5_flash_api.cpp4.ii"
2.920 #$ TOP=/usr/local/cuda/bin/..
2.920 #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
2.920 #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/src/target/release/deps:/usr/src/target/release:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2.920 #$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/root/.cargo/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2.920 #$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"
2.920 #$ LIBRARIES= "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
2.920 #$ CUDAFE_FLAGS=
2.920 #$ PTXAS_FLAGS=
2.920 #$ gcc -std=c++17 -D__CUDA_ARCH_LIST__=750 -E -x c++ -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__ -O3 -I"cutlass/include" "-I/usr/local/cuda/bin/../targets/x86_64-linux/include" -U "CUDA_NO_HALF_OPERATORS" -U "CUDA_NO_HALF_CONVERSIONS" -U "CUDA_NO_HALF2_OPERATORS" -U "CUDA_NO_BFLOAT16_CONVERSIONS" -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=2 -D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12 -D__CUDA_API_VER_MINOR__=2 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "kernels/fmha_fwd_hdim128.cu" -o "/tmp/tmpxft_00000508_00000000-5_fmha_fwd_hdim128.cpp4.ii"
2.920 #$ NVVM_BRANCH=nvvm
2.920 #$ SPACE=
2.920 #$ CUDART=cudart
2.920 #$ HERE=/usr/local/cuda/bin
2.920 #$ THERE=/usr/local/cuda/bin
2.920 #$ TARGET_SIZE=
2.920 #$ TARGET_DIR=
2.920 #$ TARGET_DIR=targets/x86_64-linux
2.920 #$ TOP=/usr/local/cuda/bin/..
2.920 #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
2.920 #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/src/target/release/deps:/usr/src/target/release:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2.920 #$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/root/.cargo/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2.920 #$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"
2.920 #$ LIBRARIES= "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
2.920 #$ NVVM_BRANCH=nvvm
2.920 #$ CUDAFE_FLAGS=
2.920 #$ PTXAS_FLAGS=
2.920 #$ SPACE=
2.920 #$ CUDART=cudart
2.920 #$ HERE=/usr/local/cuda/bin
2.920 #$ THERE=/usr/local/cuda/bin
2.920 #$ TARGET_SIZE=
2.920 #$ TARGET_DIR=
2.920 #$ TARGET_DIR=targets/x86_64-linux
2.920 #$ gcc -std=c++17 -D__CUDA_ARCH_LIST__=750 -E -x c++ -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__ -O3 -I"cutlass/include" "-I/usr/local/cuda/bin/../targets/x86_64-linux/include" -U "CUDA_NO_HALF_OPERATORS" -U "CUDA_NO_HALF_CONVERSIONS" -U "CUDA_NO_HALF2_OPERATORS" -U "CUDA_NO_BFLOAT16_CONVERSIONS" -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=2 -D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12 -D__CUDA_API_VER_MINOR__=2 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "kernels/fmha_fwd_hdim32.cu" -o "/tmp/tmpxft_00000509_00000000-5_fmha_fwd_hdim32.cpp4.ii"
2.920 #$ TOP=/usr/local/cuda/bin/..
2.920 #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
2.920 #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/src/target/release/deps:/usr/src/target/release:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2.920 #$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/root/.cargo/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2.920 #$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"
2.920 #$ LIBRARIES= "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
2.920 #$ CUDAFE_FLAGS=
2.920 #$ PTXAS_FLAGS=
2.920 #$ gcc -std=c++17 -D__CUDA_ARCH_LIST__=750 -E -x c++ -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__ -O3 -I"cutlass/include" "-I/usr/local/cuda/bin/../targets/x86_64-linux/include" -U "CUDA_NO_HALF_OPERATORS" -U "CUDA_NO_HALF_CONVERSIONS" -U "CUDA_NO_HALF2_OPERATORS" -U "CUDA_NO_BFLOAT16_CONVERSIONS" -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=2 -D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12 -D__CUDA_API_VER_MINOR__=2 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "kernels/fmha_fwd_hdim64.cu" -o "/tmp/tmpxft_0000050a_00000000-5_fmha_fwd_hdim64.cpp4.ii"
2.920 In file included from kernels/fmha/smem_tile.h:32,
2.920 from kernels/fmha_kernel.h:34,
2.920 from kernels/fmha_fprop_kernel_1xN.h:31,
2.920 from kernels/fmha_fwd_launch_template.h:12,
2.920 from kernels/fmha_fwd_hdim64.cu:5:
2.920 kernels/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
2.920 32 | #include "cutlass/cutlass.h"
2.920 | ^~~~~~~~~~~~~~~~~~~
2.920 compilation terminated.
2.920 In file included from kernels/fmha/smem_tile.h:32,
2.920 from kernels/fmha_kernel.h:34,
2.920 from kernels/fmha_fprop_kernel_1xN.h:31,
2.920 from kernels/fmha_fwd_launch_template.h:12,
2.920 from kernels/fmha_fwd_hdim128.cu:5:
2.920 kernels/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
2.920 32 | #include "cutlass/cutlass.h"
2.920 | ^~~~~~~~~~~~~~~~~~~
2.920 compilation terminated.
2.920 In file included from kernels/fmha/smem_tile.h:32,
2.920 from kernels/fmha_kernel.h:34,
2.920 from kernels/fmha_fprop_kernel_1xN.h:31,
2.920 from kernels/fmha_fwd_launch_template.h:12,
2.920 from kernels/fmha_fwd_hdim32.cu:5:
2.920 kernels/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
2.920 32 | #include "cutlass/cutlass.h"
2.920 | ^~~~~~~~~~~~~~~~~~~
2.920 compilation terminated.
2.920 #$ cudafe++ --c++17 --gnu_version=110400 --display_error_number --orig_src_file_name "kernels/flash_api.cu" --orig_src_path_name "/usr/src/candle-extensions/candle-flash-attn-v1/kernels/flash_api.cu" --allow_managed --extended-lambda --relaxed_constexpr --m64 --parse_templates --gen_c_file_name "/tmp/tmpxft_00000507_00000000-6_flash_api.cudafe1.cpp" --stub_file_name "tmpxft_00000507_00000000-6_flash_api.cudafe1.stub.c" --gen_module_id_file --module_id_file_name "/tmp/tmpxft_00000507_00000000-4_flash_api.module_id" "/tmp/tmpxft_00000507_00000000-5_flash_api.cpp4.ii"
2.920 # --error 0x1 --
2.920 # --error 0x1 --
2.920 # --error 0x1 --
2.920 #$ gcc -std=c++17 -D__CUDA_ARCH__=750 -D__CUDA_ARCH_LIST__=750 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__ -O3 -I"cutlass/include" "-I/usr/local/cuda/bin/../targets/x86_64-linux/include" -U "CUDA_NO_HALF_OPERATORS" -U "CUDA_NO_HALF_CONVERSIONS" -U "CUDA_NO_HALF2_OPERATORS" -U "CUDA_NO_BFLOAT16_CONVERSIONS" -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=2 -D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12 -D__CUDA_API_VER_MINOR__=2 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "kernels/flash_api.cu" -o "/tmp/tmpxft_00000507_00000000-7_flash_api.cpp1.ii"
2.920 #$ cicc --c++17 --gnu_version=110400 --display_error_number --orig_src_file_name "kernels/flash_api.cu" --orig_src_path_name "/usr/src/candle-extensions/candle-flash-attn-v1/kernels/flash_api.cu" --allow_managed --extended-lambda --relaxed_constexpr -arch compute_75 -m64 --no-version-ident -ftz=1 -prec_div=0 -prec_sqrt=0 -fmad=1 -fast-math --gen_div_approx_ftz --include_file_name "tmpxft_00000507_00000000-3_flash_api.fatbin.c" -tused --module_id_file_name "/tmp/tmpxft_00000507_00000000-4_flash_api.module_id" --gen_c_file_name "/tmp/tmpxft_00000507_00000000-6_flash_api.cudafe1.c" --stub_file_name "/tmp/tmpxft_00000507_00000000-6_flash_api.cudafe1.stub.c" --gen_device_file_name "/tmp/tmpxft_00000507_00000000-6_flash_api.cudafe1.gpu" "/tmp/tmpxft_00000507_00000000-7_flash_api.cpp1.ii" -o "/tmp/tmpxft_00000507_00000000-6_flash_api.ptx"
2.920 #$ ptxas -arch=sm_75 -m64 -v "/tmp/tmpxft_00000507_00000000-6_flash_api.ptx" -o "/tmp/tmpxft_00000507_00000000-8_flash_api.sm_75.cubin"
2.920 ptxas info : 0 bytes gmem
2.920 #$ fatbinary -64 --cmdline="-v " --cicc-cmdline="-ftz=1 -prec_div=0 -prec_sqrt=0 -fmad=1 " "--image3=kind=elf,sm=75,file=/tmp/tmpxft_00000507_00000000-8_flash_api.sm_75.cubin" "--image3=kind=ptx,sm=75,file=/tmp/tmpxft_00000507_00000000-6_flash_api.ptx" --embedded-fatbin="/tmp/tmpxft_00000507_00000000-3_flash_api.fatbin.c"
2.920 #$ rm /tmp/tmpxft_00000507_00000000-3_flash_api.fatbin
2.920 #$ gcc -std=c++17 -D__CUDA_ARCH__=750 -D__CUDA_ARCH_LIST__=750 -c -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -O3 -I"cutlass/include" "-I/usr/local/cuda/bin/../targets/x86_64-linux/include" -m64 "/tmp/tmpxft_00000507_00000000-6_flash_api.cudafe1.cpp" -o "/usr/src/target/release/build/candle-flash-attn-v1-89935d9d0de9255d/out/kernels/flash_api.o"
2.920 Error: nvcc error while executing compiling: "nvcc" "-O3" "-std=c++17" "--gpu-architecture=sm_75" "-c" "-o" "/usr/src/target/release/build/candle-flash-attn-v1-89935d9d0de9255d/out/kernels/fmha_fwd_hdim64.o" "--default-stream" "per-thread" "-Icutlass/include" "-U__CUDA_NO_HALF_OPERATORS__" "-U__CUDA_NO_HALF_CONVERSIONS__" "-U__CUDA_NO_HALF2_OPERATORS__" "-U__CUDA_NO_BFLOAT16_CONVERSIONS__" "--expt-relaxed-constexpr" "--expt-extended-lambda" "--use_fast_math" "--ptxas-options=-v" "--verbose" "kernels/fmha_fwd_hdim64.cu"
2.920
2.920 # stdout
2.920
2.920
2.920 # stderr
2.920
2.920 warning: build failed, waiting for other jobs to finish...

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

docker build . -f Dockerfile-cuda-all --build-arg CUDA_COMPUTE_CAP=86

Expected behavior

Build the docker image without errors

[CoolFish88]

Noticed that part of the PR includes building and pushing the image: https://github.com/huggingface/text-embeddings-inference/actions/runs/14266259038

To avoid building the image from ground up, I did: docker pull ghcr.io/huggingface/text-embeddings-inference:latest
I deployed this image to a ml.g5.2xlarge machine (A10G GPU), noticing the following error: unexpected argument 'serve' found

[alvarobartt]

Hey @CoolFish88 not sure about the issue above I'll investigate, but just as a heads up, you should refrain from using latest as the tag since it usually won't point to the actual container that you need to pull/run, instead use either the SHA or the release tag; assuming that you want to run it on a NVIDIA device with a compute capability of 8.6 you should be using:

# pull the CUDA image supporting all the allowed compute capabilities
docker pull ghcr.io/huggingface/text-embeddings-inference:cuda-1.6.1
# or the specific container with only compatibility for the compute capability 8.6
docker pull ghcr.io/huggingface/text-embeddings-inference:86-1.6.1
# or from the latest tag
docker pull ghcr.io/huggingface/text-embeddings-inference:cuda-sha-7d4d9ec

In any case, I'll investigate the issue above, thanks for reporting 🤗

[CoolFish88]

Hello @alvarobartt ,

Thank you for looking into this.

Providing additional context for the use-case:

• Sagemaker SDK offers support to fetch TEI docker images up to version 1.4.0

from sagemaker.huggingface import get_huggingface_llm_image_uri
tei_image_uri = get_huggingface_llm_image_uri("huggingface-tei",session = sm_session ,version="1.4.0")

• Deploying newer versions requires fetching TEI images from the GitHub Container Registry

While deploying the jina-embeddings-v2-small-en works with TEI 1.4.0 (using the code below), when switching to ghcr.io/huggingface/text-embeddings-inference:86-1.6.1, it fails with the error reported above.

tei_image_uri = <image_uri>
emb_model = HuggingFaceModel(
    name="jina-deployment",
    role=role,
    sagemaker_session=<my_session>,
    image_uri=tei_image_uri,
    env={"HF_TASK": "feature-extraction",
         "HF_MODEL_ID": "jinaai/jina-embeddings-v2-small-en",
    },
)

emb_predictor = emb_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    endpoint_name="jina-embeddings-tei-new"
)

When supplying model artifacts to endpoints with restricted network connectivity ( as model files couldn't be fetched from the HF hub), the deployment of a TEI 1.4.0 container would fail due to a FlashBert backend being initialized as opposed to a FlashJinaBert backend, something that your recent PR #559 addresses.

I tried supplying model artifacts to the new container, and the deployment fails with the same error: unexpected argument 'serve' found

Let me know if I can assist with this issue!

[alvarobartt]

Hmm ok @CoolFish88 thanks for the details, so I'll investigate the issue on building the image with docker build . -f Dockerfile-cuda-all --build-arg CUDA_COMPUTE_CAP=86 as you shared above on one end, and then for the Python snippet that you provided below, note that the environment variable should be MODEL_ID not HF_MODEL_ID and HF_TASK is not required, as AFAIK those are only required for the default PyTorch DLC on AWS SageMaker, used on any model that's neither TGI nor TEI, whereas for the rest it's used.

For the AWS SageMaker issue could you provide more information? Thanks in advance 🤗

[alvarobartt]

I'll try to get TEI 1.6.1 released on AWS SageMaker soon, and then make the following release available as soon as possible too (including the JinaBERT fixes mentioned above), in the meantime, could you try with the container URI ghcr.io/huggingface/text-embeddings-inference:86-1.6.1 instead? That IMO should work, but if it doesn't please let me know! Also since this is not natively integrated within AWS SageMaker yet I may recommend you to deploy that without the HuggingFaceModel as that's intended for officially supported images, so that may also lead to weird issues. Then once we land the release on AWS SageMaker I'll keep you posted and share with you the updated snippet with the official support 🤗

[CoolFish88]

Hello @alvarobartt,

I tried in the past removing HF_MODEL_ID when providing model artifacts so that it doesn't reference the path where the files are copied locally ( /opt/ml/model ), but HuggingFaceModel is very picky about this parameter being present (absence of this parameter causes the deployment to fail).

Building locally the image using docker build . -f Dockerfile-cuda-all --build-arg CUDA_COMPUTE_CAP=86 was something I attempted before realizing your PR includes a step for building and pushing the container incorporating the latest changes. While not urgent, it would be useful to understand the root cause of the reported error, in case someone else will also attempt the local build.

I attempted deploying the container you mentioned using a different approach to the one leveraging HuggingFaceModel`. Specifically, I manually created a Sagemaker Model referencing the docker image and the model artifacts, together with a supporting Endpoint Configuration, then deployed the model using boto3 SDK. CloudWatch logs list the same error:

error:unexpected argument 'serve' found
Usage: text-embeddings-router [OPTIONS]
For more information, try '--help'.

[CoolFish88]

Hello @alvarobartt,

I experimented deploying the following two images on a ml.g5.2xlarge (A10G GPU) machine:

• ghcr.io/huggingface/text-embeddings-inference:cuda-1.6.1
• ghcr.io/huggingface/text-embeddings-inference:cuda-sha-7d4d9ec

In both cases, deployment fails with CloudWatch logs listing:

./entrypoint.sh: line 10: [: -eq: unary operator expected
./entrypoint.sh: line 13: [: too many arguments
./entrypoint.sh: line 16: [: -eq: unary operator expected
cuda compute cap is not supported

I am not sure if this error is related to the problem discussed in this thread.

[Narsil]

The error is most likely because cutlass wasn't pulled into tree before building. It's defined as a git submodule within flash-attn-v1.

What you probably need to do to fix is git submodule update --init in order to get it in tree because doing the docker build.
This is a new step that wasn't necessary before because those used to be hosted on github separately and cargo would do that automatically.

Did you follow any docks for building locally ? We need to update it for the submodule to be discoverable.
Modified here #567 but there could be other places.

[alvarobartt]

Hey again @CoolFish88, do you mind moving the AWS SageMaker issue into a separate GitHub Issue? Thanks in advance 🤗

[CoolFish88]

Hello @alvarobartt and @Narsil,

@alvarobartt: I will create a separate issue for AWS Sagemaker
@Narsil: I used the custom_container.md guideline

[alvarobartt]

Thank you @CoolFish88 🤗 Indeed @Narsil has already updated the README.md to include the git submodule update --init instruction and I've just ported that to the file that you are pointing to at #568, so the documentation should be updated soon! Thanks for flagging!