From 966308a4ac3f1c218f0c9b030908dba1cb93df48 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 11:43:06 +0100 Subject: [PATCH 01/11] Add L40S to Managed Inference models --- .../reference-content/llama-3-8b-instruct.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pages/managed-inference/reference-content/llama-3-8b-instruct.mdx b/pages/managed-inference/reference-content/llama-3-8b-instruct.mdx index d018be991c..a43b272906 100644 --- a/pages/managed-inference/reference-content/llama-3-8b-instruct.mdx +++ b/pages/managed-inference/reference-content/llama-3-8b-instruct.mdx @@ -18,7 +18,7 @@ categories: | Attribute | Details | |-----------------|------------------------------------| | Provider | [Meta](https://llama.meta.com/llama3/) | -| Compatible Instances | L4, H100 (FP8, BF16) | +| Compatible Instances | L4, L40S, H100, H100-2 (FP8, BF16) | | Context size | 8192 tokens | ## Model names @@ -33,7 +33,9 @@ meta/llama-3-8b-instruct:fp8 | Instance type | Max context length | | ------------- |-------------| | L4 | 8192 (FP8, BF16) | -| H100 | 8192 (FP8, BF16) +| L40S | 8192 (FP8, BF16) | +| H100 | 8192 (FP8, BF16) | +| H100-2 | 8192 (FP8, BF16) | ## Model introduction From fb62ad273f2e027140298261b70f46e30493db67 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 11:44:49 +0100 Subject: [PATCH 02/11] Update llama-3-70b-instruct.mdx --- .../reference-content/llama-3-70b-instruct.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pages/managed-inference/reference-content/llama-3-70b-instruct.mdx b/pages/managed-inference/reference-content/llama-3-70b-instruct.mdx index 1844efb25e..a0c5c9ce68 100644 --- a/pages/managed-inference/reference-content/llama-3-70b-instruct.mdx +++ b/pages/managed-inference/reference-content/llama-3-70b-instruct.mdx @@ -18,7 +18,7 @@ categories: | Attribute | Details | |-----------------|------------------------------------| | Provider | [Meta](https://llama.meta.com/llama3/) | -| Compatible Instances | H100 (FP8) | +| Compatible Instances | H100, H100-2 (FP8) | | Context size | 8192 tokens | ## Model names @@ -30,6 +30,7 @@ meta/llama-3-70b-instruct:fp8 ## Compatible Instances - [H100 (FP8)](https://www.scaleway.com/en/h100-pcie-try-it-now/) +- H100-2 (FP8) ## Model introduction @@ -82,4 +83,4 @@ Process the output data according to your application's needs. The response will Despite efforts for accuracy, the possibility of generated text containing inaccuracies or [hallucinations](/managed-inference/concepts/#hallucinations) exists. Always verify the content generated independently. - \ No newline at end of file + From 285f294a5701d0de83fa07058796decb5045754f Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 11:49:28 +0100 Subject: [PATCH 03/11] Update llama-3.1-8b-instruct.mdx --- .../reference-content/llama-3.1-8b-instruct.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx b/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx index 0318375220..31457edcf4 100644 --- a/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx +++ b/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx @@ -19,7 +19,7 @@ categories: |-----------------|------------------------------------| | Provider | [Meta](https://llama.meta.com/llama3/) | | License | [Llama 3.1 community](https://llama.meta.com/llama3_1/license/) | -| Compatible Instances | L4, H100, H100-2 (FP8, BF16) | +| Compatible Instances | L4, L40S, H100, H100-2 (FP8, BF16) | | Context Length | up to 128k tokens | ## Model names @@ -34,6 +34,7 @@ meta/llama-3.1-8b-instruct:bf16 | Instance type | Max context length | | ------------- |-------------| | L4 | 96k (FP8), 27k (BF16) | +| L40S | 96k (FP8), 27k (BF16) | | H100 | 128k (FP8, BF16) | H100-2 | 128k (FP8, BF16) @@ -82,4 +83,4 @@ Process the output data according to your application's needs. The response will Despite efforts for accuracy, the possibility of generated text containing inaccuracies or [hallucinations](/managed-inference/concepts/#hallucinations) exists. Always verify the content generated independently. - \ No newline at end of file + From 7732c1ddf0caa88657fe9b6342d4c5e5146f57f8 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 11:50:33 +0100 Subject: [PATCH 04/11] Update deepseek-r1-distill-llama-8b.mdx --- .../reference-content/deepseek-r1-distill-llama-8b.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx b/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx index bde51b9140..f2efd7887d 100644 --- a/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx +++ b/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx @@ -19,7 +19,7 @@ categories: |-----------------|------------------------------------| | Provider | [Deepseek](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B) | | License | [MIT](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md) | -| Compatible Instances | L4, H100 (BF16) | +| Compatible Instances | L4, L40S, H100 (BF16) | | Context Length | up to 131k tokens | ## Model names @@ -33,6 +33,7 @@ deepseek/deepseek-r1-distill-llama-8b:bf16 | Instance type | Max context length | | ------------- |-------------| | L4 | 39k (BF16) | +| L40S | 39k (BF16) | | H100 | 131k (BF16) | ## Model introduction From 90ee8cf6dbb66a845f60b29092130beeca32903c Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 11:59:07 +0100 Subject: [PATCH 05/11] Update pixtral-12b-2409.mdx --- .../reference-content/pixtral-12b-2409.mdx | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/pages/managed-inference/reference-content/pixtral-12b-2409.mdx b/pages/managed-inference/reference-content/pixtral-12b-2409.mdx index 2c22572c72..61974cd34e 100644 --- a/pages/managed-inference/reference-content/pixtral-12b-2409.mdx +++ b/pages/managed-inference/reference-content/pixtral-12b-2409.mdx @@ -17,9 +17,9 @@ categories: | Attribute | Details | |-----------------|------------------------------------| -| Provider | [Mistral](https://mistral.ai/technology/#models) | -| Compatible Instances | H100, H100-2 (bf16) | -| Context size | 128k tokens | +| Provider | [Mistral](https://mistral.ai/technology/#models) | +| Compatible Instances | L40S, H100, H100-2 (bf16) | +| Context size | 128k tokens | ## Model name @@ -31,6 +31,7 @@ mistral/pixtral-12b-2409:bf16 | Instance type | Max context length | | ------------- |-------------| +| L40S | 128k (BF16) | H100 | 128k (BF16) | H100-2 | 128k (BF16) @@ -162,4 +163,4 @@ Only bitmaps can be analyzed by Pixtral, PDFs and videos are not supported. The only limitation is in context window (1 token for each 16x16 pixel). #### What is the maximum amount of images per conversation? -One conversation can handle up to 12 images (per request). The 13rd will return a 413 error. \ No newline at end of file +One conversation can handle up to 12 images (per request). The 13rd will return a 413 error. From d07ff400962a2085e3540950bbc5cd97f47d94b6 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 12:00:38 +0100 Subject: [PATCH 06/11] Update mistral-7b-instruct-v0.3.mdx --- .../reference-content/mistral-7b-instruct-v0.3.mdx | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/pages/managed-inference/reference-content/mistral-7b-instruct-v0.3.mdx b/pages/managed-inference/reference-content/mistral-7b-instruct-v0.3.mdx index 0e299bab0e..be1d2d4f2d 100644 --- a/pages/managed-inference/reference-content/mistral-7b-instruct-v0.3.mdx +++ b/pages/managed-inference/reference-content/mistral-7b-instruct-v0.3.mdx @@ -17,8 +17,8 @@ categories: | Attribute | Details | |-----------------|------------------------------------| -| Provider | [Mistral](https://mistral.ai/technology/#models) | -| Compatible Instances | L4 (BF16) | +| Provider | [Mistral](https://mistral.ai/technology/#models) | +| Compatible Instances | L4, L40S, H100, H100-2 (BF16) | | Context size | 32K tokens | ## Model name @@ -31,7 +31,10 @@ mistral/mistral-7b-instruct-v0.3:bf16 | Instance type | Max context length | | ------------- |-------------| -| L4 | 32k (BF16) +| L4 | 32k (BF16) | +| L40S | 32k (BF16) | +| H100 | 32k (BF16) | +| H100-2 | 32k (BF16) | ## Model introduction @@ -75,4 +78,4 @@ Process the output data according to your application's needs. The response will Despite efforts for accuracy, the possibility of generated text containing inaccuracies or [hallucinations](/managed-inference/concepts/#hallucinations) exists. Always verify the content generated independently. - \ No newline at end of file + From 6f31d0b6ac2b2295dddac43befbc213fb855b1d5 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 13:56:45 +0100 Subject: [PATCH 07/11] Update mistral-nemo-instruct-2407.mdx --- .../reference-content/mistral-nemo-instruct-2407.mdx | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/pages/managed-inference/reference-content/mistral-nemo-instruct-2407.mdx b/pages/managed-inference/reference-content/mistral-nemo-instruct-2407.mdx index e7d8e3c2a2..ccf76d3d0e 100644 --- a/pages/managed-inference/reference-content/mistral-nemo-instruct-2407.mdx +++ b/pages/managed-inference/reference-content/mistral-nemo-instruct-2407.mdx @@ -17,9 +17,9 @@ categories: | Attribute | Details | |-----------------|------------------------------------| -| Provider | [Mistral](https://mistral.ai/technology/#models) | -| Compatible Instances | H100 (FP8) | -| Context size | 128K tokens | +| Provider | [Mistral](https://mistral.ai/technology/#models) | +| Compatible Instances | L40S, H100, H100-2 (FP8) | +| Context size | 128K tokens | ## Model name @@ -31,7 +31,9 @@ mistral/mistral-nemo-instruct-2407:fp8 | Instance type | Max context length | | ------------- |-------------| -| H100 | 128k (FP8) +| L40 | 128k (FP8) | +| H100 | 128k (FP8) | +| H100-2 | 128k (FP8) | ## Model introduction @@ -81,4 +83,4 @@ Process the output data according to your application's needs. The response will Despite efforts for accuracy, the possibility of generated text containing inaccuracies or [hallucinations](/managed-inference/concepts/#hallucinations) exists. Always verify the content generated independently. - \ No newline at end of file + From 4fe565778412fb03b12e605292a6fbb3b050a947 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 14:01:01 +0100 Subject: [PATCH 08/11] Update pixtral-12b-2409.mdx --- pages/managed-inference/reference-content/pixtral-12b-2409.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/managed-inference/reference-content/pixtral-12b-2409.mdx b/pages/managed-inference/reference-content/pixtral-12b-2409.mdx index 61974cd34e..4899e8a3d6 100644 --- a/pages/managed-inference/reference-content/pixtral-12b-2409.mdx +++ b/pages/managed-inference/reference-content/pixtral-12b-2409.mdx @@ -31,7 +31,7 @@ mistral/pixtral-12b-2409:bf16 | Instance type | Max context length | | ------------- |-------------| -| L40S | 128k (BF16) +| L40S | 50k (BF16) | H100 | 128k (BF16) | H100-2 | 128k (BF16) From dc98cd17151853f0f812577a605faaa1877309ea Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 14:01:53 +0100 Subject: [PATCH 09/11] Update llama-3.1-8b-instruct.mdx --- .../reference-content/llama-3.1-8b-instruct.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx b/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx index 31457edcf4..a6d24ab250 100644 --- a/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx +++ b/pages/managed-inference/reference-content/llama-3.1-8b-instruct.mdx @@ -34,9 +34,9 @@ meta/llama-3.1-8b-instruct:bf16 | Instance type | Max context length | | ------------- |-------------| | L4 | 96k (FP8), 27k (BF16) | -| L40S | 96k (FP8), 27k (BF16) | -| H100 | 128k (FP8, BF16) -| H100-2 | 128k (FP8, BF16) +| L40S | 128k (FP8, BF16) | +| H100 | 128k (FP8, BF16) | +| H100-2 | 128k (FP8, BF16) | ## Model introduction From 1deba2b6c56db081df0cfd9c7a87cacbf39a7a42 Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 14:02:57 +0100 Subject: [PATCH 10/11] Update deepseek-r1-distill-llama-8b.mdx --- .../reference-content/deepseek-r1-distill-llama-8b.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx b/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx index f2efd7887d..dd9919bc93 100644 --- a/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx +++ b/pages/managed-inference/reference-content/deepseek-r1-distill-llama-8b.mdx @@ -33,7 +33,7 @@ deepseek/deepseek-r1-distill-llama-8b:bf16 | Instance type | Max context length | | ------------- |-------------| | L4 | 39k (BF16) | -| L40S | 39k (BF16) | +| L40S | 131k (BF16) | | H100 | 131k (BF16) | ## Model introduction From 3d1c553f1cdc894c90e1a8da24f7e3ae585d838f Mon Sep 17 00:00:00 2001 From: fpagny Date: Mon, 10 Feb 2025 14:03:36 +0100 Subject: [PATCH 11/11] Update bge-multilingual-gemma2.mdx --- .../reference-content/bge-multilingual-gemma2.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pages/managed-inference/reference-content/bge-multilingual-gemma2.mdx b/pages/managed-inference/reference-content/bge-multilingual-gemma2.mdx index 3e51e1b013..885eaeef94 100644 --- a/pages/managed-inference/reference-content/bge-multilingual-gemma2.mdx +++ b/pages/managed-inference/reference-content/bge-multilingual-gemma2.mdx @@ -18,7 +18,7 @@ dates: | Attribute | Details | |-----------------|------------------------------------| | Provider | [baai](https://huggingface.co/BAAI) | -| Compatible Instances | L4 (FP32) | +| Compatible Instances | L4, L40S (FP32) | | Context size | 4096 tokens | ## Model name @@ -32,6 +32,7 @@ baai/bge-multilingual-gemma2:fp32 | Instance type | Max context length | | ------------- |-------------| | L4 | 4096 (FP32) | +| L40S | 4096 (FP32) | ## Model introduction