|
1 | 1 | # Inference Providers |
2 | 2 |
|
3 | | -Please refer to the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers) for detailed information. |
| 3 | +Hugging Face's model pages have pay-as-you-go inference for thousands of models, so you can try them all out right in the browser. Service is powered by Inference Providers and includes a free-tier. |
4 | 4 |
|
5 | | -## What is HF-Inference API? |
| 5 | +Inference Providers give developers streamlined, unified access to hundreds of machine learning models, powered by the best serverless inference partners. 👉 **For complete documentation, visit the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers)**. |
6 | 6 |
|
7 | | -HF-Inference API is one of the many providers available on the Hugging Face Hub. |
8 | | -It is deployed by Hugging Face ourselves, using text-generation-inference for LLMs for instance. This service used to be called “Inference API (serverless)” prior to Inference Providers. |
| 7 | +## Inference Providers on the Hub |
9 | 8 |
|
10 | | -For more details about the HF-Inference API, check out its [dedicated page](https://huggingface.co/docs/inference-providers/providers/hf-inference). |
| 9 | +Inference Providers is deeply integrated with the Hugging Face Hub, and you can use it in a few different ways: |
11 | 10 |
|
12 | | -## What technology do you use to power the HF-Inference API? |
| 11 | +- **Interactive Widgets** - Test models directly on model pages with interactive widgets that use Inference Providers under the hood. Check out the [DeepSeek-R1-0528 model page](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) for an example. |
| 12 | +- **Inference Playground** - Easily test and compare chat completion models with your prompts. Check out the [Inference Playground](https://huggingface.co/playground) to get started. |
| 13 | +- **Search** - Filter models by inference provider on the [models page](https://huggingface.co/models?inference_provider=all) to find models available through specific providers. |
| 14 | +- **Data Studio** - Use AI to explore datasets on the Hub. Check out [Data Studio](https://huggingface.co/datasets/fka/awesome-chatgpt-prompts/viewer?views%5B%5D=train) on your favorite dataset. |
13 | 15 |
|
14 | | -The HF-Inference API is powered by [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index) under the hood. |
| 16 | +## Build with Inference Providers |
15 | 17 |
|
16 | | -## Why don't I see an inference widget, or why can't I use the API? |
| 18 | +You can integrate Inference Providers into your own applications using our SDKs or HTTP clients. Here's a quick start with Python and JavaScript, for more details, check out the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers). |
17 | 19 |
|
18 | | -For some tasks, there might not be support by any Inference Provider, and hence, there is no widget. |
| 20 | +<hfoptions id="inference-providers-quick-start"> |
19 | 21 |
|
20 | | -## How can I see my usage? |
| 22 | +<hfoption id="python"> |
21 | 23 |
|
22 | | -To check usage across all providers, check out your [billing page](https://huggingface.co/settings/billing). |
| 24 | +You can use our Python SDK to interact with Inference Providers. |
23 | 25 |
|
24 | | -To check your HF-Inference usage specifically, check out the [Inference Dashboard](https://ui.endpoints.huggingface.co/endpoints). The dashboard shows both your serverless and dedicated endpoints usage. |
| 26 | +```python |
| 27 | +from huggingface_hub import InferenceClient |
25 | 28 |
|
26 | | -## Is there programmatic access to Inference Providers? |
| 29 | +import os |
27 | 30 |
|
28 | | -Yes! We provide client wrappers in both JS and Python: |
29 | | -- [JS (`@huggingface/inference`)](https://huggingface.co/docs/huggingface.js/inference/classes/InferenceClient) |
30 | | -- [Python (`huggingface_hub`)](https://huggingface.co/docs/huggingface_hub/guides/inference) |
| 31 | +client = InferenceClient( |
| 32 | + api_key=os.environ["HF_TOKEN"], |
| 33 | + provider="auto", # Automatically selects best provider |
| 34 | +) |
| 35 | + |
| 36 | +# Chat completion |
| 37 | +completion = client.chat.completions.create( |
| 38 | + model="deepseek-ai/DeepSeek-V3-0324", |
| 39 | + messages=[{"role": "user", "content": "A story about hiking in the mountains"}] |
| 40 | +) |
| 41 | + |
| 42 | +# Image generation |
| 43 | +image = client.text_to_image( |
| 44 | + prompt="A serene lake surrounded by mountains at sunset, photorealistic style", |
| 45 | + model="black-forest-labs/FLUX.1-dev" |
| 46 | +) |
| 47 | + |
| 48 | +``` |
| 49 | + |
| 50 | +Or, you can just use the OpenAI API compatible client. |
| 51 | + |
| 52 | +```python |
| 53 | +import os |
| 54 | +from openai import OpenAI |
| 55 | + |
| 56 | +client = OpenAI( |
| 57 | + base_url="https://router.huggingface.co/v1", |
| 58 | + api_key=os.environ["HF_TOKEN"], |
| 59 | +) |
| 60 | + |
| 61 | +completion = client.chat.completions.create( |
| 62 | + model="deepseek-ai/DeepSeek-V3-0324", |
| 63 | + messages=[ |
| 64 | + { |
| 65 | + "role": "user", |
| 66 | + "content": "A story about hiking in the mountains" |
| 67 | + } |
| 68 | + ], |
| 69 | +) |
| 70 | +``` |
| 71 | + |
| 72 | +<Tip warning={true}> |
| 73 | + |
| 74 | +The OpenAI API compatible client is not supported for image generation. |
| 75 | + |
| 76 | +</Tip> |
| 77 | + |
| 78 | +</hfoption> |
| 79 | + |
| 80 | +<hfoption id="javascript"> |
| 81 | + |
| 82 | +You can use our JavaScript SDK to interact with Inference Providers. |
| 83 | + |
| 84 | +```javascript |
| 85 | +import { InferenceClient } from "@huggingface/inference"; |
| 86 | + |
| 87 | +const client = new InferenceClient(process.env.HF_TOKEN); |
| 88 | + |
| 89 | +const chatCompletion = await client.chatCompletion({ |
| 90 | + provider: "auto", // Automatically selects best provider |
| 91 | + model: "deepseek-ai/DeepSeek-V3-0324", |
| 92 | + messages: [{ role: "user", content: "Hello!" }] |
| 93 | +}); |
| 94 | + |
| 95 | +const imageBlob = await client.textToImage({ |
| 96 | + model: "black-forest-labs/FLUX.1-dev", |
| 97 | + inputs: |
| 98 | + "A serene lake surrounded by mountains at sunset, photorealistic style", |
| 99 | +}); |
| 100 | +``` |
| 101 | + |
| 102 | +Or, you can just use the OpenAI API compatible client. |
| 103 | + |
| 104 | +```javascript |
| 105 | +import { OpenAI } from "openai"; |
| 106 | + |
| 107 | +const client = new OpenAI({ |
| 108 | + baseURL: "https://router.huggingface.co/v1", |
| 109 | + apiKey: process.env.HF_TOKEN, |
| 110 | +}); |
| 111 | + |
| 112 | +const completion = await client.chat.completions.create({ |
| 113 | + model: "meta-llama/Llama-3.1-8B-Instruct", |
| 114 | + messages: [{ role: "user", content: "A story about hiking in the mountains" }], |
| 115 | +}); |
| 116 | + |
| 117 | +``` |
| 118 | + |
| 119 | +<Tip warning={true}> |
| 120 | + |
| 121 | +The OpenAI API compatible client is not supported for image generation. |
| 122 | + |
| 123 | +</Tip> |
| 124 | + |
| 125 | +</hfoption> |
| 126 | + |
| 127 | +</hfoptions> |
| 128 | + |
| 129 | +You'll need a Hugging Face token with inference permissions. Create one at [Settings > Tokens](https://huggingface.co/settings/tokens/new?ownUserPermissions=inference.serverless.write&tokenType=fineGrained). |
| 130 | + |
| 131 | +### How Inference Providers works |
| 132 | + |
| 133 | +To dive deeper into Inference Providers, check out the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers). Here are some key resources: |
| 134 | + |
| 135 | +- **[Quick Start](https://huggingface.co/docs/inference-providers)** |
| 136 | +- **[Pricing & Billing Guide](https://huggingface.co/docs/inference-providers/pricing)** |
| 137 | +- **[Hub Integration Details](https://huggingface.co/docs/inference-providers/hub-integration)** |
| 138 | + |
| 139 | +### What was the HF-Inference API? |
| 140 | + |
| 141 | +HF-Inference API is one of the providers available through Inference Providers. It was previously called "Inference API (serverless)" and is powered by [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index) under the hood. |
| 142 | + |
| 143 | +For more details about the HF-Inference provider specifically, check out its [dedicated page](https://huggingface.co/docs/inference-providers/providers/hf-inference). |
0 commit comments