You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hub/billing.md
+9-2Lines changed: 9 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Billing
2
2
3
-
At Hugging Face, we build a collaboration platform for the ML community (i.e., the Hub) and monetize by providing simple access to compute for AI.
3
+
At Hugging Face, we build a collaboration platform for the ML community (i.e., the Hub) and monetize by providing advanced features and simple access to compute for AI.
4
4
5
5
Any feedback or support request related to billing is welcome at [email protected]
6
6
@@ -61,13 +61,14 @@ You can view invoices and receipts for the last 3 months in your billing dashboa
61
61
62
62
## Enterprise Hub subscriptions
63
63
64
-
We offer advanced security and compliance features for organizations through our Enterprise Hub subscription, including [Single Sign-On](./enterprise-sso.md), [Advanced Access Control](./enterprise-hub-resource-groups.md) for repositories, control over your data location, and more.
64
+
We offer advanced security and compliance features for organizations through our Enterprise Hub subscription, including [Single Sign-On](./enterprise-sso.md), [Advanced Access Control](./enterprise-hub-resource-groups.md) for repositories, control over your data location, higher [storage capacity](./storage-limits.md) for private repositories, and more.
65
65
66
66
The Enterprise Hub is billed like a typical subscription. It renews automatically, but you can choose to cancel it at any time in the organization's billing settings.
67
67
68
68
You can pay for the Enterprise Hub subscription with a credit card or your AWS account.
69
69
70
70
Upon renewal, the number of seats in your Enterprise Hub subscription will be updated to match the number of members of your organization.
71
+
Private repository storage above the [included storage](./storage-limits.md) will be billed along with your subscription renewal.
71
72
72
73
73
74
<divclass="flex justify-center">
@@ -80,6 +81,7 @@ Upon renewal, the number of seats in your Enterprise Hub subscription will be up
80
81
The PRO subscription unlocks additional features for users, including:
81
82
82
83
- Higher free tier for the Serverless Inference API and when consuming ZeroGPU Spaces
84
+
- Higher [storage capacity](./storage-limits.md) for private repositories
83
85
- Ability to create ZeroGPU Spaces and use Dev Mode
84
86
- Ability to write Social Posts and Community Blogs
85
87
- Leverage the Dataset Viewer on private datasets
@@ -89,5 +91,10 @@ View the full list of benefits at https://huggingface.co/subscribe/pro
89
91
Similarly to the Enterprise Hub subscription, PRO subscriptions are billed like a typical subscription. The subscription renews automatically for you. You can choose to cancel the subscription at anytime in your billing settings: https://huggingface.co/settings/billing
90
92
91
93
You can only pay for the PRO subscription with a credit card. The subscription is billed separately from any pay-as-you-go compute usage.
94
+
Private repository storage above the [included storage](./storage-limits.md) will be billed along with your subscription renewal.
92
95
93
96
Note: PRO benefits are also included in the Enterprise Hub subscription.
97
+
98
+
## Pay-as-you-go private storage
99
+
100
+
Above the included 1TB (or 1TB per seat) of private storage in PRO and Enterprise Hub, private storage is invoiced at **$25/TB/month**, in 1TB increments.
Copy file name to clipboardExpand all lines: docs/hub/datasets-adding.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -111,4 +111,4 @@ The Hugging Face Hub supports large scale datasets, usually uploaded in Parquet
111
111
112
112
You can upload large scale datasets at high speed using the `huggingface_hub` library.
113
113
114
-
See [how to upload a folder by chunks](/docs/huggingface_hub/guides/upload#upload-a-folder-by-chunks), the [tips and tricks for large uploads](/docs/huggingface_hub/guides/upload#tips-and-tricks-for-large-uploads) and the [repository limitations and recommendations](./repositories-recommendations).
114
+
See [how to upload a folder by chunks](/docs/huggingface_hub/guides/upload#upload-a-folder-by-chunks), the [tips and tricks for large uploads](/docs/huggingface_hub/guides/upload#tips-and-tricks-for-large-uploads) and the [repository storage limits and recommendations](./storage-limits).
Finally, Enterprise Hub includes 1TB of [private repository storage](./storage-limits) per seat in the subscription, i.e. if your organization has 40 members, then you have 40TB included storage for your private models and datasets.
Copy file name to clipboardExpand all lines: docs/hub/storage-limits.md
+42-8Lines changed: 42 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,33 @@
1
-
# Repository limitations and recommendations
1
+
# Storage limits
2
2
3
-
There are some limitations to be aware of when dealing with a large amount of data in your repo. Given the time it takes to stream the data,
4
-
getting an upload/push to fail at the end of the process or encountering a degraded experience, be it on hf.co or when working locally, can be very annoying.
3
+
At Hugging Face our intent is to provide the AI community with **free storage space for public repositories**. We do bill for storage space for **private repositories**, above a free tier (see table below).
5
4
6
-
## Recommendations
5
+
We [optimize our infrastructure](https://huggingface.co/blog/xethub-joins-hf) continuously to [scale our storage](https://x.com/julien_c/status/1821540661973160339) for the coming years of growth in Machine learning.
6
+
7
+
We do have mitigations in place to prevent abuse of free public storage, and in general we ask users and organizations to make sure any uploaded large model or dataset is **as useful to the community as possible** (as represented by numbers of likes or downloads, for instance).
8
+
9
+
## Storage plans
10
+
11
+
| Type of account | Public storage | Private storage |
💡 Enterprise Hub includes 1TB of private storage per seat in the subscription: for example, if your organization has 40 members, then you have 40TB of included private storage.
18
+
19
+
*We aim to continue providing the AI community with free storage space for public repositories, please don't abuse and upload dozens of TBs of generated anime. If possible, we still ask that you consider upgrading to PRO and/or Enterprise Hub whenever possible.
20
+
21
+
### Pay-as-you-go price
22
+
23
+
Above the included 1TB (or 1TB per seat) of private storage in PRO and Enterprise Hub, private storage is invoiced at **$25/TB/month**, in 1TB increments. See our [billing doc](./billing) for more details.
24
+
25
+
## Repository limitations and recommendations
26
+
27
+
In parallel to storage limits at the account (user or organization) level, there are some limitations to be aware of when dealing with a large amount of data in a specific repo. Given the time it takes to stream the data,
28
+
getting an upload/push to fail at the end of the process or encountering a degraded experience, be it on hf.co or when working locally, can be very annoying. In the following section, we describe our recommendations on how to best structure your large repos.
29
+
30
+
### Recommendations
7
31
8
32
We gathered a list of tips and recommendations for structuring your repo. If you are looking for more practical tips, check out [this guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#tips-and-tricks-for-large-uploads) on how to upload large amount of data using the Python library.
9
33
@@ -21,7 +45,7 @@ _* Not relevant when using `git` CLI directly_
21
45
22
46
Please read the next section to understand better those limits and how to deal with them.
23
47
24
-
## Explanations
48
+
###Explanations
25
49
26
50
What are we talking about when we say "large uploads", and what are their associated limitations? Large uploads can be
27
51
very diverse, from repositories with a few huge files (e.g. model weights) to repositories with thousands of small files
@@ -31,9 +55,9 @@ Under the hood, the Hub uses Git to version the data, which has structural impli
31
55
If your repo is crossing some of the numbers mentioned in the previous section, **we strongly encourage you to check out [`git-sizer`](https://github.com/github/git-sizer)**,
32
56
which has very detailed documentation about the different factors that will impact your experience. Here is a TL;DR of factors to consider:
33
57
34
-
-**Repository size**: The total size of the data you're planning to upload. We generally support repositories up to 300GB. If you would like to upload more than 300 GBs (or even TBs) of data, you will need to ask us to grant more storage. To do that, please send an email with details of your project to [email protected].
58
+
-**Repository size**: The total size of the data you're planning to upload. We generally support repositories up to 300GB. If you would like to upload more than 300 GBs (or even TBs) of data, you will need to ask us to grant more storage. To do that, please send an email with details of your project to [email protected] (for datasets) or [email protected] (for models).
35
59
-**Number of files**:
36
-
- For optimal experience, we recommend keeping the total number of files under 100k. Try merging the data into fewer files if you have more.
60
+
- For optimal experience, we recommend keeping the total number of files under 100k, and ideally much less. Try merging the data into fewer files if you have more.
37
61
For example, json files can be merged into a single jsonl file, or large datasets can be exported as Parquet files or in [WebDataset](https://github.com/webdataset/webdataset) format.
38
62
- The maximum number of files per folder cannot exceed 10k files per folder. A simple solution is to
39
63
create a repository structure that uses subdirectories. For example, a repo with 1k folders from `000/` to `999/`, each containing at most 1000 files, is already enough.
@@ -57,7 +81,7 @@ happen (in rare cases) that even if the timeout is raised client-side, the proce
57
81
completed server-side. This can be checked manually by browsing the repo on the Hub. To prevent this timeout, we recommend
58
82
adding around 50-100 files per commit.
59
83
60
-
## Sharing large datasets on the Hub
84
+
###Sharing large datasets on the Hub
61
85
62
86
One key way Hugging Face supports the machine learning ecosystem is by hosting datasets on the Hub, including very large ones. However, if your dataset is bigger than 300GB, you will need to ask us to grant more storage.
63
87
@@ -78,3 +102,13 @@ For hosting large datasets on the Hub, we require the following for your dataset
78
102
- Avoid the use of custom loading scripts when using datasets. In our experience, datasets that require custom code to use often end up with limited reuse.
79
103
80
104
Please get in touch with us if any of these requirements are difficult for you to meet because of the type of data or domain you are working in.
105
+
106
+
### Sharing large volumes of models on the Hub
107
+
108
+
Similarly to datasets, if you host models bigger than 300GB or if you plan on uploading a large number of smaller sized models (for instance, hundreds of automated quants) totalling more than 1TB, you will need to ask us to grant more storage.
109
+
110
+
To do that, to ensure we can effectively support the open-source ecosystem, please send an email with details of your project to [email protected].
111
+
112
+
### Grants for private repositories
113
+
114
+
If you need more model/ dataset storage than your allocated private storage for academic/ research purposes, please reach out to us at [email protected] or [email protected] along with a proposal of how you will use the storage grant.
0 commit comments