storage-limits

Author: julien-c

docs/hub/_redirects.yml

@@ -17,3 +17,4 @@ searching-the-hub: /docs/huggingface_hub/searching-the-hub
 api-webhook: webhooks
 adapter-transformers: adapters
 security-two-fa: security-2fa
+repositories-recommendations: storage-limits

docs/hub/_toctree.yml

@@ -25,8 +25,8 @@
         title: "How-to: Create automatic metadata quality reports"
   - local: notebooks
     title: Notebooks
-  - local: repositories-recommendations
-    title: Repository size recommendations
+  - local: storage-limits
+    title: Storage Limits
   - local: repositories-next-steps
     title: Next Steps
   - local: repositories-licenses

docs/hub/billing.md

@@ -1,6 +1,6 @@
 # Billing
 
-At Hugging Face, we build a collaboration platform for the ML community (i.e., the Hub) and monetize by providing simple access to compute for AI.
+At Hugging Face, we build a collaboration platform for the ML community (i.e., the Hub) and monetize by providing advanced features and simple access to compute for AI.
 
 Any feedback or support request related to billing is welcome at [email protected]
 

docs/hub/datasets-adding.md

@@ -111,4 +111,4 @@ The Hugging Face Hub supports large scale datasets, usually uploaded in Parquet
 
 You can upload large scale datasets at high speed using the `huggingface_hub` library.
 
-See [how to upload a folder by chunks](/docs/huggingface_hub/guides/upload#upload-a-folder-by-chunks), the [tips and tricks for large uploads](/docs/huggingface_hub/guides/upload#tips-and-tricks-for-large-uploads) and the [repository limitations and recommendations](./repositories-recommendations).
+See [how to upload a folder by chunks](/docs/huggingface_hub/guides/upload#upload-a-folder-by-chunks), the [tips and tricks for large uploads](/docs/huggingface_hub/guides/upload#tips-and-tricks-for-large-uploads) and the [repository storage limits and recommendations](./storage-limits).

docs/hub/enterprise-hub.md

@@ -23,3 +23,5 @@ In this section we will document the following Enterprise Hub features:
 - [Tokens Management](./enterprise-hub-tokens-management)
 - [Analytics](./enterprise-hub-analytics)
 - [Network Security](./enterprise-hub-network-security)
+
+Finally, Enterprise Hub includes 1TB of [private repository storage](./storage-limits) per seat in the subscription, i.e. if your organization has 40 members, then you have 40TB included storage for your private models and datasets.

docs/hub/other.md

@@ -6,6 +6,7 @@
   - [Managing Organizations](./organizations-managing)
   - [Organization Cards](./organizations-cards)
   - [Access control in organizations](./organizations-security)
+- [Enterprise Hub](./enterprise-hub)
 - [Moderation](./moderation)
 - [Billing](./billing)
 - [Digital Object Identifier (DOI)](./doi)

docs/hub/repositories.md

@@ -13,6 +13,6 @@ In these pages, you will go over the basics of getting started with Git and inte
 - [Webhooks](./webhooks)
 - [Notifications](./notifications)
 - [Collections](./collections)
-- [Repository size recommendations](./repositories-recommendations)
+- [Repository storage limits](./storage-limits)
 - [Next Steps](./repositories-next-steps)
 - [Licenses](./repositories-licenses)

docs/hub/repositories-recommendations.md renamed to docs/hub/storage-limits.md

@@ -1,9 +1,32 @@
-# Repository limitations and recommendations
+# Storage limits
 
-There are some limitations to be aware of when dealing with a large amount of data in your repo. Given the time it takes to stream the data,
-getting an upload/push to fail at the end of the process or encountering a degraded experience, be it on hf.co or when working locally, can be very annoying.
+At Hugging Face our intent is to provide the AI community with **free storage space for public models and datasets**. We do bill for storage space for **private repositories**, above a free tier (see table below).
 
-## Recommendations
+We [optimize our infrastructure](https://huggingface.co/blog/xethub-joins-hf) continuously to [scale our storage](https://x.com/julien_c/status/1821540661973160339) for the coming years of growth in Machine learning.
+
+We do have mitigations in place to prevent abuse of free public storage, and in general we ask users and organizations to make sure any uploaded large model or dataset is **as useful to the community as possible** (as represented by numbers of likes or downloads, for instance).
+
+## Storage plans
+
+| Type of account  | Public storage | Private storage              |
+| ---------------- | -------------- | ---------------------------- |
+| Free user or org | Unlimited ✅   | 100GB                        |
+| PRO              | Unlimited ✅   | 1TB + pay-as-you-go          |
+| Enterprise Hub   | Unlimited ✅   | 1TB per seat + pay-as-you-go |
+
+
+💡 Enterprise Hub includes 1TB of private storage per seat in the subscription: for example, if your organization has 40 members, then you have 40TB included private storage.
+
+### Pay-as-you-go price
+
+Above the included 1TB (or 1TB per seat) of private storage in PRO and Enterprise Hub, private storage is invoiced at **$25/TB/month**. See our [billing doc](./billing) for more details.
+
+## Repository limitations and recommendations
+
+In parallel to storage limits at the account (user or organization) level, there are some limitations to be aware of when dealing with a large amount of data in a specific repo. Given the time it takes to stream the data,
+getting an upload/push to fail at the end of the process or encountering a degraded experience, be it on hf.co or when working locally, can be very annoying. In the following section, we describe our recommendations on how to best structure your large repos.
+
+### Recommendations
 
 We gathered a list of tips and recommendations for structuring your repo. If you are looking for more practical tips, check out [this guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#tips-and-tricks-for-large-uploads) on how to upload large amount of data using the Python library.
 
@@ -21,7 +44,7 @@ _* Not relevant when using `git` CLI directly_
 
 Please read the next section to understand better those limits and how to deal with them.
 
-## Explanations
+### Explanations
 
 What are we talking about when we say "large uploads", and what are their associated limitations? Large uploads can be
 very diverse, from repositories with a few huge files (e.g. model weights) to repositories with thousands of small files
@@ -31,9 +54,9 @@ Under the hood, the Hub uses Git to version the data, which has structural impli
 If your repo is crossing some of the numbers mentioned in the previous section, **we strongly encourage you to check out [`git-sizer`](https://github.com/github/git-sizer)**,
 which has very detailed documentation about the different factors that will impact your experience. Here is a TL;DR of factors to consider:
 
-- **Repository size**: The total size of the data you're planning to upload. We generally support repositories up to 300GB. If you would like to upload more than 300 GBs (or even TBs) of data, you will need to ask us to grant more storage. To do that, please send an email with details of your project to [email protected].
+- **Repository size**: The total size of the data you're planning to upload. We generally support repositories up to 300GB. If you would like to upload more than 300 GBs (or even TBs) of data, you will need to ask us to grant more storage. To do that, please send an email with details of your project to [email protected] (for datasets) or [email protected] (for models).
 - **Number of files**:
-    - For optimal experience, we recommend keeping the total number of files under 100k. Try merging the data into fewer files if you have more.
+    - For optimal experience, we recommend keeping the total number of files under 100k, and ideally much less. Try merging the data into fewer files if you have more.
       For example, json files can be merged into a single jsonl file, or large datasets can be exported as Parquet files or in [WebDataset](https://github.com/webdataset/webdataset) format.
     - The maximum number of files per folder cannot exceed 10k files per folder. A simple solution is to
       create a repository structure that uses subdirectories. For example, a repo with 1k folders from `000/` to `999/`, each containing at most 1000 files, is already enough.
@@ -57,7 +80,7 @@ happen (in rare cases) that even if the timeout is raised client-side, the proce
 completed server-side. This can be checked manually by browsing the repo on the Hub. To prevent this timeout, we recommend
 adding around 50-100 files per commit.
 
-## Sharing large datasets on the Hub
+### Sharing large datasets on the Hub
 
 One key way Hugging Face supports the machine learning ecosystem is by hosting datasets on the Hub, including very large ones. However, if your dataset is bigger than 300GB, you will need to ask us to grant more storage.
 
@@ -78,3 +101,9 @@ For hosting large datasets on the Hub, we require the following for your dataset
 - Avoid the use of custom loading scripts when using datasets. In our experience, datasets that require custom code to use often end up with limited reuse.
 
 Please get in touch with us if any of these requirements are difficult for you to meet because of the type of data or domain you are working in.
+
+### Sharing large volumes of models on the Hub
+
+Similarly to datasets, if you host models bigger than 300GB or if you plan on uploading a large number of smaller sized models (for instance, hundreds of automated quants) totalling more than 1TB, you will need to ask us to grant more storage. 
+
+To do that, to ensure we can effectively support the open-source ecosystem, please send an email with details of your project to [email protected].