You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Docs for integrating data libraries and tools with Hub (#1574)
* Draft documentation for integrating data libraries and tools with the Hub
* move to existing page
* toc tree
* Update docs/hub/datasets-libraries.md
Co-authored-by: Julien Chaumond <[email protected]>
---------
Co-authored-by: Julien Chaumond <[email protected]>
|[Argilla](./datasets-argilla)| Collaboration tool for AI engineers and domain experts that value high quality data. | ✅ | ✅ |
12
-
|[Dask](./datasets-dask)| Parallel and distributed computing library that scales the existing Python and PyData ecosystem. | ✅ | ✅ |
13
-
|[Datasets](./datasets-usage)| 🤗 Datasets is a library for accessing and sharing datasets for Audio, Computer Vision, and Natural Language Processing (NLP). | ✅ | ✅ |
14
-
|[Distilabel](./datasets-distilabel)| The framework for synthetic data generation and AI feedback. | ✅ | ✅ |
|[Argilla](./datasets-argilla)| Collaboration tool for AI engineers and domain experts that value high quality data. | ✅ | ✅ |
12
+
|[Dask](./datasets-dask)| Parallel and distributed computing library that scales the existing Python and PyData ecosystem. | ✅ | ✅ |
13
+
|[Datasets](./datasets-usage)| 🤗 Datasets is a library for accessing and sharing datasets for Audio, Computer Vision, and Natural Language Processing (NLP). | ✅ | ✅ |
14
+
|[Distilabel](./datasets-distilabel)| The framework for synthetic data generation and AI feedback. | ✅ | ✅ |
|[FiftyOne](./datasets-fiftyone)| FiftyOne is a library for curation and visualization of image, video, and 3D data. | ✅ | ✅ |
17
+
|[Pandas](./datasets-pandas)| Python data analysis toolkit. | ✅ | ✅ |
18
+
|[Polars](./datasets-polars)| A DataFrame library on top of an OLAP query engine. | ✅ | ✅ |
19
+
|[Spark](./datasets-spark)| Real-time, large-scale data processing tool in a distributed environment. | ✅ | ✅ |
20
+
|[WebDataset](./datasets-webdataset)| Library to write I/O pipelines for large datasets. | ✅ | ❌ |
21
+
22
+
## Integrating data libraries and tools with the Hub
23
+
24
+
This guide is designed for developers and maintainers of data libraries and tools who want to integrate with the Hugging Face Hub. Whether you're building a data processing library, analysis tool, or any software that needs to interact with datasets, this documentation will help you implement a Hub integration.
25
+
26
+
The guide covers:
27
+
28
+
- Possible approaches to loading data from the Hub into your library/tool
29
+
- Possible approaches to uploading data from your library/tool to the Hub
30
+
31
+
### Loading data from the Hub
32
+
33
+
If you have a library for working with data, it can be helpful for your users to load data from the Hub.
34
+
35
+
In general, we suggest relying on an existing library like `datasets`, `pandas` or `polars` to do this unless you have a specific reason to implement your own. If you require more control over the loading process, you can use the `huggingface_hub` library, which will allow you, for example, to download a specific subset of files from a repository.
36
+
37
+
You can find more information about loading data from the Hub [here](https://huggingface.co/docs/hub/datasets-downloading).
38
+
39
+
#### Integrating via the Dataset Viewer and Parquet Files
40
+
41
+
The Hub's dataset viewer and Parquet conversion system provide a standardized way to integrate with datasets, regardless of their original format. This infrastructure is a reliable integration layer between the Hub and external libraries.
42
+
43
+
If the dataset is not already in Parquet, the Hub automatically converts the first 5GB of every dataset to Parquet format to power the dataset viewer and provide consistent access patterns. This standardization offers several benefits for library integrations:
44
+
45
+
- Consistent data access patterns regardless of original format
46
+
- Built-in dataset preview and exploration through the Hub's dataset viewer. The dataset viewer can also be embedded as an iframe in your applications, making it easy to provide rich dataset previews. For more information about embedding the viewer, see the [dataset viewer embedding documentation](https://huggingface.co/docs/hub/en/datasets-viewer-embed).
47
+
- Efficient columnar storage optimized for querying. For example, you could use a tool like [DuckDB](https://duckdb.org/) to query or filter for a specific subset of data.
48
+
- Parquet is well supported across the machine learning and data science ecosystem.
49
+
50
+
For more details on working with the Dataset Viewer API, see the [Dataset Viewer API documentation](https://huggingface.co/docs/dataset-viewer/index)
51
+
52
+
### Uploading data to the Hub
53
+
54
+
This section covers possible approaches for adding the ability to upload data to the Hub in your library, i.e. how to implement a `push_to_hub` method.
55
+
56
+
This guide will cover three primary ways to upload data to the Hub:
57
+
58
+
- using the `datasets` library and the `push_to_hub` method
59
+
- using `pandas` to write to the Hub
60
+
- using the `huggingface_hub` library and the `hf_hub_download` method
61
+
- directly using the API or Git LFS
62
+
63
+
#### Use the `datasets` library
64
+
65
+
The most straightforward approach to pushing data to the Hub is to rely on the existing [`push_to_hub`](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.Dataset.push_to_hub) method from the `datasets` library. The `push_to_hub` method will automatically handle:
66
+
67
+
- the creation of the repository
68
+
- the conversion of the dataset to Parquet
69
+
- chunking the dataset into suitable parts
70
+
- uploading the data
71
+
72
+
For example, if you have a synthetic data generation library that returns a list of dictionaries, you could simply do the following:
73
+
74
+
```python
75
+
from datasets import Dataset
76
+
77
+
data = [{"prompt": "Write a cake recipe", "response": "Measure 1 cup ..."}]
#### Rely on an existing libraries integration with the Hub
87
+
88
+
Polars, Pandas, Dask, Spark and DuckDB all can write to a Hugging Face Hub repository. See [datasets libraries](https://huggingface.co/docs/hub/datasets-libraries) for more details.
89
+
90
+
If you are already using one of these libraries in your code, adding the ability to push to the Hub is straightforward. For example, if you have a synthetic data generation library that can return a Pandas DataFrame, here is the code you would need to write to the Hub:
The `huggingface_hub` Python library offers a more flexible approach to uploading data to the Hub. The library allows you to upload specific files or subsets of files to a repository. This is useful if you have a large dataset that you don't want to convert to Parquet, want to upload a specific subset of files, or want more control over the repo structure.
108
+
109
+
Depending on your use case, you can upload a file or folder at a specific point in your code, i.e., export annotations from a tool to the Hub when a user clicks "push to Hub". For example,
110
+
111
+
```python
112
+
from huggingface_hub import HfApi
113
+
api = HfApi(token=HF_TOKEN)
114
+
115
+
api.upload_folder(
116
+
folder_path="/my-cool-library/data-folder",
117
+
repo_id="username/my-cool-space",
118
+
repo_type="dataset",
119
+
commit_message="Push annotations to Hub"
120
+
allow_patterns="*.jsonl",
121
+
)
122
+
```
123
+
124
+
You can find more information about ways to upload data to the Hub [here](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload).
125
+
126
+
Alternatively, there are situations where you may want to upload data in the background, for example, synthetic data being generated every 10 minutes. In this case you can use the `scheduled_uploads` feature of the `huggingface_hub` library. For more details, see the [scheduled uploads documentation](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#scheduled-uploads).
127
+
128
+
You can see an example of using this approach to upload data to the Hub in
129
+
130
+
- The [fastdata](https://github.com/AnswerDotAI/fastdata/blob/main/nbs/00_core.ipynb) library
131
+
- This [magpie](https://huggingface.co/spaces/davanstrien/magpie/blob/fc79672c740b8d3d098378dca37c0f191c208de0/app.py#L67) Demo Space
132
+
133
+
## More support
134
+
135
+
For technical questions about integration, feel free to contact the datasets team at [email protected].
0 commit comments