Skip to content

Commit 35884ae

Browse files
lhoestqseveroWauplinjulien-cmishig25
authored
Dataset Viewer, Structure and Libraries docs (#1070)
* more datasets docs * add configure your dataset * minor * minor * update toc * minor * add dataset structure docs * rename sections * Apply suggestions from code review Co-authored-by: Sylvain Lesage <[email protected]> Co-authored-by: Lucain <[email protected]> Co-authored-by: Julien Chaumond <[email protected]> * sylvain's comments: Adding a new dataset * sylvain-s comments: Configure the Dataset Viewer * sylvain's comments: File names and splits * sylvain's comments: Manual Configuration * sylvain's comments: Libraries * sylvain's comment: login * sylvain's comments: Using 🤗 Datasets * lucain's comment: Adding a new dataset * add duckdb write * add create repo step for dask and pandas * minor * minor * typo * Update docs/hub/_toctree.yml Co-authored-by: Mishig <[email protected]> * Apply suggestions from code review Co-authored-by: Daniel van Strien <[email protected]> * rename titles for consistency * Add Downloading Datasets * Apply suggestions from code review Co-authored-by: Polina Kazakova <[email protected]> * more links to list of supported formats * move supported file formats to upload page * fix link to row * fix links * again * remove duplicate * move configure viewer to bottom of page * fix to_parquet example * minor fix * Update docs/hub/datasets-file-names-and-splits.md Co-authored-by: Lucain <[email protected]> * Apply suggestions from code review Co-authored-by: Polina Kazakova <[email protected]> --------- Co-authored-by: Sylvain Lesage <[email protected]> Co-authored-by: Lucain <[email protected]> Co-authored-by: Julien Chaumond <[email protected]> Co-authored-by: Mishig <[email protected]> Co-authored-by: Daniel van Strien <[email protected]> Co-authored-by: Polina Kazakova <[email protected]>
1 parent 5d0b187 commit 35884ae

15 files changed

+710
-30
lines changed

‎docs/hub/_toctree.yml‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -127,12 +127,35 @@
127127
title: Dataset Cards
128128
- local: datasets-gated
129129
title: Gated Datasets
130+
- local: datasets-adding
131+
title: Uploading Datasets
132+
- local: datasets-downloading
133+
title: Downloading Datasets
134+
- local: datasets-libraries
135+
title: Integrated Libraries
136+
sections:
137+
- local: datasets-dask
138+
title: Dask
139+
- local: datasets-usage
140+
title: Datasets
141+
- local: datasets-duckdb
142+
title: DuckDB
143+
- local: datasets-pandas
144+
title: Pandas
145+
- local: datasets-webdataset
146+
title: WebDataset
130147
- local: datasets-viewer
131148
title: Dataset Viewer
132-
- local: datasets-usage
133-
title: Using Datasets
134-
- local: datasets-adding
135-
title: Adding New Datasets
149+
sections:
150+
- local: datasets-viewer-configure
151+
title: Configure the Dataset Viewer
152+
- local: datasets-data-files-configuration
153+
title: Data files Configuration
154+
sections:
155+
- local: datasets-file-names-and-splits
156+
title: File names and splits
157+
- local: datasets-manual-configuration
158+
title: Manual Configuration
136159
- local: spaces
137160
title: Spaces
138161
isExpanded: true

‎docs/hub/datasets-adding.md‎

Lines changed: 95 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,100 @@
1-
# Adding new datasets
1+
# Uploading datasets
22

3-
Any Hugging Face user can create a dataset! You can start by [creating your dataset repository](https://huggingface.co/new-dataset) and choosing one of the following methods to upload your dataset:
3+
The [Hub](https://huggingface.co/datasets) is home to an extensive collection of community-curated and research datasets. We encourage you to share your dataset to the Hub to help grow the ML community and accelerate progress for everyone. All contributions are welcome; adding a dataset is just a drag and drop away!
44

5-
* [Add files manually to the repository through the UI](https://huggingface.co/docs/datasets/upload_dataset#upload-your-files)
6-
* [Push files with the `push_to_hub` method from 🤗 Datasets](https://huggingface.co/docs/datasets/upload_dataset#upload-from-python)
7-
* [Use Git to commit and push your dataset files](https://huggingface.co/docs/datasets/share#clone-the-repository)
5+
Start by [creating a Hugging Face Hub account](https://huggingface.co/join) if you don't have one yet.
86

9-
While in many cases it's possible to just add raw data to your dataset repo in any supported formats (JSON, CSV, Parquet, text, images, audio files, …), for some large datasets you may want to [create a loading script](https://huggingface.co/docs/datasets/dataset_script#create-a-dataset-loading-script). This script defines the different configurations and splits of your dataset, as well as how to download and process the data.
7+
## Upload using the Hub UI
108

11-
## Datasets outside a namespace
9+
The Hub's web-based interface allows users without any developer experience to upload a dataset.
1210

13-
Datasets outside a namespace are maintained by the Hugging Face team. Unlike the naming convention used for community datasets (`username/dataset_name` or `org/dataset_name`), datasets outside a namespace can be referenced directly by their name (e.g. [`glue`](https://huggingface.co/datasets/glue)). If you find that an improvement is needed, use their "Community" tab to open a discussion or submit a PR on the Hub to propose edits.
11+
### Create a repository
12+
13+
A repository hosts all your dataset files, including the revision history, making storing more than one dataset version possible.
14+
15+
1. Click on your profile and select **New Dataset** to create a [new dataset repository](https://huggingface.co/new-dataset).
16+
2. Pick a name for your dataset, and choose whether it is a public or private dataset. A public dataset is visible to anyone, whereas a private dataset can only be viewed by you or members of your organization.
17+
18+
<div class="flex justify-center">
19+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/create_repo.png"/>
20+
</div>
21+
22+
### Upload dataset
23+
24+
1. Once you've created a repository, navigate to the **Files and versions** tab to add a file. Select **Add file** to upload your dataset files. We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others (see full list [here](./datasets-viewer-configure.md)).
25+
26+
<div class="flex justify-center">
27+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/upload_files.png"/>
28+
</div>
29+
30+
2. Drag and drop your dataset files.
31+
32+
<div class="flex justify-center">
33+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/commit_files.png"/>
34+
</div>
35+
36+
3. After uploading your dataset files, they are stored in your dataset repository.
37+
38+
<div class="flex justify-center">
39+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/files_stored.png"/>
40+
</div>
41+
42+
### Create a Dataset card
43+
44+
Adding a Dataset card is super valuable for helping users find your dataset and understand how to use it responsibly.
45+
46+
1. Click on **Create Dataset Card** to create a [Dataset card](./datasets-cards). This button creates a `README.md` file in your repository.
47+
48+
<div class="flex justify-center">
49+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/dataset_card.png"/>
50+
</div>
51+
52+
2. At the top, you'll see the **Metadata UI** with several fields to select from such as license, language, and task categories. These are the most important tags for helping users discover your dataset on the Hub (when applicable). When you select an option for a field, it will be automatically added to the top of the dataset card.
53+
54+
You can also look at the [Dataset Card specifications](https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1), which has a complete set of allowed tags, including optional like `annotations_creators`, to help you choose the ones that are useful for your dataset.
55+
56+
<div class="flex justify-center">
57+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/metadata_ui.png"/>
58+
</div>
59+
60+
3. Write your dataset documentation in the Dataset Card to introduce your dataset to the community and help users understand what is inside: what are the use cases and limitations, where the data comes from, what are important ethical considerations, and any other relevant details.
61+
62+
You can click on the **Import dataset card template** link at the top of the editor to automatically create a dataset card template. For a detailed example of what a good Dataset card should look like, take a look at the [CNN DailyMail Dataset card](https://huggingface.co/datasets/cnn_dailymail).
63+
64+
### Dataset Viewer
65+
66+
The [Dataset Viewer](./datasets-viewer) is useful to know how the data actually looks like before you download it.
67+
It is enabled by default for all public datasets.
68+
69+
Make sure the Dataset Viewer correctly shows your data, or [Configure the Dataset Viewer](./datasets-viewer-configure).
70+
71+
## Using the `huggingface_hub` client library
72+
73+
The rich features set in the `huggingface_hub` library allows you to manage repositories, including creating repos and uploading datasets to the Model Hub. Visit [the client library's documentation](https://huggingface.co/docs/huggingface_hub/index) to learn more.
74+
75+
## Using other libraries
76+
77+
Some libraries like [🤗 Datasets](https://huggingface.co/docs/datasets/index), [Pandas](https://pandas.pydata.org/), [Dask](https://www.dask.org/) or [DuckDB](https://duckdb.org/) can upload files to the Hub.
78+
See the list of [Libraries supported by the Datasets Hub](./datasets-libraries) for more information.
79+
80+
## Using Git
81+
82+
Since dataset repos are just Git repositories, you can use Git to push your data files to the Hub. Follow the guide on [Getting Started with Repositories](repositories-getting-started) to learn about using the `git` CLI to commit and push your datasets.
83+
84+
## File formats
85+
86+
The Hub natively supports multiple file formats:
87+
88+
- CSV (.csv, .tsv)
89+
- JSON Lines, JSON (.jsonl, .json)
90+
- Parquet (.parquet)
91+
- Text (.txt)
92+
- Images (.png, .jpg, etc.)
93+
- Audio (.wav, .mp3, etc.)
94+
95+
It also supports files compressed using ZIP (.zip), GZIP (.gz), ZSTD (.zst), BZ2 (.bz2), LZ4 (.lz4) and LZMA (.xz).
96+
97+
Image and audio resources can also have additional metadata files, see the [Data files Configuration](./datasets-data-files-configuration) on image and audio datasets.
98+
99+
You may want to convert your files to these formats to benefit from all the Hub features.
100+
Other formats and structures may not be recognized by the Hub.

‎docs/hub/datasets-dask.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Dask
2+
3+
[Dask](https://github.com/dask/dask) is a parallel and distributed computing library that scales the existing Python and PyData ecosystem.
4+
Since it uses [fsspec](https://filesystem-spec.readthedocs.io) to read and write remote data, you can use the Hugging Face paths (`hf://`) to read and write data on the Hub:
5+
6+
First you need to [Login with your Hugging Face account](../huggingface_hub/quick-start#login), for example using:
7+
8+
```
9+
huggingface-cli login
10+
```
11+
12+
Then you can [Create a dataset repository](../huggingface_hub/quick-start#create-a-repository), for example using:
13+
14+
```python
15+
from huggingface_hub import HfApi
16+
17+
HfApi().create_repo(repo_id="username/my_dataset", repo_type="dataset")
18+
```
19+
20+
Finally, you can use Hugging Face paths in Dask:
21+
22+
```python
23+
import dask.dataframe as dd
24+
25+
df.to_parquet("hf://datasets/username/my_dataset")
26+
27+
# or write in separate directories if the dataset has train/validation/test splits
28+
df_train.to_parquet("hf://datasets/username/my_dataset/train")
29+
df_valid.to_parquet("hf://datasets/username/my_dataset/validation")
30+
df_test .to_parquet("hf://datasets/username/my_dataset/test")
31+
```
32+
33+
This creates a dataset repository `username/my_dataset` containing your Dask dataset in Parquet format.
34+
You can reload it later:
35+
36+
```python
37+
import dask.dataframe as dd
38+
39+
df = dd.read_parquet("hf://datasets/username/my_dataset")
40+
41+
# or read from separate directories if the dataset has train/validation/test splits
42+
df_train = dd.read_parquet("hf://datasets/username/my_dataset/train")
43+
df_valid = dd.read_parquet("hf://datasets/username/my_dataset/validation")
44+
df_test = dd.read_parquet("hf://datasets/username/my_dataset/test")
45+
```
46+
47+
For more information on the Hugging Face paths and how they are implemented, please refer to the [the client library's documentation on the HfFileSystem](https://huggingface.co/docs/huggingface_hub/guides/hf_file_system).

‎docs/hub/datasets-data-files-configuration.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Data files Configuration
2+
3+
There are no constraints on how to structure dataset repositories.
4+
5+
However, if you want the Dataset Viewer to show certain data files, or to separate your dataset in train/validation/test splits, you need to structure your dataset accordingly.
6+
Often it is as simple as naming your data files according to their split names, e.g. `train.csv` and `test.csv`.
7+
8+
## File names and splits
9+
10+
To structure your dataset by naming your data files or directories according to their split names, see the [File names and splits](./datasets-file-names-and-splits) documentation.
11+
12+
## Manual configuration
13+
14+
You can choose the data files to show in the Dataset Viewer for your dataset using YAML.
15+
It is useful if you want to specify which file goes into which split manually.
16+
17+
You can also define multiple configurations (or subsets) for your dataset, and pass dataset building parameters (e.g. the separator to use for CSV files).
18+
19+
See the documentation on [Manual configuration](./datasets-manual-configuration) for more information.
20+
21+
## Image and Audio datasets
22+
23+
For image and audio classification datasets, you can also use directories to name the image and audio classes.
24+
And if your images/audio files have metadata (e.g. captions, bounding boxes, transcriptions, etc.), you can have metadata files next to them.
25+
26+
We provide two guides that you can check out:
27+
28+
- [How to create an image dataset](https://huggingface.co/docs/datasets/image_dataset)
29+
- [How to create an audio dataset](https://huggingface.co/docs/datasets/audio_dataset)

‎docs/hub/datasets-downloading.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Downloading datasets
2+
3+
## Integrated libraries
4+
5+
If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use in _Library_" button on the dataset page to see how to do so. For example, `samsum` shows how to do so with 🤗 Datasets below.
6+
7+
<div class="flex justify-center">
8+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage.png"/>
9+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-dark.png"/>
10+
</div>
11+
12+
<div class="flex justify-center">
13+
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-modal.png"/>
14+
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-modal-dark.png"/>
15+
</div>
16+
17+
## Using the Hugging Face Client Library
18+
19+
You can use the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) library to create, delete, update and retrieve information from repos. You can also download files from repos or integrate them into your library! For example, you can quickly load a CSV dataset with a few lines using Pandas.
20+
21+
```py
22+
from huggingface_hub import hf_hub_download
23+
import pandas as pd
24+
25+
REPO_ID = "YOUR_REPO_ID"
26+
FILENAME = "data.csv"
27+
28+
dataset = pd.read_csv(
29+
hf_hub_download(repo_id=REPO_ID, filename=FILENAME, repo_type="dataset")
30+
)
31+
```
32+
33+
## Using Git
34+
35+
Since all datasets on the dataset Hub are Git repositories, you can clone the datasets locally by running:
36+
37+
```bash
38+
git lfs install
39+
git clone [email protected]:datasets/<dataset ID> # example: git clone [email protected]:datasets/allenai/c4
40+
```
41+
42+
If you have write-access to the particular dataset repo, you'll also have the ability to commit and push revisions to the dataset.
43+
44+
Add your SSH public key to [your user settings](https://huggingface.co/settings/keys) to push changes and/or access private repos.

‎docs/hub/datasets-duckdb.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# DuckDB
2+
3+
[DuckDB](https://github.com/duckdb/duckdb) is an in-process SQL [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing) database management system.
4+
Since it supports [fsspec](https://filesystem-spec.readthedocs.io) to read and write remote data, you can use the Hugging Face paths (`hf://`) to read and write data on the Hub:
5+
6+
First you need to [Login with your Hugging Face account](../huggingface_hub/quick-start#login), for example using:
7+
8+
```
9+
huggingface-cli login
10+
```
11+
12+
Then you can [Create a dataset repository](../huggingface_hub/quick-start#create-a-repository), for example using:
13+
14+
```python
15+
from huggingface_hub import HfApi
16+
17+
HfApi().create_repo(repo_id="username/my_dataset", repo_type="dataset")
18+
```
19+
20+
Finally, you can use Hugging Face paths in DuckDB:
21+
22+
```python
23+
>>> from huggingface_hub import HfFileSystem
24+
>>> import duckdb
25+
26+
>>> fs = HfFileSystem()
27+
>>> duckdb.register_filesystem(fs)
28+
>>> duckdb.sql("COPY tbl TO 'hf://datasets/username/my_dataset/data.parquet' (FORMAT PARQUET);")
29+
```
30+
31+
This creates a file `data.parquet` in the dataset repository `username/my_dataset` containing your dataset in Parquet format.
32+
You can reload it later:
33+
34+
```python
35+
>>> from huggingface_hub import HfFileSystem
36+
>>> import duckdb
37+
38+
>>> fs = HfFileSystem()
39+
>>> duckdb.register_filesystem(fs)
40+
>>> df = duckdb.query("SELECT * FROM 'hf://datasets/username/my_dataset/data.parquet' LIMIT 10;").df()
41+
```

0 commit comments

Comments
 (0)