update #7067

aihao2000 · 2024-02-23T07:19:32Z

What does this PR do?

load_dataset data_dir
Supports custom local datasets. Just like add load_datasete data_dir parameter #5747 (comment)
Update fingerprint hash parameters. Avoid irrelevant parameters causing repeated calculations in the map cache

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul and @patrickvonplaten

sayakpaul · 2024-02-23T07:26:27Z

Thanks for your contributions! Can we please break this PR into multiple PRs as it targets multiple mutually exclusive areas?

aihao2000 · 2024-02-23T09:05:37Z

@sayakpaul Yes, feel free to change it.

sayakpaul · 2024-02-23T09:17:54Z

Sorry for not being clear. I was asking you to tackle this in multiple PRs

aihao2000 · 2024-02-26T16:38:50Z

@sayakpaul I'm not familiar with git and it seems a little troublesome. Are there any doubtful changes? I can restore that part.

sayakpaul · 2024-02-26T16:41:12Z

It's a general good practice to not introduce multiple changes in a single PR. And I would like to follow that. For example, for each of the points you enlisted in the PR description, you could create PRs (four PRs).

github-actions · 2024-03-24T15:03:05Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

sayakpaul · 2024-03-24T15:14:21Z

@aihao2000 a gentle ping here

aihao2000 · 2024-03-29T08:24:30Z

@sayakpaul IIt is now only about the addition of the data_dir parameter. Is this okay?

HuggingFaceDocBuilderDev · 2024-03-29T10:51:09Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lhoestq

Cool :) ust one comment:

lhoestq · 2024-03-29T11:16:43Z

examples/text_to_image/train_text_to_image_sdxl.py

-    if args.dataset_name is None and args.train_data_dir is None:
-        raise ValueError("Need either a dataset name or a training folder.")


maybe keep this one ?

@lhoestq oh my bad

github-actions · 2024-04-22T15:03:50Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

yiyixuxu · 2024-04-22T18:56:23Z

@aihao2000 can we address the last comment and get this merged?

aihao2000 · 2024-04-23T03:31:17Z

@yiyixuxu Should I submit a new commit to restore this code?

github-actions · 2024-09-14T15:18:29Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

yiyixuxu · 2024-09-17T21:58:26Z

@aihao2000 yes

github-actions · 2024-10-12T15:09:09Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

yiyixuxu · 2024-12-03T04:17:32Z

can you run make style?

yiyixuxu · 2024-12-03T21:26:56Z

thanks @hlky !

* add data_dir parameter to load_dataset --------- Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: YiYi Xu <[email protected]> Co-authored-by: hlky <[email protected]>

github-actions bot added the stale Issues that haven't received updates label Mar 24, 2024

github-actions bot removed the stale Issues that haven't received updates label Mar 25, 2024

add data_dir parameter to load_dataset

77697d4

aihao2000 force-pushed the update-train-scripts branch from 3432404 to 77697d4 Compare March 29, 2024 08:18

sayakpaul requested a review from lhoestq March 29, 2024 10:43

Merge branch 'main' into update-train-scripts

032b14f

lhoestq approved these changes Mar 29, 2024

View reviewed changes

github-actions bot added the stale Issues that haven't received updates label Apr 22, 2024

yiyixuxu removed the stale Issues that haven't received updates label Apr 22, 2024

restore sanity checks

fcf5364

github-actions bot added the stale Issues that haven't received updates label Sep 14, 2024

Merge branch 'main' into update-train-scripts

034b914

github-actions bot removed the stale Issues that haven't received updates label Sep 18, 2024

github-actions bot added the stale Issues that haven't received updates label Oct 12, 2024

yiyixuxu added close-to-merge and removed stale Issues that haven't received updates labels Dec 3, 2024

make

a98b946

yiyixuxu merged commit b58f67f into huggingface:main Dec 3, 2024
8 checks passed

yiyixuxu removed the close-to-merge label Dec 3, 2024

sayakpaul added a commit that referenced this pull request Dec 23, 2024

update (#7067)

70557b9

* add data_dir parameter to load_dataset --------- Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: YiYi Xu <[email protected]> Co-authored-by: hlky <[email protected]>

		if args.dataset_name is None and args.train_data_dir is None:
		raise ValueError("Need either a dataset name or a training folder.")

update #7067

update #7067

Uh oh!

Conversation

aihao2000 commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

sayakpaul commented Feb 23, 2024

Uh oh!

aihao2000 commented Feb 23, 2024

Uh oh!

sayakpaul commented Feb 23, 2024

Uh oh!

aihao2000 commented Feb 26, 2024

Uh oh!

sayakpaul commented Feb 26, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

sayakpaul commented Mar 24, 2024

Uh oh!

aihao2000 commented Mar 29, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Mar 29, 2024

Uh oh!

lhoestq left a comment

Choose a reason for hiding this comment

Uh oh!

lhoestq Mar 29, 2024

Choose a reason for hiding this comment

Uh oh!

aihao2000 Mar 29, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 22, 2024

Uh oh!

yiyixuxu commented Apr 22, 2024

Uh oh!

aihao2000 commented Apr 23, 2024

Uh oh!

github-actions bot commented Sep 14, 2024

Uh oh!

yiyixuxu commented Sep 17, 2024

Uh oh!

github-actions bot commented Oct 12, 2024

Uh oh!

yiyixuxu commented Dec 3, 2024

Uh oh!

Uh oh!

yiyixuxu commented Dec 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aihao2000 commented Feb 23, 2024 •

edited

Loading