Skip to content

Commit 82c0cc8

Browse files
author
The TensorFlow Datasets Authors
committed
Fix croissant CLI to support the preparation of multiple CroissantBuilder configs.
PiperOrigin-RevId: 646790040
1 parent 5166562 commit 82c0cc8

File tree

1 file changed

+19
-8
lines changed

1 file changed

+19
-8
lines changed

tensorflow_datasets/scripts/cli/croissant.py

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -139,11 +139,22 @@ def prepare_croissant_builder(args: CmdArgs) -> None:
139139
mapping=args.mapping_json,
140140
overwrite_version=args.overwrite_version,
141141
)
142-
cli_utils.download_and_prepare(
143-
builder=builder,
144-
download_config=None,
145-
download_dir=args.download_dir,
146-
publish_dir=args.publish_dir,
147-
skip_if_published=args.skip_if_published,
148-
overwrite=args.overwrite,
149-
)
142+
143+
# Generate each config sequentially.
144+
for config in builder.BUILDER_CONFIGS:
145+
builder_for_config = croissant_builder.CroissantBuilder(
146+
jsonld=args.jsonld,
147+
record_set_ids=[config.name],
148+
file_format=args.file_format,
149+
data_dir=args.data_dir,
150+
mapping=args.mapping_json,
151+
overwrite_version=args.overwrite_version,
152+
)
153+
cli_utils.download_and_prepare(
154+
builder=builder_for_config,
155+
download_config=None,
156+
download_dir=args.download_dir,
157+
publish_dir=args.publish_dir,
158+
skip_if_published=args.skip_if_published,
159+
overwrite=args.overwrite,
160+
)

0 commit comments

Comments
 (0)