Skip to content

Commit dbb7fa5

Browse files
authored
Pass in the 'cache_dir' to use local cache (#690)
* Pass in the 'cache_dir' to use local cache We can pass in the 'cache_dir' to the huggingface snapshot_download function if we have 'videos' in dataset_kwargs to avoid force-redownloading. * Fix: apply Black formatting Apply Black formatting
1 parent 7c03938 commit dbb7fa5

File tree

1 file changed

+9
-2
lines changed

1 file changed

+9
-2
lines changed

lmms_eval/api/task.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -942,10 +942,17 @@ def _download_from_youtube(path):
942942
force_unzip = dataset_kwargs.get("force_unzip", False)
943943
revision = dataset_kwargs.get("revision", "main")
944944
create_link = dataset_kwargs.get("create_link", False)
945-
cache_path = snapshot_download(repo_id=self.DATASET_PATH, revision=revision, repo_type="dataset", force_download=force_download, etag_timeout=60)
945+
cache_path = snapshot_download(
946+
repo_id=self.DATASET_PATH,
947+
cache_dir=cache_dir,
948+
revision=revision,
949+
repo_type="dataset",
950+
force_download=force_download,
951+
etag_timeout=60,
952+
)
946953
zip_files = glob(os.path.join(cache_path, "**/*.zip"), recursive=True)
947954
tar_files = glob(os.path.join(cache_path, "**/*.tar*"), recursive=True)
948-
955+
949956
def unzip_video_data(zip_file):
950957
import os
951958
import zipfile

0 commit comments

Comments
 (0)