Fix race-condition issue when downloading from multiple threads (#2534)

Wauplin · hanouticelina · commit 97ea5787213e · 2024-09-12T10:54:23.000+02:00
* Fix race-condition issue when downloading from multiple threads

* correct fix
diff --git a/src/huggingface_hub/file_download.py b/src/huggingface_hub/file_download.py
@@ -1396,7 +1396,8 @@ def _hf_hub_download_to_cache_dir(
             filename=filename,
             force_download=force_download,
         )
-        _create_symlink(blob_path, pointer_path, new_blob=True)
+        if not os.path.exists(pointer_path):
+            _create_symlink(blob_path, pointer_path, new_blob=True)
 
     return pointer_path
 

Original file line number	Diff line number	Diff line change
`@@ -1396,7 +1396,8 @@ def _hf_hub_download_to_cache_dir(`
`1396`	`1396`	`filename=filename,`
`1397`	`1397`	`force_download=force_download,`
`1398`	`1398`	`)`
`1399`		`- _create_symlink(blob_path, pointer_path, new_blob=True)`
	`1399`	`+ if not os.path.exists(pointer_path):`
	`1400`	`+ _create_symlink(blob_path, pointer_path, new_blob=True)`
`1400`	`1401`
`1401`	`1402`	`return pointer_path`
`1402`	`1403`