dvc and not finding files in cache

[rrazavipour]

I am running git and dvc on an Ubuntu server.
So I am running dvc on a very large data set.
8 out of 10, when I run dvc pull --verbose, I see the following errors and not sure how to get around them since it fails the whole dvc pull.

[stdout] 2024-11-07 19:55:20,391 DEBUG: failed to load ('f86e8f8d-acd3-482c-bd35-85069275e973',) from storage local (/opt/ml/input/data/train/test_mm_code/.dvc/cache) - [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/14/16241f6e56784e46bdf946657ffffd.dir'
[stdout] Traceback (most recent call last):
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_data/index/index.py", line 611, in _load_from_storage
[stdout] _load_from_object_storage(trie, entry, storage)
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_data/index/index.py", line 547, in _load_from_object_storage
[stdout] obj = Tree.load(storage.odb, root_entry.hash_info, hash_name=storage.odb.hash_name)
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_data/hashfile/tree.py", line 193, in load
[stdout] with obj.fs.open(obj.path, "r") as fobj:
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 324, in open
[stdout] return self.fs.open(path, mode=mode, **kwargs)
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 131, in open
[stdout] return open(path, mode=mode, encoding=encoding) # noqa: SIM115
[stdout] FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/14/16241f6e56784e46bdf946657ffffd.dir'
[stdout]
[stdout] 2024-11-07 19:55:20,913 DEBUG: failed to load ('69641d3d-57ed-4a5f-b1aa-409f767d74a9',) from storage local (/opt/ml/input/data/train/test_mm_code/.dvc/cache) - [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/ad/569ea9f63c7f034fbd532826d9eda4.dir'
[stdout] Traceback (most recent call last):
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_data/index/index.py", line 611, in _load_from_storage
[stdout] _load_from_object_storage(trie, entry, storage)
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_data/index/index.py", line 547, in _load_from_object_storage
[stdout] obj = Tree.load(storage.odb, root_entry.hash_info, hash_name=storage.odb.hash_name)
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_data/hashfile/tree.py", line 193, in load
[stdout] with obj.fs.open(obj.path, "r") as fobj:
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 324, in open
[stdout] return self.fs.open(path, mode=mode, **kwargs)
[stdout] File "/opt/conda/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 131, in open
[stdout] return open(path, mode=mode, encoding=encoding) # noqa: SIM115
[stdout] FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/ad/569ea9f63c7f034fbd532826d9eda4.dir'
[stdout]
[stdout] 2024-11-07 19:55:21,343 DEBUG: failed to load ('72e06de0-8ddb-4120-ad10-0c788303923a',) from storage local (/opt/ml/input/data/train/test_mm_code/.dvc/cache) - [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/fb/5817896f6cc91cc659d74006553b6b.dir'

Please shed some light on this problem.

Regards
Reza

[shcheklein]

@rrazavipour please include dvc version.

[rrazavipour]

I have tested with the latest and 2.10.2 with the same error

[rrazavipour]

I run dvc run --verbose to get more logging