Skip to content

Fix race condition in FileSystemPackageRepository directory creation, modernize usage of os.makedirs#1913

Merged
JeanChristopheMorinPerso merged 4 commits intoAcademySoftwareFoundation:mainfrom
nrusch:os_makedirs_exist_ok
Jun 21, 2025
Merged

Fix race condition in FileSystemPackageRepository directory creation, modernize usage of os.makedirs#1913
JeanChristopheMorinPerso merged 4 commits intoAcademySoftwareFoundation:mainfrom
nrusch:os_makedirs_exist_ok

Conversation

@nrusch
Copy link
Contributor

@nrusch nrusch commented Jan 28, 2025

The original impetus for this PR was to fix a race condition in FileSystemPackageRepository.pre_variant_install if multiple processes attempted to create a family directory in a narrow timing window.

From there, I decided to also go through and "modernize" any usage of os.makedirs to leverage the new exist_ok kwarg added in Python 3.2.

The PR is split up into 3 commits, in case you'd prefer to only integrate a subset:

  1. Fix the race condition.
  2. Replace rez.utils.filesystem.safe_makedirs with direct os.makedirs calls.
  3. Update any remaining uses of os.makedirs to use exist_ok=True where it seems appropriate.

Signed-off-by: Nathan Rusch <nathan.rusch@scanlinevfx.com>
Signed-off-by: Nathan Rusch <nathan.rusch@scanlinevfx.com>
Signed-off-by: Nathan Rusch <nathan.rusch@scanlinevfx.com>
@nrusch nrusch requested a review from a team as a code owner January 28, 2025 22:37
@nrusch
Copy link
Contributor Author

nrusch commented Jan 28, 2025

Thanks to @mgcollie for pointing out the original race condition.

@codecov
Copy link

codecov bot commented Jan 28, 2025

Codecov Report

❌ Patch coverage is 82.14286% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.07%. Comparing base (d152c9a) to head (7e21324).
⚠️ Report is 12 commits behind head on main.

Files with missing lines Patch % Lines
src/rez/package_maker.py 50.00% 1 Missing ⚠️
src/rez/pip.py 0.00% 1 Missing ⚠️
src/rez/utils/filesystem.py 50.00% 1 Missing ⚠️
src/rez/utils/py_dist.py 0.00% 1 Missing ⚠️
src/rezplugins/package_repository/filesystem.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1913      +/-   ##
==========================================
+ Coverage   60.05%   60.07%   +0.01%     
==========================================
  Files         163      163              
  Lines       20120    20096      -24     
  Branches     3507     3493      -14     
==========================================
- Hits        12084    12072      -12     
+ Misses       7216     7208       -8     
+ Partials      820      816       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JeanChristopheMorinPerso JeanChristopheMorinPerso added this to the Next milestone Feb 1, 2025
Copy link
Member

@JeanChristopheMorinPerso JeanChristopheMorinPerso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nrusch. I agree with you that the semantics of safe_makedirs seem to be almost the same as os.makedirs(path, exists_ok=True), at least according to https://github.com/python/cpython/blob/b14986c91464b06e9016678637e41d0e0192bbe8/Lib/os.py#L211-L241, which hasn't changed since at least 3.7. There are small differences, but nothing major, only improvements.

@JeanChristopheMorinPerso JeanChristopheMorinPerso merged commit 0c4b28f into AcademySoftwareFoundation:main Jun 21, 2025
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants