Add type annotations by chadrik · Pull Request #1761 · AcademySoftwareFoundation/rez

chadrik · 2024-05-19T17:38:29Z

This is a first pass at adding type annotations throughout the code-base. Mypy is not fully passing yet, but it's getting close.

Fixes #1631

linux-foundation-easycla · 2024-05-19T17:38:33Z

The committers listed above are authorized under a signed CLA.

✅ login: chadrik / name: Chad Dombrova (8b590da)

src/rez/package_filter.py

src/rez/package_order.py

src/rez/package_resources.py

chadrik · 2024-05-19T17:49:24Z

src/rez/packages.py

        for variant in self.iter_variants():
            if variant.index == index:
                return variant
+        return None


mypy prefer explicit return None statements

This is a safe change so I left it.

chadrik · 2024-05-19T17:50:06Z

src/rez/resolved_context.py

-                raise ResolvedContextError(
-                    "Cannot perform operation in a failed context")
-        return _check
-


mypy does not like these decorators defined at the class-level.

This is a relatively safe change so I left it

chadrik · 2024-05-19T18:08:22Z

Protocol and TypedDict are not available in the typing module until python 3.8.

We have a few options:

vendor typing_extensions
remove use of these typing classes until Rez drops support for python 3.7
I can create a mock of Protocol and trick mypy into using it which is safe because it has no runtime behavior. Doing the same thing for TypedDict is more complicated, but possible.

codecov · 2024-05-19T18:33:50Z

Codecov Report

❌ Patch coverage is 88.06604% with 253 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.08%. Comparing base (9afb325) to head (8b590da).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
src/rez/resolved_context.py	80.43%	25 Missing and 2 partials ⚠️
src/rez/resolver.py	67.30%	16 Missing and 1 partial ⚠️
src/rez/plugin_managers.py	76.19%	14 Missing and 1 partial ⚠️
src/rez/version/_version.py	92.26%	10 Missing and 4 partials ⚠️
src/rez/package_order.py	84.70%	11 Missing and 2 partials ⚠️
src/rez/package_resources.py	79.59%	8 Missing and 2 partials ⚠️
src/rez/packages.py	87.17%	9 Missing and 1 partial ⚠️
src/rez/pip.py	0.00%	7 Missing ⚠️
src/rez/build_process.py	77.77%	5 Missing and 1 partial ⚠️
src/rez/solver.py	96.68%	5 Missing and 1 partial ⚠️
... and 48 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1761      +/-   ##
==========================================
+ Coverage   59.98%   60.08%   +0.10%     
==========================================
  Files         163      164       +1     
  Lines       20118    20527     +409     
  Branches     3519     3558      +39     
==========================================
+ Hits        12067    12333     +266     
- Misses       7230     7336     +106     
- Partials      821      858      +37

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

chadrik · 2024-05-20T05:34:07Z

I got bored and added lots more, particularly focused on the solver module. Once the solver module is complete, we can experiment with compiling it to a c-extension using mypyc, which could provide a big speed boost!

chadrik · 2024-05-21T03:28:27Z

I now have rez.solver compiling as a C-extension with all tests passing. I'm very interested to see how the performance compares. Does anyone want to volunteer to help put together a performance comparison? Are there any known complex collection of packages to test against?

chadrik · 2024-05-21T03:37:46Z

src/rez/package_order.py

-        self.dirty = True
-        return super().append(*args, **kwargs)
+    if not TYPE_CHECKING:
+        def append(self, *args, **kwargs):


Since this class inherits from list it's easier to rely on the type hints coming from that base class than to redefine them here, so we hide them by placing them behind not TYPE_CHECKING. In reality, the runtime value of TYPE_CHECKING is always False.

src/rez/plugin_managers.py

chadrik · 2024-05-21T03:41:43Z

src/rez/resolved_context.py


        Args:
-            package_requests (list[typing.Union[str, PackageRequest]]): request
+            package_requests (list[typing.Union[str, Requirement]]): request


I noticed that everywhere that we've documented types as PackageRequest, they appear to actually be Requirement. I'm not sure if there any real-world exceptions to this.

Here it's really supposed to be a PackageRequest if I'm not mistaken. But there is technically no differences between the two once the instantiated since PackageRequest inherits from Requirement and only overloads __init__ to check the inputs.

The kind of haphazard use and documentation of PackageRequest and Requirement results in some very difficult situations to accurately add type annotations. If you want to see for yourself, check out the code, change this to PackageRequest and observe the new errors produced by mypy.

Ok, I took a look and I think you are right. I was afraid that something down the line would use the .ephemeral attribute of PackageRequest, but it doesn't look like it. And Honestly, the ResolvedContext is cursed. PackageRequest exists to validate the name in the request, but it's kind of used both as a validation layer and just as a simple Requirement. Its usage is pretty inconsistent, and it's also used in places where I wouldn't expect it to be used.

chadrik · 2024-05-21T03:42:10Z

src/rez/resolved_context.py


        _pr("resolved packages:", heading)
-        rows = []
+        rows3: list[tuple[str, str, str]] = []


you can't redefine types with mypy, so you need to use new variable names.

src/rez/system.py

chadrik · 2024-05-21T03:43:31Z

src/rez/utils/data_utils.py

+if TYPE_CHECKING:
+    cached_property = property
+else:
+    class cached_property(object):


it's much easier to pretend that cached_property is property than to type hint all the subtleties of a descriptor.

But we loose the uncache method.

typing.TYPE_CHECKING always resolve to False at runtime and True only during static analysis. So the code within the if TYPE_CHECKING block will never run. It's a way to simplify certain type analysis situations that arise.

typing.TYPE_CHECKING always resolve to False

I know, but we are loosing stuff during typing. That's my whole point (the same apply to all my comments that are similar to this one).

in what way would the static analysis be degraded by using property instead of cached_property? To my knowledge, they are functionally equivalent from a static analysis POV: the types returned are the same.

In another thread you mentioned losing the uncache method. Unfortunately, it's not possible to type annotate this in a way both the return type and preserve the uncached method. It's something I've looked into pretty extensively.

IMO, the best path forward is to migrate to functools.cached_property. Note that to clear the cache with functools.cached_property you simply delete the attribute.

src/rez/solver.py

chadrik · 2024-05-21T03:49:58Z

src/rez/solver.py

        """Reset the solver, removing any current solve."""
        if not self.request_list.conflict:
-            phase = _ResolvePhase(self.request_list.requirements, solver=self)
+            phase = _ResolvePhase(solver=self)


This appears to be a bug: _ResolvePhase only takes one argument. mypy to the rescue.

chadrik · 2024-05-21T04:48:54Z

I found rez-benchmark. Interestingly, rez is slower with the compiled rez.solver. It could be because there are many modules and classes used by rez.solver which have not been compiled.

I probably won't have time to dig into this much more, but once this PR is merged I'll make a new PR with the changes necessary for people to test the compiled version of rez.

chadrik · 2024-05-21T18:27:53Z

Note: this PR likely invalidates #1745

chadrik · 2024-06-05T18:04:33Z

@instinct-vfx Can you or someone from the Rez group have a look at this, please?

JeanChristopheMorinPerso · 2024-06-22T16:46:20Z

src/rez/build_process.py

        return self.build_system.working_dir

-    def build(self, install_path=None, clean=False, install=False, variants=None):
+    def build(self, install_path: str | None = None, clean: bool = False,


Using str | None means we need to drop support for python 3.7. I'm not sure we are ready for this yet.

No, use of str | None is safe in python 3.7 as long as you use from __future__ import annotations. This backports behavior from python 3.9 that ensures that type annotations are recorded as strings within __annotations__ attributes, which means they are not evaluated at runtime unless inspect.get_annoations is called. The effect of from __future__ import annotations is that you can put basically anything you want into an annotation, it doesn't need to be valid at runtime.

The only thing breaking python 3.7 compatibility here is the use of TypedDict and Protocol, as mentioned in another comment. I presented 3 options for preserving the use of these classes in the other comment.

I noticed that the only python 3.7 tests that are currently run are for MacOS, which I took as an indicator that python 3.7 would be dropped soon. Is there a schedule for deprecation?

By the way, I fixed the python 3.7 compatibility issue with TypedDict and Protocol, so that should not be a blocker anymore.

It's been a year since this thread was started: has rez dropped python 3.7 support yet?

This PR will work for python 3.7, but I can remove some workarounds if we've dropped support.

Not yet, but we could do this in the next release.

I added a check to mypy to ensure it raises errors if we use any unsupported typing features.

There are a few cases of unsupported features:

dict[X], list[X]: use from __future__ import annotations

X | None: use from __future__ import annotations

typing.Self or other type not in typing: use if TYPE_CHECKING`

Should I document this somewhere?

For now, I recommend that we keep using from __future__ import annotations so that we can use the modern form.

Here's a doc I compiled of typing changes by python version:

https://docs.google.com/document/d/1uOJUvgjPDwP-PRYkmlypOAPtJ9s4nWYm6WRfrAAsn0g/edit?usp=sharing

Thank you. I guess we could add this to our contribution docs.

JeanChristopheMorinPerso · 2024-06-22T16:47:01Z

@chadrik You need to sign the CLA before we can even start to look at the PR.

chadrik · 2024-06-22T19:47:27Z

@JeanChristopheMorinPerso

@chadrik You need to sign the CLA before we can even start to look at the PR.

I work for Scanline, which is owned by Netflix, and I'm meeting with our CLA manager on Monday. I made this contribution on my own time: can choose individual vs corporate CLA on a per PR basis?

JeanChristopheMorinPerso · 2024-06-22T20:28:37Z

I made this contribution on my own time: can choose individual vs corporate CLA on a per PR basis?

I don't think you "can" but your account can be associated to an an ICLA and a CCLA. But I'm not a lawyer so I can't help more than that. If you and or your employer/CLA manager have questions, you can contact the LF support by following the link in the EasyCLA comment: #1761 (comment).

chadrik · 2024-07-01T21:09:04Z

CLA is signed!

src/rez/build_system.py

chadrik · 2024-07-01T21:21:56Z

src/rez/package_order.py



-class PackageOrderList(list):
+class PackageOrderList(List[PackageOrder]):


Note that we use typing.List here instead of list because list does not become indexable at runtime until python 3.9. It's still safe to use list[X] inside annotations as long as we use from __future__ import annotations.

chadrik · 2024-07-18T14:49:11Z

Any thoughts on this PR?

JeanChristopheMorinPerso · 2024-07-21T19:49:46Z

Hey @chadrik, I made a first good read last week and I'll try to do another one soon. If you have the time, I would really love to see a GitHub Actions workflow that would run mypy on all pull requests.

chadrik · 2024-07-21T20:20:16Z

I would really love to see a GitHub Actions workflow that would run mypy on all pull requests.

Me too!

The challenge is that there are still a lot of errors. These are not the result of incorrect annotations, but rather due to code patterns which are difficult to annotate correctly without more invasion refactors. For example, there are quite a few objects which dynamically generate attributes, and that's a static typing anti-pattern.

If you'd like an Action that runs mypy but allows failure for now, that's pretty easy, but if you want failures to block PRs, that'll take a lot more work. I'd prefer not to make that a blocker to merging this, though, because I've had to rebase and fix merge conflicts a few times already.

I do have a plan for how we can get to zero failures in the mid-term: I wrote a tool which allows you to specify patterns for errors to ignore, but I need to update it.

JeanChristopheMorinPerso · 2024-07-21T20:27:18Z

I think we can introduce a workflow that will fail for newly introduced errors and warnings. I'm sure someone already thought of that somewhere and we can probably re-use what they did?

Basically, I'd like to verify that your changes work as expected and that we don't regress in the future and that new code is typed hint. Mypy can also be configured on a per module basis right?

JeanChristopheMorinPerso

Second batch of comments. Note that I'm not yet done with the review. Please don't push changes until I'm done.

src/rez/build_system.py

src/rez/package_order.py

src/rez/utils/patching.py

src/rez/utils/platform_.py

src/rez/utils/resources.py

src/rez/version/_requirement.py

chadrik · 2025-03-01T01:47:14Z

It would be great if all this work didn't go to waste. If I rebase this, can we get it merged?

chadrik · 2025-04-24T15:48:29Z

I'm motivated again to get this merged. I've started addressing some of the outstanding notes here, but I would love it if we could focus on blocker issues.

chadrik · 2025-06-20T04:56:11Z

In the next week or two I'll do a pass to rebase this and remove anything that looks even remotely like a runtime change.

JeanChristopheMorinPerso · 2025-06-21T18:01:42Z

In the next week or two I'll do a pass to rebase this and remove anything that looks even remotely like a runtime change.

Thanks @chadrik. This will definitely help us get this merged with more confidence (not that we don't trust you, but this PR is quite big).

chadrik · 2025-07-09T03:28:34Z

@JeanChristopheMorinPerso I've removed everything that could be considered remotely risky.

chadrik · 2025-07-19T00:43:11Z

Any chance that this can get merged? I’ve removed everything but the addition of annotations.

JeanChristopheMorinPerso · 2025-10-17T22:37:46Z

Hey @chadrik when you get a chance, could you rebase your PR please? Now that we dropped support for Python 3.7, your PR is next in line to be released.

JeanChristopheMorinPerso · 2025-10-17T22:38:24Z

You should also look into the CLA. EasyCLA doesn't seem to be happy.

chadrik · 2025-12-16T01:59:39Z

@JeanChristopheMorinPerso this is rebased and all tests are passing. I just have to get the CLA fixed up.

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

chadrik · 2026-01-06T16:58:01Z

Happy New Year! CLA and commits are both signed.

JeanChristopheMorinPerso · 2026-01-07T00:55:49Z

Yay! Thank you @chadrik! This will go in the next release. I'll do one last review (maybe later this week) before merging, just in case.

Thanks again!

JeanChristopheMorinPerso

Thanks @chadrik I left a few comments. Most are very easy to address. I also left a few questions that I would like to get an answer on before merging, if possible.

I promise, this is the last batch of comments and questions :)

JeanChristopheMorinPerso · 2026-01-10T17:17:57Z

src/rez/version/_version.py

        self.value = value

-    def __eq__(self, other):
+    def __eq__(self, other: object) -> bool:


In all the other classes in this module, you use the actual class for other in the operators. Is there a reason why you use object here instead of the class?

JeanChristopheMorinPerso · 2026-01-10T17:29:33Z

src/rez/version/_version.py

 _Bound.any = _Bound()


+def action(fn: CallableT) -> CallableT:


Suggested change

def action(fn: CallableT) -> CallableT:

def _action(fn: CallableT) -> CallableT:

I feel like this should be a private function.

JeanChristopheMorinPerso · 2026-01-10T17:29:49Z

src/rez/version/_version.py

-            return result
-        return fn_
-
    @action


Suggested change

@action

@_action

JeanChristopheMorinPerso · 2026-01-10T17:29:58Z

src/rez/version/_version.py


        self.bounds.append(_Bound(lower_bound, upper_bound))

    @action


Suggested change

@action

@_action

JeanChristopheMorinPerso · 2026-01-10T17:30:04Z

src/rez/version/_version.py


        self.bounds.append(_Bound(lower_bound, upper_bound))

    @action


Suggested change

@action

@_action

JeanChristopheMorinPerso · 2026-01-10T19:46:20Z

src/rez/resolved_context.py


        Args:
-            package_requests (list[typing.Union[str, PackageRequest]]): request
+            package_requests (list[typing.Union[str, Requirement]]): request


Ok, I took a look and I think you are right. I was afraid that something down the line would use the .ephemeral attribute of PackageRequest, but it doesn't look like it. And Honestly, the ResolvedContext is cursed. PackageRequest exists to validate the name in the request, but it's kind of used both as a validation layer and just as a simple Requirement. Its usage is pretty inconsistent, and it's also used in places where I wouldn't expect it to be used.

JeanChristopheMorinPerso · 2026-01-10T19:50:54Z

src/rez/resolved_context.py

+            request: list[Requirement | PackageRequest] = []
            for variant in self.resolved_packages:
                req = PackageRequest(variant.qualified_package_name)
-                request.append(req)


This seems weird to me. I believe this was removed by mistake?

JeanChristopheMorinPerso · 2026-01-10T20:09:00Z

src/rezplugins/build_process/local.py

@@ -5,6 +5,8 @@
 """
 Builds packages on local host
 """
+from __future__ import annotations
+
 from rez.config import config
 from rez.package_repository import package_repository_manager
 from rez.build_process import BuildProcessHelper, BuildType


Suggested change

from rez.build_process import BuildProcessHelper, BuildType

from rez.build_process import BuildProcessHelper, BuildType

from rez.build_system import BuildResult

JeanChristopheMorinPerso · 2026-01-10T20:10:46Z

src/rezplugins/build_process/local.py

+    from rez.build_system import BuildResult
+
+    # FIXME: move this out of TYPE_CHECKING block when python 3.7 support is dropped
+    class LocalBuildResult(BuildResult, total=False):
+        package_install_path: str
+        variant_install_path: str
+
+else:
+    LocalBuildResult = dict
+


Suggested change

from rez.build_system import BuildResult

# FIXME: move this out of TYPE_CHECKING block when python 3.7 support is dropped

class LocalBuildResult(BuildResult, total=False):

package_install_path: str

variant_install_path: str

else:

LocalBuildResult = dict

class LocalBuildResult(BuildResult, total=False):

package_install_path: str

variant_install_path: str

JeanChristopheMorinPerso · 2026-01-10T20:15:54Z

src/rezplugins/shell/_utils/windows.py

-        else:
-            expanded_value = winreg.ExpandEnvironmentStrings(reg_value)
-            paths.extend(expanded_value.split(os.pathsep))
+    if sys.platform == "win32":


Why is this if needed?

chadrik requested a review from a team as a code owner May 19, 2024 17:38

chadrik commented May 19, 2024

View reviewed changes

chadrik commented May 21, 2024

View reviewed changes

JeanChristopheMorinPerso added the Blocked by CLA Waiting on CLA to be signed label Jun 22, 2024

JeanChristopheMorinPerso reviewed Jun 22, 2024

View reviewed changes

chadrik force-pushed the typing branch 4 times, most recently from e73c6c1 to 961420b Compare June 22, 2024 23:10

chadrik force-pushed the typing branch from 44f4ac3 to efcae39 Compare July 1, 2024 21:07

chadrik commented Jul 1, 2024

View reviewed changes

src/rez/build_system.py Show resolved Hide resolved

chadrik commented Jul 1, 2024

View reviewed changes

JeanChristopheMorinPerso removed the Blocked by CLA Waiting on CLA to be signed label Jul 1, 2024

chadrik force-pushed the typing branch from 2e7fc29 to 3e61ca7 Compare August 5, 2024 23:24

JeanChristopheMorinPerso reviewed Oct 15, 2024

View reviewed changes

chadrik force-pushed the typing branch from dd86192 to 3e15a99 Compare April 24, 2025 15:26

chadrik force-pushed the typing branch 4 times, most recently from 84c1e43 to 95e64a9 Compare April 25, 2025 01:12

chadrik mentioned this pull request Jun 16, 2025

Add python type annotations shotgunsoftware/python-api#392

Closed

chadrik force-pushed the typing branch 3 times, most recently from c0a653d to f83ccc6 Compare June 27, 2025 16:37

chadrik force-pushed the typing branch 2 times, most recently from 4129ab0 to 1ca4a2f Compare July 9, 2025 03:21

chadrik force-pushed the typing branch from 1ca4a2f to faca3a9 Compare July 9, 2025 04:18

JeanChristopheMorinPerso added this to the Next milestone Oct 17, 2025

chadrik mentioned this pull request Nov 7, 2025

SG-39414 Add type annotations shotgunsoftware/python-api#422

Merged

chadrik force-pushed the typing branch from faca3a9 to 9334eb4 Compare December 15, 2025 23:56

Add type annotations

8b590da

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

chadrik force-pushed the typing branch from 74c03c6 to 8b590da Compare January 6, 2026 16:57

JeanChristopheMorinPerso reviewed Jan 10, 2026

View reviewed changes



		class PackageOrderList(list):
		class PackageOrderList(List[PackageOrder]):

		_Bound.any = _Bound()


		def action(fn: CallableT) -> CallableT:

	def action(fn: CallableT) -> CallableT:
	def _action(fn: CallableT) -> CallableT:

	from rez.build_process import BuildProcessHelper, BuildType
	from rez.build_process import BuildProcessHelper, BuildType
	from rez.build_system import BuildResult

Conversation

chadrik commented May 19, 2024 • edited by JeanChristopheMorinPerso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla bot commented May 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chadrik commented May 19, 2024

Uh oh!

codecov bot commented May 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chadrik commented May 20, 2024

Uh oh!

chadrik commented May 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chadrik Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chadrik commented May 21, 2024

Uh oh!

chadrik commented May 21, 2024

Uh oh!

chadrik commented Jun 5, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chadrik Jun 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chadrik commented May 19, 2024 •

edited by JeanChristopheMorinPerso

Loading

linux-foundation-easycla bot commented May 19, 2024 •

edited

Loading

codecov bot commented May 19, 2024 •

edited

Loading

chadrik Oct 15, 2024 •

edited

Loading

chadrik Jun 22, 2024 •

edited

Loading