feat: add shortcut module `optree.pytree` and `optree.treespec` by lqhuang · Pull Request #189 · metaopt/optree

lqhuang · 2025-02-13T11:59:28Z

A much simpler way to add new modules pytree and treespec.

Check PR #186 for more history context and motivation.

The major difference between them:

PR refactor(module)!: A proposal to reorg modules and functions with new tree module and treespec module #186 has concrete and refined docs and migrate to new API design like what jax does.
This PR leverages name re-export with lightweight tests.

Merge of current PR will close #186

…from `ops` module

XuehaiPan

Generally looks good with some nit suggestions. Thanks!

optree/treespec.py

README.md

optree/pytree.py

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

optree/pytree.py

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

lqhuang · 2025-02-13T12:49:51Z

The failing lint says there is a spelling error.

optree/treespec.py:11:0: C0402: Wrong spelling of a word 'versionadded' in a docstring:
.. versionadded:: 0.14.1
   ^^^^^^^^^^^^
Did you mean: ''version added' or 'version-added' or 'versioned' or 'diversionary''? (wrong-spelling-in-docstring)

According to https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionadded, it's the right directive.

XuehaiPan · 2025-02-13T12:51:21Z

The failing lint says there is a spelling error.

You can add versionadded to docs/source/spelling_wordlist.txt.

lqhuang · 2025-02-13T13:42:02Z

Failed again

pylint...................................................................Failed
- hook id: pylint
- exit code: 4

************* Module optree.pytree
optree/pytree.py:36:0: W0622: Redefining built-in 'all' (redefined-builtin)
optree/pytree.py:37:0: W0622: Redefining built-in 'any' (redefined-builtin)
optree/pytree.py:48:0: W0622: Redefining built-in 'iter' (redefined-builtin)
optree/pytree.py:50:0: W0622: Redefining built-in 'map' (redefined-builtin)
optree/pytree.py:56:0: W0622: Redefining built-in 'max' (redefined-builtin)
optree/pytree.py:57:0: W0622: Redefining built-in 'min' (redefined-builtin)
optree/pytree.py:62:0: W0622: Redefining built-in 'sum' (redefined-builtin)
************* Module optree.treespec
optree/treespec.py:32:0: W0622: Redefining built-in 'dict' (redefined-builtin)
optree/treespec.py:35:0: W0622: Redefining built-in 'list' (redefined-builtin)
optree/treespec.py:40:0: W0622: Redefining built-in 'tuple' (redefined-builtin)

-----------------------------------
Your code has been rated at 9.94/10

make: *** [Makefile:191: pre-commit] Error 1

Actually, I tried to run make lint local but with error

$ make pylint
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pylint &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pylint[spelling])
Collecting pylint[spelling]
  Using cached pylint-3.3.4-py3-none-any.whl.metadata (12 kB)
Collecting dill>=0.3.6 (from pylint[spelling])
  Using cached dill-0.3.9-py3-none-any.whl.metadata (10 kB)
Collecting platformdirs>=2.2.0 (from pylint[spelling])
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: astroid<=3.4.0-dev0,>=3.3.8 in ./Git/optree/.venv/lib/python3.12/site-packages (from pylint[spelling]) (3.3.8)
Collecting isort!=5.13.0,<7,>=4.2.5 (from pylint[spelling])
  Using cached isort-6.0.0-py3-none-any.whl.metadata (11 kB)
Collecting mccabe<0.8,>=0.6 (from pylint[spelling])
  Using cached mccabe-0.7.0-py2.py3-none-any.whl.metadata (5.0 kB)
Collecting tomlkit>=0.10.1 (from pylint[spelling])
  Using cached tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Requirement already satisfied: pyenchant~=3.2 in ./Git/optree/.venv/lib/python3.12/site-packages (from pylint[spelling]) (3.2.2)
Using cached dill-0.3.9-py3-none-any.whl (119 kB)
Using cached isort-6.0.0-py3-none-any.whl (94 kB)
Using cached mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached tomlkit-0.13.2-py3-none-any.whl (37 kB)
Using cached pylint-3.3.4-py3-none-any.whl (522 kB)
Installing collected packages: tomlkit, platformdirs, mccabe, isort, dill, pylint
Successfully installed dill-0.3.9 isort-6.0.0 mccabe-0.7.0 platformdirs-4.3.6 pylint-3.3.4 tomlkit-0.13.2
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pyenchant &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pyenchant)
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint --version
pylint 3.3.4
astroid 3.3.8
Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint optree
usage: pylint [options]
pylint: error: argument --spelling-dict: invalid choice: 'en_US' (choose from '')
make: *** [pylint] Error 32

I searched the error, but google redirect me to an unsolved issue pylint-dev/pylint#5092

XuehaiPan · 2025-02-13T13:45:18Z

I tried to run make lint local but with error

Run

brew install enchant
pip3 install -U --pre pyenchant

lqhuang · 2025-02-13T13:47:03Z

I tried to run make lint local but with error

Run
brew install enchant
pip3 install -U --pre pyenchant

Doesn't work at all indeed without --pre.

So the --pre is the key?

…pec`

lqhuang · 2025-02-13T13:52:50Z

Oh man, it works. @XuehaiPan

Please approve again, sorry for that.

$ make pylint
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pylint &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pylint[spelling])
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pyenchant &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pyenchant)
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint --version
pylint 3.3.4
astroid 3.3.8
Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint optree

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.95/10, +0.05)

XuehaiPan · 2025-02-13T13:56:56Z

So the --pre is the key?

The patch is only included in the rc release. You will need to pass --pre to install the pre-release package.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (1b47694) to head (cb3679b).
Report is 50 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##              main      #189    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           36        42     +6     
  Lines         3789      3945   +156     
==========================================
+ Hits          3789      3945   +156

Flag	Coverage Δ
unittests	`100.00% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

XuehaiPan · 2025-02-13T21:32:03Z

@lqhuang Thanks for the contribution!

lqhuang · 2025-02-14T02:25:12Z

Seriously? You wake up at 6 a.m.?

Never mind, optree is obviously underrated, your work is like a gem 👋

feat: Add two new modules pytree and treespec re-exporting utils …

73afbb3

…from `ops` module

lqhuang requested a review from XuehaiPan as a code owner February 13, 2025 11:59

bugfix: Add missing function name to __all__

22906dc

lqhuang force-pushed the new-modules-simpler-impl branch from 33b44be to 22906dc Compare February 13, 2025 12:01

lqhuang added 2 commits February 13, 2025 20:09

test: Add testcase for re-export shortcut

34c4a44

lint: Fix test function name

02654aa

XuehaiPan approved these changes Feb 13, 2025

View reviewed changes

optree/treespec.py Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Show resolved Hide resolved

optree/pytree.py Outdated Show resolved Hide resolved

lqhuang mentioned this pull request Feb 13, 2025

refactor(module)!: A proposal to reorg modules and functions with new tree module and treespec module #186

Closed

29 tasks

lqhuang and others added 3 commits February 13, 2025 20:31

docs: Update optree/treespec.py

ed171b9

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

Update README.md

a6b3cdd

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

Update optree/pytree.py

1897493

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

XuehaiPan reviewed Feb 13, 2025

View reviewed changes

optree/pytree.py Outdated Show resolved Hide resolved

lqhuang and others added 2 commits February 13, 2025 20:37

Update README.md for pt.leaves demo cases

3a72c2e

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

Update optree/pytree.py: simplify doctest

624ca40

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

docs: Add versionadded into spelling wordlist

66c636b

XuehaiPan added enhancement New feature or request python Something related to the Python source code labels Feb 13, 2025

lint: Disable pylint rule redefined-builtin

872c143

lint: disable redefined-builtin from pylint for pytree and `trees…

4f8e7ba

…pec`

XuehaiPan added 2 commits February 13, 2025 23:22

style: nit updates

37aaeba

lint: disable pylint false positive

7e793e2

XuehaiPan changed the title ~~refactor!(module): A proposal to reorg modules and functions with new tree module and treespec module~~ feat: add shortcut module optree.pytree and optree.treespec Feb 13, 2025

XuehaiPan added 2 commits February 14, 2025 01:00

feat: also export registry APIs

dc01df6

docs: improve sphinx cross-reference

a65aecd

XuehaiPan force-pushed the new-modules-simpler-impl branch from 9267dff to a65aecd Compare February 13, 2025 18:47

XuehaiPan force-pushed the main branch from a9f02e4 to 1150954 Compare February 13, 2025 21:20

XuehaiPan added 2 commits February 14, 2025 05:28

docs: fix docs

ba534eb

Merge branch 'main' into pr/lqhuang/189

cb3679b

XuehaiPan merged commit f442bec into metaopt:main Feb 13, 2025
28 of 29 checks passed

lqhuang deleted the new-modules-simpler-impl branch February 14, 2025 02:16

Conversation

lqhuang commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuehaiPan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lqhuang commented Feb 13, 2025

Uh oh!

XuehaiPan commented Feb 13, 2025

Uh oh!

lqhuang commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuehaiPan commented Feb 13, 2025

Uh oh!

lqhuang commented Feb 13, 2025

Uh oh!

lqhuang commented Feb 13, 2025

Uh oh!

XuehaiPan commented Feb 13, 2025

Uh oh!

codecov bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

XuehaiPan commented Feb 13, 2025

Uh oh!

Uh oh!

lqhuang commented Feb 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lqhuang commented Feb 13, 2025 •

edited

Loading

lqhuang commented Feb 13, 2025 •

edited

Loading

codecov bot commented Feb 13, 2025 •

edited

Loading