Merge development branch 'kotopes-develop'

Author: Daniel Zagaynov

README.md

@@ -1,61 +1,210 @@
 # Py3DepHell
 This project presents tools to work with dependencies and provides of python3 projects.
 
-## py3req
-This module detects dependencies of python3 packages. It has verbose **--help** option, but here is simple example how to use it:
 
 ## py3prov
 This module generate provides for python3 packages. As for **py3req** its **--help** is verbose enough
 
+## py3req
+This module detects dependencies of python3 packages. It has verbose **--help** option, but here is simple example how to use it:
+
 ## How to
-Imagine you have simple project like this one:
-```
-src/
-├── pkg1
-│   ├── mod1.py
-│   └── subpkg
-│       └── mod3.py
+[Imagine](https://packaging.python.org/en/latest/tutorials/packaging-projects/) you have simple project like this one:
+```shell
+├── src
+│   └── pkg1
+│       ├── mod1.py
+│       └── subpkg
+│           └── mod3.py
 └── tests
     └── test1.py
 ```
+With the following context:
+
+**src/pkg1/mod1.py**:
+```python3
+import re
+import sys
+import os
+import os.path
 
-Now you want to detect its dependencies:
+import numpy
+
+from .subpkg import mod3
 ```
-% python3 -m py3dephell.py3req --pip_format src
-unittest
-re
+
+**src/pkg1/subpkg/mod3.py**
+```python3
+import ast
+```
+
+**tests/test1.py**
+```python3
+import unittest
+
+import pytest
+```
+
+### Detecting dependencies
+Let's run **py3req** to detect deps for our project:
+
+```shell
+% py3req src tests
+numpy
+pytest
+```
+
+Let's turn on verbose mode and check what happened with dependencies:
+```shell
+% py3req --verbose src tests
+py3prov: bad name for provides from path:config-3.12-x86_64-linux-gnu
+py3req:/tmp/dummy/src/pkg1/mod1.py: "re" lines:[1] is possibly a self-providing dependency, skip it
+py3req:/tmp/dummy/src/pkg1/mod1.py: skipping "sys" lines:[2]
+py3req:/tmp/dummy/src/pkg1/mod1.py: "os" lines:[3] is possibly a self-providing dependency, skip it
+py3req:/tmp/dummy/src/pkg1/mod1.py: "os.path" lines:[4] is possibly a self-providing dependency, skip it
+py3req:/tmp/dummy/src/pkg1/mod1.py: "tmp.dummy.src.pkg1.subpkg" lines:[8] is possibly a self-providing dependency, skip it
+py3req:/tmp/dummy/src/pkg1/subpkg/mod3.py: "ast" lines:[1] is possibly a self-providing dependency, skip it
+py3req:/tmp/dummy/tests/test1.py: "unittest" lines:[1] is possibly a self-providing dependency, skip it
+/tmp/dummy/src/pkg1/mod1.py:numpy
+/tmp/dummy/tests/test1.py:pytest
+```
+
+As you can see, **py3req** recognised dependency from **src/pkg1/mod1.py** to **src/pkg1/subpkg/mod3.py**, but since it is provided by given file list, **py3req** filtered it out.
+
+#### Filtering dependencies
+
+According to the previouse example, **sys** was not classified as a dependency, because **sys** is built-in module, which is provided by interpreter by itself. So such deps are filtered out by **py3req**. To make it visible for **py3req** use option **--include_built-in**:
+
+```shell
+% py3req --include_built-in src tests
+sys
+numpy
+pytest
+```
+
+Now let's include dependencies, that are provided by python3 standard library:
+
+```shell
+% py3req --include_stdlib src tests
 re
+numpy
+os.path
+os
+ast
+pytest
+unittest
 ```
-Feel free to make it more verbose:
+
+But what if we have dependency, that is provided by our environment or another one package, so we want **py3req** to find it and exclude from dependencies? For such problem we have **--add_prov_path** option:
+
+```shell
+% py3req  --add_prov_path src2 src tests
+numpy
 ```
-% python3 -m py3dephell.py3req --pip_format --verbose src
-py3prov: detected potential module:src
-/tmp/.private/kotopesutility/src/tests/test1.py:unittest
-/tmp/.private/kotopesutility/src/pkg1/mod1.py:requests os
-/tmp/.private/kotopesutility/src/pkg1/subpkg/mod3.py:re
+
+Where **src2** has the following structure:
+```shell
+src2
+└── pytest
+    └── __init__.py
 ```
-As you can see, there are some modules from standard library, so let py3req to learn it:
+
+Another way to exclude such dependency is to ignore it manually, using **--ignore_list** option:
+```shell
+% py3req --ignore_list pytest src tests
+numpy
 ```
-% python3 -m py3dephell.py3req --pip_format --add_prov_path /usr/lib64/python3.11 src
-requests
+
+#### Context dependencies
+
+Finally, there can be deps, that are hidden inside conditions or function calls. For example:
+
+**anime_dld.py**
+```python3
+import os
+
+
+def func():
+    import pytest
+
+
+try:
+    import specific_module
+except Exception as ex:
+    print(f"I'm sorry, but {ex}")
+
+
+a = int(input())
+if a == 10:
+    import ast
+else:
+    import re
 ```
-That's it! But what if we want to detect its provides, to understand which dependencies it could satisfy? Let's use py3prov!
+
+In general it is impossible to check if condition **a == 10** is True or False. Moreover it is not clear if **specific_module** is really important for such project or not. So, by default **py3req** catch them all:
+
+```shell
+% py3req anime_dld.py
+pytest
+specific_module
 ```
-% python3 -m py3dephell.py3prov src
-test1
-tests.test1
-src.tests.test1
-mod1
-pkg1.mod1
+
+But it is possible to ignore all deps, that are hidden inside contexts:
+```shell
+% py3req --exclude_hidden_deps anime_dld.py
+%
+```
+
+Other options are little bit specific, but there is clear **--help** option output. Please, check it.
+
+
+### Detecting provides
+
+While dependency is something, that is required (imported) by your project, provides are requirements, that are exported by other projects for yours.
+
+To detect provides for our **src** use **py3prov**:
+
+```shell
+% py3prov src
+src.pkg1.subpkg.mod3
 src.pkg1.mod1
+```
+
+To get all possible provides (including even modules) use **--full_mode**:
+
+```shell
+% py3prov --full_mode src
 mod3
 subpkg.mod3
 pkg1.subpkg.mod3
 src.pkg1.subpkg.mod3
+mod1
+pkg1.mod1
+src.pkg1.mod1
 ```
-Yeah, let's enhance the verbosity level!
+
+But all provides are prefixed by **src**, while your project should install **pkg1** in user system. To remove such prefixes use **--prefixes** option:
+
+```shell
+% py3prov --prefixes src src
+pkg1.subpkg.mod3
+pkg1.mod1
 ```
-% python3 -m py3dephell.py3prov --verbose src/pkg1 src/tests
-src/tests:['test1', 'tests.test1', 'src.tests.test1']
-src/pkg1:['mod1', 'pkg1.mod1', 'src.pkg1.mod1', 'mod3', 'subpkg.mod3', 'pkg1.subpkg.mod3', 'src.pkg1.subpkg.mod3']
+
+By default **--prefixes** is set to **sys.path**, while **$TMP/env/lib/python3/site-packages/** is included in **sys.path**.
+
+```shell
+% py3prov  $TMP/env/lib/python3/site-packages/py3dephell
+py3dephell.__init__
+py3dephell
+py3dephell.py3prov
+py3dephell.py3req
 ```
+
+
+
+Other options, such as **--only_prefix** and **--skip_pth** are little bit specific, but it is clear, what they can be used for. **--only_prefix** exclude those provides, that are not under prefixes. **--skip_pth** ignore [**.pth**](https://docs.python.org/3/library/site.html) files
+
+
+# API documentation
+For **API** documentation just use **help** command from interpreter or visit this [link](https://altlinux.github.io/py3dephell/).

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "py3dephell"
-version = "0.2.1"
+version = "0.3.1"
 authors = [
 {name = "Daniel Zagaynov", email = "[email protected]"},
 ]
@@ -23,7 +23,7 @@ requires = ["setuptools"]
 build-backend = "setuptools.build_meta"
 
 [project.urls]
-Homepage = "https://git.altlinux.org/people/kotopesutility/packages/py3dephell.git"
+Homepage = "https://github.com/altlinux/py3dephell.git"
 
 [project.scripts]
 py3req = "py3dephell.py3req:main"

src/py3dephell/py3prov.py

@@ -42,7 +42,8 @@ def processing_pth(path):
 
 
 def create_provides_from_path(path, prefixes=sys.path, abs_mode=False,
-                              pkg_mode=False, skip_wrong_names=True, skip_namespace_pkgs=True):
+                              pkg_mode=False, skip_wrong_names=True, skip_namespace_pkgs=True, verbose=False,
+                              _bad_provides=set()):
     '''
     Creates provides from given path for 1 file.
 
@@ -58,6 +59,8 @@ def create_provides_from_path(path, prefixes=sys.path, abs_mode=False,
     :type skip_wrong_names: Bool
     :param skip_namespace_pkgs: do not build provides for namespace packages
     :type skip_namespace_pkgs: Bool
+    :param verbose: turn on verbose mode
+    :type verbose: Bool
     :return: list of provides created from given path
     :rtype: list[str]
     '''
@@ -97,7 +100,9 @@ def create_provides_from_path(path, prefixes=sys.path, abs_mode=False,
             top_package_flag = True
 
         if '.' in parts[-1]:
-            print(f'py3prov: bad name for provides from path:{path.as_posix()}', file=sys.stderr)
+            if parts[-1] not in _bad_provides and verbose:
+                print(f'py3prov: bad name for provides from path:{path.as_posix()}', file=sys.stderr)
+                _bad_provides.add(parts[-1])
 
         if abs_mode and (all([part.isidentifier() for part in parts]) or not skip_wrong_names):
             provides.append('.'.join(parts))
@@ -114,13 +119,15 @@ def create_provides_from_path(path, prefixes=sys.path, abs_mode=False,
 
     if (top_package_flag or not skip_namespace_pkgs) and parent.as_posix() != '.':
         provides += create_provides_from_path(parent, prefixes,
-                                              pkg_mode=True, abs_mode=abs_mode, skip_wrong_names=skip_wrong_names)
+                                              pkg_mode=True, abs_mode=abs_mode, skip_wrong_names=skip_wrong_names,
+                                              verbose=verbose, _bad_provides=_bad_provides)
 
     return provides
 
 
 def search_for_provides(path, prefixes=sys.path, abs_mode=False,
-                        skip_wrong_names=True, skip_namespace_pkgs=True):
+                        skip_wrong_names=True, skip_namespace_pkgs=True, verbose=False,
+                        _bad_provides=set()):
     '''
     This function walks through given path and search for provides
 
@@ -134,6 +141,8 @@ def search_for_provides(path, prefixes=sys.path, abs_mode=False,
     :type skip_wrong_names: Bool
     :param skip_namespace_pkgs: do not build provides for namespace packages
     :type skip_namespace_pkgs: Bool
+    :param verbose: turn on verbose mode
+    :type verbose: Bool
     :return: list of provides created from given path
     :rtype: list[str]
     '''
@@ -142,10 +151,12 @@ def search_for_provides(path, prefixes=sys.path, abs_mode=False,
 
     if path.is_file() or path.is_symlink():
         return create_provides_from_path(path.as_posix(), prefixes, abs_mode=abs_mode,
-                                         skip_wrong_names=skip_wrong_names, skip_namespace_pkgs=skip_namespace_pkgs)
+                                         skip_wrong_names=skip_wrong_names, skip_namespace_pkgs=skip_namespace_pkgs,
+                                         verbose=verbose, _bad_provides=_bad_provides)
     elif path.is_dir() and '__pycache__' not in path.as_posix():
         for subpath in path.iterdir():
-            provides += search_for_provides(subpath, prefixes, abs_mode, skip_wrong_names, skip_namespace_pkgs)
+            provides += search_for_provides(subpath, prefixes, abs_mode, skip_wrong_names, skip_namespace_pkgs,
+                                            verbose=verbose, _bad_provides=_bad_provides)
     return provides
 
 
@@ -287,7 +298,8 @@ def generate_provides(files, prefixes=sys.path, skip_pth=False, only_prefix=Fals
     for path, module_name in files_dict.items():
         provides[path] = {'provides': search_for_provides(path, prefixes, abs_mode=abs_mode,
                                                           skip_wrong_names=skip_wrong_names,
-                                                          skip_namespace_pkgs=skip_namespace_pkgs),
+                                                          skip_namespace_pkgs=skip_namespace_pkgs,
+                                                          _bad_provides=set(), verbose=verbose),
                           'package': module_name}
 
     if not skip_pth:
@@ -313,8 +325,8 @@ def generate_provides(files, prefixes=sys.path, skip_pth=False, only_prefix=Fals
 def main():
     args = argparse.ArgumentParser(description='Search provides for module')
     args.add_argument('--prefixes', help='List of prefixes')
-    args.add_argument('--abs_mode', action='store_true',
-                      help='Turn on plugin mode (build only absolute provides)')
+    args.add_argument('--full_mode', action='store_true',
+                      help='Build all provides, not just absolute')
     args.add_argument('--only_prefix', action='store_true',
                       help='Skip all provides, that are not in prefix')
     args.add_argument('--skip_pth', action='store_true', help='Skip pth files')
@@ -329,7 +341,7 @@ def main():
     prefixes = args.prefixes.split(',') if args.prefixes else sys.path
 
     path_provides = generate_provides(files=args.input, prefixes=prefixes,
-                                      skip_pth=args.skip_pth, abs_mode=args.abs_mode,
+                                      skip_pth=args.skip_pth, abs_mode=not args.full_mode,
                                       only_prefix=args.only_prefix, verbose=args.verbose)
     for path, provides in path_provides.items():
         if args.verbose: