Populating whl and imports entrypoint (#42677)

* Add Check class to be used as common definition for a check
* add whl and import_all check entrypoints
* refactor to allow automatic venv isolation
* update readme
  
Co-authored-by: Laurent Mazuel <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>

Author: 3 people

.gitignore

@@ -107,8 +107,8 @@ src/build
 
 # [begoldsm] ignore virtual env if it exists.
 adlEnv/
-venv/
-.venv/
+venv*
+.venv*
 code_reports
 
 # Azure Storage test credentials

eng/tools/azure-sdk-tools/README.md

@@ -24,10 +24,13 @@ This package is intended for usage in direct combination with the azure-sdk-for-
 
 | Module | Description |
 |---|---|
+| `azpysdk` | CI check entrypoints and accompanying implementations. |
+| `ci_tools` | Various azure-sdk-for-python specific build and test abstractions. Heavily used in CI functionality. |
 | `devtools_testutils` | Primary location for test classes, pytest fixtures, and test-proxy integration. |
-| `ci_tools` | Various azure-sdk-for-python specific build and test abstractions. |
 | `packaging_tools` | Templated package generator for management packages. |
+| `parsing` | Parse information _about_ python packages at rest on disk. Used to interrogate the monorepo to find relevant packages. |
 | `pypi_tools` | Helper functionality build upon interactions with PyPI. |
+| `scenario` | Functionality to do with installing packages constrained by environmental and input factors. Heavily used in `mindependency` and `latestdependency` CI checks. |
 | `testutils` | Backwards compatible extension of test classes. |
 
 **PLEASE NOTE.** For the "script" entrypoints provided by the package, all should either be run from somewhere **within** the azure-sdk-for-python repository. Barring that, an argument `--repo` should be provided that points to the repo root if a user must start the command from a different CWD.
@@ -37,33 +40,39 @@ This package is intended for usage in direct combination with the azure-sdk-for-
 After installing azure-sdk-tools, package build functionality is available through `sdk_build`.
 
 ```text
-usage: sdk_build [-h] [-d DISTRIBUTION_DIRECTORY] [--service SERVICE] [--pkgfilter PACKAGE_FILTER_STRING] [--devbuild IS_DEV_BUILD]
-                 [--produce_apiview_artifact] [--repo REPO] [--build_id BUILD_ID]
+usage: sdk_build [-h] [-d DISTRIBUTION_DIRECTORY] [--service SERVICE] [--pkgfilter PACKAGE_FILTER_STRING]
+                 [--devbuild IS_DEV_BUILD] [--inactive] [--produce_apiview_artifact] [--repo REPO] [--build_id BUILD_ID]
                  [glob_string]
 
-This is the primary entrypoint for the "build" action. This command is used to build any package within the azure-sdk-for-python repository.
+This is the primary entrypoint for the "build" action. This command is used to build any package within the azure-sdk-for-
+python repository.
 
 positional arguments:
-  glob_string           A comma separated list of glob strings that will target the top level directories that contain packages. Examples: All == "azure-*",
-                        Single = "azure-keyvault"
+  glob_string           A comma separated list of glob strings that will target the top level directories that contain
+                        packages. Examples: All == "azure-*", Single = "azure-keyvault"
 
-optional arguments:
+options:
   -h, --help            show this help message and exit
   -d DISTRIBUTION_DIRECTORY, --distribution-directory DISTRIBUTION_DIRECTORY
-                        The path to the distribution directory. Should be passed $(Build.ArtifactStagingDirectory) from the devops yaml definition.If that
-                        is not provided, will default to env variable SDK_ARTIFACT_DIRECTORY -> <calculated_repo_root>/.artifacts.
+                        The path to the distribution directory. Should be passed $(Build.ArtifactStagingDirectory) from the
+                        devops yaml definition.If that is not provided, will default to env variable SDK_ARTIFACT_DIRECTORY
+                        -> <calculated_repo_root>/.artifacts.
   --service SERVICE     Name of service directory (under sdk/) to build.Example: --service applicationinsights
   --pkgfilter PACKAGE_FILTER_STRING
-                        An additional string used to filter the set of artifacts by a simple CONTAINS clause. This filters packages AFTER the set is built
-                        with compatibility and omission lists accounted.
+                        An additional string used to filter the set of artifacts by a simple CONTAINS clause. This filters
+                        packages AFTER the set is built with compatibility and omission lists accounted.
   --devbuild IS_DEV_BUILD
-                        Set build type to dev build so package requirements will be updated if required package is not available on PyPI
+                        Set build type to dev build so package requirements will be updated if required package is not
+                        available on PyPI
+  --inactive            Include inactive packages when assembling artifacts. CI builds will include inactive packages as a
+                        way to ensure that the yml controlled artifacts can be associated with a wheel/sdist.
   --produce_apiview_artifact
-                        Should an additional build artifact that contains the targeted package + its direct dependencies be produced?
-  --repo REPO           Where is the start directory that we are building against? If not provided, the current working directory will be used. Please
-                        ensure you are within the azure-sdk-for-python repository.
-  --build_id BUILD_ID   The current build id. It not provided, will default through environment variables in the following order: GITHUB_RUN_ID ->
-                        BUILD_BUILDID -> SDK_BUILD_ID -> default value.
+                        Should an additional build artifact that contains the targeted package + its direct dependencies be
+                        produced?
+  --repo REPO           Where is the start directory that we are building against? If not provided, the current working
+                        directory will be used. Please ensure you are within the azure-sdk-for-python repository.
+  --build_id BUILD_ID   The current build id. It not provided, will default through environment variables in the following
+                        order: GITHUB_RUN_ID -> BUILD_BUILDID -> SDK_BUILD_ID -> default value.
 ```
 
 Some common invocations.
@@ -149,6 +158,77 @@ optional arguments:
                         ensure you are within the azure-sdk-for-python repository.
 ```
 
+## Using `Parsing` Modules
+
+```python
+from ci_tools.parsing import ParsedSetup
+
+path_to_package = "path/to/your/possible/python/package"
+
+pkg_metadata = ParsedSetup.from_path(path_to_package)
+
+# pkg_metadata will contain all metadata about the package, EG "name", "version", "python_requires", "dependencies"
+# any information that any of our checks would need to operate properly.
+```
+
+## Writing additional `checks` within `azpysdk`
+
+To add a new check to `azpysdk`, follow these steps.
+
+- Create a new file within `azpysdk`. Name it after what your check will be.
+- When creating a class for your new check, you should inherit from the class `Check` within the `azpysdk` namespace.
+- Implement the required functions within the `Check` abstract class in your new check class.
+
+This is a commented implementation providing the reasoning for implementation alongside the code.
+
+```python
+class my_check(Check):
+    def __init__(self) -> None:
+        super().__init__()
+
+
+    def register(self, subparsers: "argparse._SubParsersAction", parent_parsers: Optional[List[argparse.ArgumentParser]] = None) -> None:
+        """Register `my_check`. `my_check` does X and Y while doing Z.
+
+        This function will be used to add your command to the set of commands being shown through the `azpysdk` entrypoint.
+        """
+        parents = parent_parsers or []
+        p = subparsers.add_parser("whl", parents=parents, help="Run the `my_check` check")
+        p.set_defaults(func=self.run)
+        # Add any additional arguments specific to your check here (do not re-add common handled by parents. See `main.py` build_parser to see the common parents provided to the checks)
+
+    def run(self, args: argparse.Namespace) -> int:
+        """This is the recommended """
+        set_envvar_defaults()
+
+        if args.target == ".":
+            targeted = [os.getcwd()]
+        else:
+            target_dir = os.getcwd()
+            targeted = discover_targeted_packages(args.target, target_dir)
+        results: List[int] = []
+
+        for pkg in targeted:
+            parsed = ParsedSetup.from_path(pkg)
+            print(f"Processing {pkg.name} for my_check")
+```
+
+- Once the new check has been created, register it in `azpysdk/main.py` on line 58. This will likely be automated by decorator in the near future but this is out how it should be done for now.
+
+```python
+# ...other registrations above
+my_check().register(subparsers, [common])
+```
+
+- Your command will now be available through `azpysdk`
+
+```bash
+/> `azpysdk mycheck azure-storage*`
+
+/> cd sdk/storage/azure-storage
+sdk/storage/azure-storage/> azpysdk mycheck
+```
+
 ## Relevant Environment Variables
 
 This package honors a few different environment variables as far as logging, artifact placement, etc.

eng/tools/azure-sdk-tools/azpysdk/Check.py

@@ -0,0 +1,31 @@
+import abc
+import argparse
+from typing import Sequence, Optional, List, Any
+
+class Check(abc.ABC):
+    """
+    Base class for checks.
+
+    Subclasses must implement register() to add a subparser for the check.
+    """
+
+    def __init__(self) -> None:
+        pass
+
+    @abc.abstractmethod
+    def register(self, subparsers: "argparse._SubParsersAction", parent_parsers: Optional[List[argparse.ArgumentParser]] = None) -> None:
+        """
+        Register this check with the CLI subparsers.
+
+        `subparsers` is the object returned by ArgumentParser.add_subparsers().
+        `parent_parsers` can be a list of ArgumentParser objects to be used as parents.
+        Subclasses MUST implement this method.
+        """
+        raise NotImplementedError
+
+    def run(self, args: argparse.Namespace) -> int:
+        """Run the check command.
+
+        Subclasses can override this to perform the actual work.
+        """
+        return 0

eng/tools/azure-sdk-tools/azpysdk/import_all.py

@@ -0,0 +1,93 @@
+import argparse
+import os
+import sys
+import logging
+import tempfile
+
+from typing import Optional,List
+from subprocess import check_call
+
+from .Check import Check
+from ci_tools.parsing import ParsedSetup
+from ci_tools.functions import discover_targeted_packages
+from ci_tools.scenario.generation import create_package_and_install
+
+# keyvault has dependency issue when loading private module _BearerTokenCredentialPolicyBase from azure.core.pipeline.policies
+# azure.core.tracing.opencensus and azure.eventhub.checkpointstoreblob.aio are skipped due to a known issue in loading azure.core.tracing.opencensus
+excluded_packages = [
+    "azure",
+    "azure-mgmt",
+    ]
+
+def should_run_import_all(package_name: str) -> bool:
+    return not (package_name in excluded_packages or "nspkg" in package_name)
+
+class import_all(Check):
+    def __init__(self) -> None:
+        super().__init__()
+
+    def register(self, subparsers: "argparse._SubParsersAction", parent_parsers: Optional[List[argparse.ArgumentParser]] = None) -> None:
+        """Register the `import_all` check. The import_all check checks dependencies of a package
+        by installing just the package + its dependencies, then attempts to import * from the base namespace.
+
+        If it fails, there is a dependency being imported somewhere in the package namespace that doesn't
+        exist in the `pyproject.toml`/`setup.py` dependencies. EG failure to import `isodate` within `azure.storage.blob.BlobClient`
+        because `isodate` is not listed as a dependency for the package.
+        """
+        parents = parent_parsers or []
+        p = subparsers.add_parser("import_all", parents=parents, help="Run the import_all check")
+        p.set_defaults(func=self.run)
+        # Add any additional arguments specific to WhlCheck here (do not re-add common args)
+
+    # todo: figure out venv abstraction mechanism via override
+    def run(self, args: argparse.Namespace) -> int:
+        """Run the import_all check command."""
+        print("Running import_all check in isolated venv...")
+
+        # this is common. we should have an abstraction layer for this somehow
+        if args.target == ".":
+            targeted = [os.getcwd()]
+        else:
+            target_dir = os.getcwd()
+            targeted = discover_targeted_packages(args.target, target_dir)
+
+        # {[tox]pip_command} freeze
+        # python {repository_root}/eng/tox/import_all.py -t {tox_root}
+
+        outcomes: List[int] = []
+
+        for pkg in targeted:
+            parsed = ParsedSetup.from_path(pkg)
+
+            staging_area = tempfile.mkdtemp()
+            create_package_and_install(
+                distribution_directory=staging_area,
+                target_setup=pkg,
+                skip_install=False,
+                cache_dir=None,
+                work_dir=staging_area,
+                force_create=False,
+                package_type="wheel",
+                pre_download_disabled=False,
+            )
+
+            if should_run_import_all(parsed.name):
+                # import all modules from current package
+                logging.info(
+                    "Importing all modules from namespace [{0}] to verify dependency".format(
+                        parsed.namespace
+                    )
+                )
+                import_script_all = "from {0} import *".format(parsed.namespace)
+                commands = [
+                    sys.executable,
+                    "-c",
+                    import_script_all
+                ]
+
+                outcomes.append(check_call(commands))
+                logging.info("Verified module dependency, no issues found")
+            else:
+                logging.info("Package {} is excluded from dependency check".format(parsed.name))
+
+        return max(outcomes) if outcomes else 0