Support releasing individual packages in nimi-python (#2090)

* Update of changelog and python scripts to support the indivitual versioning .

* Updated files are per new implementation

* CHangelog Sample

* New ChangeLog

* Updated changelog

* ni Dc power to Test

* Added for nidmm and digital

* Fixed for all drviers

* Including seperator

* Spacing issue fixed

* Updated Contributing.md file

* Updated alignment

* Alignment fixed

* Updated help in UpdateReleaseInfo.py

* Accept only one input

* Updated changelog and contributing.md

* Contibuting MD  version change.

* changes for flake command

* Will be moving the changelog changes to different PR

* Included changes for code reveiw comments.

* Revert "Included changes for code reveiw comments."

This reverts commit 9de9d57.

* Reapply "Included changes for code reveiw comments."

This reverts commit cc126e8.

* Minor bug fix.

* Code Review comments implementations

* Removed build version increment

* Updated review comments

* Code review comments incorporated.

* Updating example.rst.mako and check_latest_release.yml file

* Incorporated review comments.

* check_latest_release.ywm file updated

---------

Co-authored-by: dummy <[email protected]>

Author: rahulr-NI

.github/workflows/check_latest_release.yml

@@ -25,24 +25,23 @@ jobs:
       - x64
       - rdss-nimibot-win-10-py32
     timeout-minutes: 30
-    strategy:
-      matrix:
-        module_name:
-          - nidigital
-          - nitclk
-          - nifgen
-          - nidcpower
-          - nidmm
-          - niscope
-          - nimodinst
-          - nise
-          - niswitch
     steps:
       - name: checkout repository
         uses: actions/checkout@v3
+
+      - name: Extract module name and version from release tag
+        id: extract_tag
+        run: |
+          # Extract module name and version from the release tag
+          # Assuming the tag format is <module_name>-<version>, e.g., nidigital-1.4.0
+          TAG="${{ github.event_name == 'workflow_dispatch' && inputs.release_tag || github.event.release.tag_name }}"
+          MODULE_NAME=$(echo "$TAG" | cut -d'-' -f1)
+          MODULE_VERSION=$(echo "$TAG" | cut -d'-' -f2-)
+          echo "module_name=$MODULE_NAME" >> "$GITHUB_OUTPUT"
+          echo "module_version=$MODULE_VERSION" >> "$GITHUB_OUTPUT"
       # NOTE: we don't upload test coverage for this
       - name: run examples using PyPI uploads
         uses: ./.github/actions/run_examples_using_pypi_uploads
         with:
-          module_name: ${{ matrix.module_name }}
-          module_version: ${{ github.event_name == 'workflow_dispatch' && inputs.release_tag || github.event.release.tag_name }}
+          module_name: ${{ steps.extract_tag.outputs.module_name }}
+          module_version: ${{ steps.extract_tag.outputs.module_version }}

CONTRIBUTING.md

@@ -149,6 +149,17 @@ Release Process
         ```
     1. Ensure no commits are made on ni/nimi-python/master until the release is complete
     1. Create and checkout a branch for release-related changes
+    1. Perform Version Bump (If Needed)
+        * If you need to upgrade the major or minor versions, include any of the following parameters:
+           * --increment-major-version - To increment the major version of package. This will update the version to (N+1).X.X.dev0
+           * --increment-minor-version - To increment the minor version of package. This will update the version to X.(N+1).X.dev0
+             * Example: `python3 tools/build_release.py --increment-minor-version`
+        * If you need to update the version for any specific driver(s), include the `drivers` parameter. By default, all drivers will be considered.
+          For example: 
+              ```bash
+              python3 tools/build_release.py --drivers nidcpower --increment-minor-version
+              ```
+        * Commit to branch
     1. Update [CHANGELOG.md](./CHANGELOG.md)
         * For packages that are releasing:
           * Delete empty (i.e. No changes) sub-sections under "Unreleased" section 
@@ -157,8 +168,13 @@ Release Process
           * Change [Unreleased] in TOC to the version of the release
         * Commit to branch
     1. Update release versions
-        * `python3 tools/build_release.py --update --release`
+        * `python3 tools/build_release.py --update-for-release`
             * For each module, this will drop the .devN from our versions in config_addon.py and update the LATEST_RELEASE versions to match.
+            * If you need to release any specific module(s), include the `drivers` parameter. 
+          For example: 
+              ```bash
+              python3 tools/build_release.py --drivers nidcpower --update-for-release
+              ```
         * Commit to branch
     1. Clean and build to update generated files with new version
         * `python3 tools/build_release.py --build`
@@ -170,19 +186,33 @@ Release Process
     1. Wait until the pull request has been approved
     1. Upload the releases to PyPI
         * `python3 tools/build_release.py --upload`
+        * If you need to upload any specific module(s), include the `drivers` parameter. 
+        For example: 
+             ```bash
+            python3 tools/build_release.py --drivers nidcpower --upload
+            ```
         * You will need to type in your PyPI credentials
     1. Merge the pull request to origin/master
-    1. Create a release on GitHub using the portion from the changelog for this release for the description
-        * Add the ZIP files under `generated/examples` for each module as a release artifact.
+    1. For each package released, create a release on GitHub using the module's portion from the changelog for this release for the description
+        * The release tag should be named as follows: `MODULE_NAME-version`.
+          * Example: `nidcpower-1.5.0`.
+          * This tag format allows the individual `Read the Docs` projects to determine whether a release applies to them.
+        * Add the ZIP files under `generated/examples` for each module (not just the releasing one) as a release artifact.
+          * Internal test code will only look for the latest release tag and expect it to have examples attached for any module
         * This should trigger the [check_latest_release](.github/workflows/check_latest_release.yml) workflow. Check the [results](https://github.com/ni/nimi-python/actions/workflows/check_latest_release.yml) before continuing.
 1. Post-Release Steps
     1. Create and checkout another branch for post-release changes
-    1. Update the module versions
-        * `python3 tools/build_release.py --update`
-            * This will update the version to X.X.(N+1).dev0
+    1. Update the module version for a patch version upgrade. This will update the version to X.X.(N+1).dev0
+        * `python3 tools/build_release.py --increment-patch-version`
+        * If you need to update any specific module(s), include the `drivers` parameter. 
+        For example: 
+             ```bash
+            python3 tools/build_release.py --drivers nidcpower --increment-patch-version
+            ```
         * Commit to branch
     1. Clean and build to update generated files with new version
         * `python3 tools/build_release.py --build`
+        * Ensure that all changes made as part of build command are specific to intended drivers.
         * Commit to branch
     1. Update changelog
         * Copy Unreleased section from the bottom of the changelog. Modify the package name in the example and TOC. Paste the modified section at the top of intended package's changelog and add a corresponding link to it in the package's TOC.

build/templates/examples.rst.mako

@@ -27,13 +27,23 @@
 
     with open(f'./src/{module_name}/LATEST_RELEASE') as vf:
         latest_release_version = vf.read().strip()
-    released_zip_url = 'https://github.com/ni/nimi-python/releases/download/{}/{}_examples.zip'.format(latest_release_version, module_name)
-
-    example_url_base = 'https://github.com/ni/nimi-python/blob/'
 
     from packaging.version import Version
     v = Version(module_version)
 
+    # Check if the module name and version match the old tag formatting criteria
+    use_old_tag_format = (
+        module_name in ['nidcpower', 'nidigital', 'nidmm', 'nifake', 'niswitch', 'nimodinst', 'nifgen', 'niscope', 'nise', 'nitclk']
+        and latest_release_version == '1.4.9'
+    )
+
+    if not use_old_tag_format:
+        latest_release_version = module_name + '-' + latest_release_version
+    
+    released_zip_url = 'https://github.com/ni/nimi-python/releases/download/{}/{}_examples.zip'.format(latest_release_version, module_name)
+
+    example_url_base = 'https://github.com/ni/nimi-python/blob/'
+
     if v.dev is None and v.pre is None:
         examples_zip_url_text = '`You can download all {} examples here <{}>`_'.format(module_name, released_zip_url)
         example_url_base += latest_release_version
@@ -54,4 +64,4 @@ ${helper.get_rst_header_snippet(os.path.basename(e), '-')}
    :encoding: utf8
    :caption: `(${os.path.basename(e)}) <${example_url_base}/${e.replace('\\', '/')}>`_
 
-% endfor
+% endfor

tools/build_release.py

@@ -9,8 +9,6 @@
 pp = pprint.PrettyPrinter(indent=4, width=100)
 
 default_python_cmd = ['python']
-drivers_to_upload = ['nidcpower', 'nidigital', 'nidmm', 'niswitch', 'nimodinst', 'nifgen', 'niscope', 'nise', 'nitclk']
-drivers_to_update = ['nifake'] + drivers_to_upload
 
 
 class CustomFormatter(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
@@ -23,6 +21,8 @@ class CustomFormatter(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescri
 
 
 def main():
+    drivers_to_update = ['nidcpower', 'nidigital', 'nidmm', 'nifake', 'niswitch', 'nimodinst', 'nifgen', 'niscope', 'nise', 'nitclk']
+
     # Setup the required arguments for this script
     usage = """Release script
 Prereqs
@@ -33,18 +33,31 @@ def main():
     parser = argparse.ArgumentParser(description=usage, formatter_class=CustomFormatter)
 
     build_group = parser.add_argument_group("Build configuration")
-    build_group.add_argument("--release", action="store_true", default=False, help="This is a release build, so only remove '.devN'. build, then update with .dev0")
     build_group.add_argument("--upload", action="store_true", default=False, help="Upload build distributions to PyPI")
-    build_group.add_argument("--update", action="store_true", default=False, help="Update version in config.py files")
     build_group.add_argument("--build", action="store_true", default=False, help="Clean and build")
     build_group.add_argument("--python-cmd", action="store", default=None, help=f"Command to use for invoking python. Default: {default_python_cmd}")
+    build_group.add_argument("--drivers", action="store", default=None, help="Comma-separated list of drivers to update. Default: All Drivers")
+    build_group.add_argument("--increment-major-version", action="store_true", default=False, help="Increment the major version")
+    build_group.add_argument("--increment-minor-version", action="store_true", default=False, help="Increment the minor version")
+    build_group.add_argument("--increment-patch-version", action="store_true", default=False, help="Increment the patch version")
+    build_group.add_argument("--update-for-release", action="store_true", default=False, help="This is a release build, so only remove '.devN'. build, then update with .dev0")
 
     verbosity_group = parser.add_argument_group("Verbosity, Logging & Debugging")
     verbosity_group.add_argument("-v", "--verbose", action="count", default=0, help="Verbose output")
     verbosity_group.add_argument("--preview", action="store_true", default=False, help="Show what would happen when running with given parameters")
     verbosity_group.add_argument("--log-file", action="store", default=None, help="Send logging to listed file instead of stdout")
     args = parser.parse_args()
 
+    # Validate that only one of the version-related flags is provided
+    version_flags = [
+        args.increment_major_version,
+        args.increment_minor_version,
+        args.increment_patch_version,
+        args.update_for_release,
+    ]
+    if sum(version_flags) > 1:
+        raise ValueError("Only one of --increment-major-version, --increment-minor-version, --increment-patch-version or --update-for-release can be provided.")
+
     if args.verbose > 1:
         configure_logging(logging.DEBUG, args.log_file)
     elif args.verbose == 1:
@@ -64,14 +77,28 @@ def main():
         passthrough_params.append('--preview')
     if args.log_file:
         passthrough_params.append('--log-file').append(args.log_file)
-    if args.release:
+    if args.update_for_release:
         passthrough_params.append('--release')
-
-    if args.update:
+    if args.increment_patch_version:
+        passthrough_params.append('--update-type=patch')
+    if args.increment_minor_version:
+        passthrough_params.append('--update-type=minor')
+    if args.increment_major_version:
+        passthrough_params.append('--update-type=major')
+
+    if args.drivers:
+        provided_drivers = args.drivers.split(",")
+        invalid_drivers = [driver for driver in provided_drivers if driver not in drivers_to_update]
+
+        if invalid_drivers:
+            raise ValueError(f"The following drivers are invalid: {', '.join(invalid_drivers)}. Valid drivers are: {','.join(drivers_to_update)}")
+        drivers_to_update = provided_drivers
+
+    if any([args.increment_major_version, args.increment_minor_version, args.increment_patch_version, args.update_for_release]):
         logging.info('Updating versions')
 
         for d in drivers_to_update:
-            logging.info(pp.pformat(python_cmd + ['tools/updateReleaseInfo.py', '--src-folder', f'src/{d}', ] + passthrough_params))
+            logging.info(pp.pformat(python_cmd + ['tools/updateReleaseInfo.py', '--src-folder', f'src/{d}',] + passthrough_params))
             check_call(python_cmd + ['tools/updateReleaseInfo.py', '--src-folder', f'src/{d}', ] + passthrough_params)
 
     if args.build:
@@ -86,6 +113,7 @@ def main():
     if args.upload:
         logging.info('Uploading to PyPI')
         complete_twine_cmd = twine_cmd + ['upload']
+        drivers_to_upload = [driver for driver in drivers_to_update if driver != 'nifake']
         for d in drivers_to_upload:
             complete_twine_cmd += [f'generated/{d}/dist/*']
 

tools/updateReleaseInfo.py

@@ -1,5 +1,3 @@
-# !python
-
 import argparse
 from configure_logging import configure_logging
 import logging
@@ -10,15 +8,32 @@
 pp = pprint.PrettyPrinter(indent=4, width=100)
 
 
+# Increment version based on bump type ('major', 'minor', 'patch').
+def bump_version(version, bump_type):
+    major, minor, patch = map(int, version.split('.'))
+
+    if bump_type == 'patch':
+        patch += 1
+    elif bump_type == 'minor':
+        minor += 1
+        patch = 0
+    elif bump_type == 'major':
+        major += 1
+        minor = 0
+        patch = 0
+
+    return f"{major}.{minor}.{patch}"
+
+
 def main():
-    # Setup the required arguments for this script
     usage = """
 Update version in files. Example: X.Y.Z.devN to X.Y.Z
 """
     parser = argparse.ArgumentParser(description=usage)
     file_group = parser.add_argument_group("Input and Output files")
     file_group.add_argument("--src-folder", action="store", required=True, help="Source folder")
     file_group.add_argument("--release", action="store_true", default=False, help="This is a release build, so only remove '.devN'. Error if not there")
+    file_group.add_argument("--update-type", action="store", default=None, choices=["major", "minor", "patch"], help="Specify the type of update: major, minor or patch. ")
 
     verbosity_group = parser.add_argument_group("Verbosity, Logging & Debugging")
     verbosity_group.add_argument("-v", "--verbose", action="count", default=0, help="Verbose output")
@@ -34,27 +49,37 @@ def main():
         configure_logging(logging.WARNING, args.log_file)
 
     logging.info(pp.pformat(args))
-
     metadata_file = os.path.join(args.src_folder, "metadata", "config_addon.py")
     with open(metadata_file) as content_file:
         contents = content_file.read()
 
-    module_dev_version_re = re.compile(r"'module_version': '(\d+\.\d+\.\d+)\.dev(\d+)'")
-    m = module_dev_version_re.search(contents)
+    module_version_re = re.compile(r"'module_version': '(\d+\.\d+\.\d+)(?:\.dev(\d+))?'")
+    m = module_version_re.search(contents)
+    logging.debug(f"Version regex match: {m}")
+
     if m:
+        base_version = m.group(1)
+        dev_number = int(m.group(2)) if m.group(2) else None
+
+        if dev_number is not None:
+            logging.info("Dev version found")
+            current_version = f"{base_version}.dev{dev_number}"
+        else:
+            logging.info("Release version found")
+            current_version = base_version
+
         if args.release:
-            logging.info('Dev version found, updating {0}.dev{1} to {0}'.format(m.group(1), int(m.group(2))))
-            contents = module_dev_version_re.sub(f"'module_version': '{m.group(1)}'", contents)
-            new_version = m.group(1)
+            if dev_number is not None:
+                new_version = base_version
+            else:
+                logging.error("Error: Attempting to release an already released version.")
+                return
         else:
-            logging.info('Dev version found, updating {0}.dev{1} to {0}.dev{2}'.format(m.group(1), int(m.group(2)), int(m.group(2)) + 1))
-            contents = module_dev_version_re.sub(f"'module_version': '{m.group(1)}.dev{int(m.group(2)) + 1}'", contents)
+            bumped_version = bump_version(base_version, args.update_type)
+            new_version = f"{bumped_version}.dev0"
 
-    module_version_re = re.compile(r"'module_version': '(\d+\.\d+\.)(\d+)'")
-    m = module_version_re.search(contents)
-    if m and not args.release:
-        logging.info('Release version found, updating {0}{1} to {0}{2}.dev0'.format(m.group(1), int(m.group(2)), int(m.group(2)) + 1))
-        contents = module_version_re.sub(f"'module_version': '{m.group(1)}{int(m.group(2)) + 1}.dev0'", contents)
+        logging.info(f"Updating {current_version} to {new_version}")
+        contents = module_version_re.sub(f"'module_version': '{new_version}'", contents)
 
     if not args.preview:
         with open(metadata_file, 'w') as content_file:
@@ -70,4 +95,3 @@ def main():
 
 if __name__ == '__main__':
     main()
-