Skip to content

Comments

feat: Add GitHub CI/CD versioning support with env profile#393

Merged
Chaffelson merged 8 commits intomainfrom
feature/github-cicd-versioning
Dec 8, 2025
Merged

feat: Add GitHub CI/CD versioning support with env profile#393
Chaffelson merged 8 commits intomainfrom
feature/github-cicd-versioning

Conversation

@Chaffelson
Copy link
Owner

@Chaffelson Chaffelson commented Dec 3, 2025

| GitHub CI/CD Integration - Native support for NiFi's GitHub Flow Registry Client

GitHub Flow Registry Support

  • Git-specific versioning helpers: New functions to work with NiFi's native GitHub Flow Registry Client
    • list_git_registry_buckets: List buckets (folders) in a Git-backed registry
    • get_git_registry_bucket: Get a specific bucket by name
    • list_git_registry_flows: List flows in a bucket
    • get_git_registry_flow: Get a specific flow by name
    • list_git_registry_flow_versions: List all versions (commits) of a flow
    • deploy_git_registry_flow: Deploy a versioned flow from GitHub to the NiFi canvas
    • update_git_flow_ver: Change version of an already-deployed Git-registry flow
  • Registry client management: ensure_registry_client and update_registry_client for idempotent registry configuration
  • Enhanced revert: revert_flow_ver now accepts wait=True parameter for synchronous operation

Profile System Enhancements

  • "env" profile for CI/CD: New special profile name that configures nipyapi entirely from environment variables
    • No profiles file required - ideal for GitHub Actions, containers, and CI/CD pipelines
    • Uses nipyapi.profiles.switch('env') to activate
    • All standard environment variable mappings (NIFI_API_ENDPOINT, NIFI_USERNAME, etc.) apply

Controller Service Management

  • Bulk controller service operations: schedule_all_controllers(pg_id, scheduled) to enable/disable all controller services in a process group
    • Uses NiFi's native bulk activation API
    • Handles all descendant controller services automatically
    • Simplifies flow start/stop operations in CI/CD workflows

Bug Fixes

  • Fixed test_create_controller leaving orphaned ADLS controller services after test runs
  • Improved test cleanup fixtures for better isolation
  • Fixed version sorting in deploy_git_registry_flow to correctly identify latest version

Infrastructure

  • Updated pylintrc for pylint 3.x compatibility
  • Increased module line limit for versioning.py to accommodate new functions
  • Added comprehensive test fixtures for Git registry integration testing

Related Projects

Add native support for NiFi's GitHub Flow Registry Client to enable
CI/CD workflows for versioned NiFi flows.

New Git-specific versioning helpers:
- list_git_registry_buckets, get_git_registry_bucket
- list_git_registry_flows, get_git_registry_flow
- list_git_registry_flow_versions, deploy_git_registry_flow
- ensure_registry_client, update_registry_client

Profile system enhancements:
- Add 'env' profile for pure environment variable configuration
- Ideal for GitHub Actions, containers, and CI/CD pipelines
- No profiles file required when using nipyapi.profiles.switch('env')

Controller service management:
- Add schedule_all_controllers for bulk enable/disable operations
- Uses NiFi's native bulk activation API with descendant support

Bug fixes:
- Fix test_create_controller leaving orphaned controller services

Documentation:
- Add 1.1.0 release notes to history
- Add env profile usage guide to profiles documentation

Related: nipyapi-actions and nipyapi-workflow companion repositories
@codecov-commenter
Copy link

codecov-commenter commented Dec 3, 2025

Codecov Report

❌ Patch coverage is 77.57848% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.82%. Comparing base (107e324) to head (39e2fd5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
nipyapi/versioning.py 79.65% 35 Missing ⚠️
nipyapi/profiles.py 60.00% 10 Missing ⚠️
nipyapi/canvas.py 80.76% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #393      +/-   ##
==========================================
+ Coverage   70.38%   70.82%   +0.44%     
==========================================
  Files           9        9              
  Lines        1830     2026     +196     
==========================================
+ Hits         1288     1435     +147     
- Misses        542      591      +49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Add nifi-github, nifi-github.localdomain, host.docker.internal to certificate SANs
- Configure nifi-github service to mount generated certificates
- Add NIFI_WEB_PROXY_HOST for host.docker.internal access
- Enables act testing with proper SSL certificate validation
- update_processor now accepts name param for renaming processors
- add auto_stop param (default False) to handle running processors
- revert_flow_ver now refreshes revision internally to prevent stale errors
@Chaffelson Chaffelson marked this pull request as ready for review December 4, 2025 17:15
@Chaffelson Chaffelson self-assigned this Dec 4, 2025
Copy link
Collaborator

@ottobackwards ottobackwards left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Minor version bump?

- Add nifi_bearer_token to configuration structure and keys
- Document bearer as highest priority NiFi auth method
- Add bearer-token profile example in profiles.yml
- Add NIFI_BEARER_TOKEN environment variable mapping
@Chaffelson
Copy link
Owner Author

This looks good to me. Minor version bump?

Yeah this is additional functionality, should be entirely backwards compatible but it should be a 1.1.0 release.
I've also added the bearer token direct support commit from my local set of changes, so this should generally be ready to use for initial CICD testing and I'll work one some more actions / helper functions for the next round.

@Chaffelson Chaffelson merged commit 4507a52 into main Dec 8, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants