CLI and CI/CD Automation

v1.2.0 - CLI and CI/CD Automation

CLI and CI/CD automation release

Command-line interface for shell scripting and pipeline integration (tested against NiFi 2.7.2)

Command-Line Interface

• Optional CLI install (pip install nipyapi[cli]): Shell access to all nipyapi modules via Google Fire
• JSON output by default with automatic CI platform detection (GitHub Actions, GitLab CI)
• Profile selection via --profile flag or environment variables
• --version flag and verbosity controls (-v, -vv, -vvv)

CI Operations Module (nipyapi.ci)

26 automation-focused commands for flow deployment pipelines:

• verify_config, ensure_registry, deploy_flow, start_flow, stop_flow
• list_flows, list_registry_flows, get_status, get_flow_versions
• commit_flow, detach_flow, get_flow_diff, revert_flow, change_flow_version
• configure_params, configure_inherited_params, export_parameters, upload_asset
• export_flow_definition, import_flow_definition, cleanup, purge_flowfiles
• list_nars, upload_nar, delete_nar, resolve_git_ref

New Modules

• nipyapi.bulletins: Bulletin retrieval with filtering; clear_all_bulletins for NiFi 2.7.0+
• nipyapi.extensions: NAR management, Python processor lifecycle, bundle version switching
• nipyapi.layout: Canvas positioning with below(), fork(), automatic flow layout, grid alignment
• nipyapi.nifi_registry: NiFi Registry operations extracted from versioning module

Canvas Module Enhancements

• FlowFile inspection: list_flowfiles, get_flowfile_details, get_flowfile_content, peek_flowfiles
• Component state: get_processor_state, clear_processor_state, get_controller_state, clear_controller_state
• schedule_processor now supports RUN_ONCE and DISABLED states
• schedule_all_controllers now waits for target state
• get_processor_docs, get_controller_service_docs for processor discovery
• get_flow_components returns connected subgraph for flow analysis

Parameters Module Enhancements

• get_parameter_context_hierarchy with include_bindings and include_parameters options

Utilities Enhancements

• fs_write supports binary=True for raw byte content
• is_uuid for UUID format detection
• format_timestamp for NiFi-compatible timestamps

Documentation

• CLI reference (docs/cli.rst)
• CI operations guide with examples (docs/ci.rst)
• NAR management guide (docs/extensions.rst)

Infrastructure

• Updated NiFi and Registry API clients to 2.7.2
• New NiFi 2.7.2 API features: bulletin stack traces, listen ports, clear bulletins
• Simplified revert_flow_ver() using VCI process_group_revision

Installation

pip install nipyapi==1.2.0
pip install "nipyapi[cli]==1.2.0"  # With CLI

Related

• nipyapi-actions - GitHub Actions and GitLab CI examples
• Documentation
• PyPI

v1.1.0 - GitHub CI/CD Integration & Bearer Token Authentication

What's New

GitHub Flow Registry Support

Native support for NiFi's GitHub Flow Registry Client, enabling CI/CD workflows for versioned NiFi flows:

• Git-specific versioning helpers: list_git_registry_buckets, get_git_registry_bucket, list_git_registry_flows, get_git_registry_flow, list_git_registry_flow_versions, deploy_git_registry_flow, update_git_flow_ver
• Registry client management: ensure_registry_client and update_registry_client for idempotent configuration
• Enhanced revert: revert_flow_ver now accepts wait=True for synchronous operation

Profile System Enhancements

• "env" profile: Configure nipyapi entirely from environment variables - no profiles file required

  nipyapi.profiles.switch('env') # Reads NIFI_API_ENDPOINT, NIFI_BEARER_TOKEN, etc.

  • Bearer token authentication: Simplest auth method - just URL + token
    export NIFI_API_ENDPOINT=https://nifi.example.com/nifi-api
    export NIFI_BEARER_TOKEN=eyJhbGciOiJSUzI1NiIs...

  Canvas Improvements

• Bulk controller service operations: schedule_all_controllers(pg_id, scheduled) for enabling/disabling all controller services

• Processor updates: update_processor now accepts name param and auto_stop for handling running processors

Bug Fixes

• Fixed test_create_controller leaving orphaned controller services
• revert_flow_ver now refreshes revision internally to prevent stale errors

Installation

pip install --upgrade nipyapi

Related Projects

• nipyapi-actions - GitHub Action for NiFi CI/CD workflows

Full Changelog: v1.0.1...v1.1.0

Infrastructure and documentation fixes for post-1.0 release

What's Changed

• Update for NiFi-2.6.0 by @Chaffelson in #391

Infrastructure Fixes

• ReadTheDocs configuration: Fixed .readthedocs.yaml to properly install package with docs extras, ensuring setuptools-scm is available for dynamic version detection. Stable docs now display correct version instead of 0.19.
• Pylint 3.x compatibility: Updated pylintrc with ignore-paths regex patterns to properly exclude generated code directories (nipyapi/nifi/, nipyapi/registry/, _version.py) in CI environments using pylint 3.x.
• CI workflow optimization: Reordered GitHub Actions steps to upload coverage reports before make test-dist runs, preventing make clean from deleting coverage.xml before codecov upload.

Documentation Updates

• Branch references: Updated all documentation links from NiFi2x development branch to main default branch (README badges, contribution guide links, Sphinx source code references).
• LICENSE: Added full Apache 2.0 license text to repository root.
• Development workflow: Added .cursor/ to .gitignore for IDE-specific files.

Full Changelog: v1.0.0...v1.0.1

v1.0.0 - Migrate to NiFI 2.x, drops support for NiFi 1.x on main

1.0.0 (2025-09-28)

---

Major migration to Apache NiFi/Registry 2.x (tested against 2.5.0). Drops 1.x support on main
IMPORTANT To continue working with NiFi-1.x you should pin your requirements to nipyapi<1.0.0

Breaking Changes - Action Required

• Function renaming: Upstream API specification changes result in operation IDs now using suffixed names (e.g., update_run_status1) and some other functions are also renamed
• Authentication and configuration overhaul: Significant changes to align with modern API standards and upstream API changes
• Users must review and update authentication patterns - legacy configuration methods will be different

Profile Management System

• Extensible file format (YAML/JSON) with environment variable overrides and sane defaults - familiar workflow like AWS CLI
• Intelligent authentication method detection: OIDC, mTLS, and Basic authentication based on available configuration parameters
• Built-in profiles for common deployment patterns: single-user, secure-ldap, secure-mtls, secure-oidc
• 15+ configurable parameters (URLs, credentials, certificates, SSL settings) with NIPYAPI_PROFILE environment variable
• Profile switching with nipyapi.profiles.switch() configures endpoints, authentication, and SSL settings in single function call

Automated Development Workflow

• Comprehensive Makefile targets for all key development and release processes
• End-to-end automation: entire client generation and testing sequence from test certificates to final integration tests
• GitHub Actions CI with full Docker NiFi integration tests and coverage reporting
• Smart certificate regeneration and optimized rebuild flows to avoid unnecessary infrastructure cycling

Quick Start and Migration Tools

• Sandbox Docker environment for testing different authentication mechanisms with make sandbox target
• FDLC example retained and modernized to demonstrate proper multi-environment workflows (single-user dev, secure-ldap prod)
• Comprehensive migration guide (docs/migration.rst) for upgrading from NiPyAPI 0.x/NiFi 1.x to 1.x/2.x

Core Technical Improvements

• Documentation system rebuild: Complete Sphinx overhaul with individual pages for all Core client, NiFi APIs and Registry APIs - with a flat API structure with optimal navigation
• Test coverage expansion: Comprehensive test suite with profile-driven automation rather than manual reconfiguration
• Pre-commit checks: Automated code quality with trailing whitespace, debug statements, flake8, and pylint hooks
• Modern dependency management: Migrated to python -m build, replaced ruamel.yaml with PyYAML, explicit urllib3/certifi/requests inclusion
• Enhanced documentation: Direct GitHub source code links with line-level precision, standardized docstrings throughout generated clients
• Profile-driven testing: Deprecated complex tox regression suite in favor of spec-driven single version testing
• Legacy pruning: Removed Python 2.7 and NiFi 1.x references, deprecated template-era dependencies (lxml, xmltodict)
• Codecov migration: Switched from Coveralls with pytest-cov integration and build process automation
• Enhanced convenience functions: Improved set_endpoint and various ensure_* object functions
• Certificate handling improvements: Resolved user complexity with automatic CA certificate setup and validation
• Extensive authentication documentation: OIDC setup instructions, Safari keychain guidance, development vs production practices
• API augmentation system: Client build-time patching for upstream API issues (currently: enum handling and missing security schemes)

Additional Details

• Deprecate Travis CI by @Chaffelson in #385
• Deprecate Python2 support by @Chaffelson in #386
• Fixed a camel-case typo in the name of the NifiRegistryFlowRegistryCl… by @asdaraujo in #387
• NiPyAPI 1.0 Release - Apache NiFi 2.x Compatibility by @Chaffelson in #388
• Fix Codecov action parameters in CI workflow by @Chaffelson in #390

New Contributors

• @asdaraujo made their first contribution in #387
  With thanks to @ottobackwards for release testing

Full Changelog: v0.22.0...v1.0.0

v0.22.0

What's Changed

• Updates to supported Python versions, handing Template deprecation in NiFi-2.x, and Windows development support by @Chaffelson in #381
• Add support for mTLS by @Chaffelson in #382
• Updated nifi and registry low level clients to v1.28.1 by @Chaffelson in #383

Full Changelog: v0.21.0...v0.22.0

v0.21.0

What's Changed

• Fix API model generator by @michaelarnauts in #356
• issue-360: handle -9 error messages better by @ottobackwards in #361
• Handle plain text response types so json values are correctly returned by @michaelarnauts in #358
• update clients to 1.27.0 by @Chaffelson in #365
• Simplified the use of setattr in recurse_flow, flatten, and list_all_by_kind methods in nipyapi/canvas.py.
• Added support for key_password in the Configuration class and its usage in nipyapi/nifi/rest.py and nipyapi/registry/rest.py.
• Fixed the method to retrieve HTTP headers in nipyapi/nifi/rest.py and nipyapi/registry/rest.py.
• Updated pylintrc to match more modern python standards
• Fixed issue #326 where the latest flow version was not being deployed by default
• Fixes nipyapi.nifi.ProcessGroupsApi.upload_process_group_with_http_info() incomplete #310
• VersionedReportingTask added with appropriate functions
• Move docker requirement to extras to avoid dependency install during standard usage
• Set latest python3 version to 3.12
• Added NiFi-2.0.0-M4 to testing suite, appears to work but compatibility not guaranteed using currently 1.x clients
• Deprecate usage of py.test in favour of newer pytest.
• Update readme

New Contributors

• @michaelarnauts made their first contribution in #356

Full Changelog: v0.20.0...v0.21.0

Update to NiFI 1.23.2

What's Changed

• NiFi 1.23.2 update by @Hexoplon in #330

• Update Python support to 3.7 through 3.11.

• Update get_access_policy_for_resource no policy test by @jrittenh in #317

• Update README with NiFi 1.17 support by @rsaggino in #314

• feat(config): add ReportingTaskEntity to registered filters by @Hexoplon in #337

• remove library setting root logger configuration by @ottobackwards in #340

• Fix ruamel.yaml pinning. Fixes docs and test process for EC2. by @Chaffelson in #346

New Contributors

• @Hexoplon made their first contribution in #330

Full Changelog: v0.19.1...v0.20.0

Update to NiFi 1.17

What's Changed since 0.17

• New github release notes feature, woop
• NiFi 1.17 support
• Added link to doc index for parameter contexts by @Jimvin in #302
• Allow version validation to ignore NiFi or Registry when checking for parameter support by @ChrisSamo632 in #304
• Adds child cascading to assign_context_to_process_group by @sdairs in #267
• Fix bug in security.bootstrap_security_policies by @PieterDM in #261

New Contributors since 0.17

• @Jimvin made their first contribution in #302
• @ChrisSamo632 made their first contribution in #304
• @sdairs made their first contribution in #267
• @PieterDM made their first contribution in #261

Update to NiFi-1.16

• modify utils.check_version to accept a default version to check against, and remove nested error when version check fails as it is overzealous
• Update test file paths in test_utils in case someone runs tests as root which actually can write to fake devices and therefore break the tests
• Updated clients for 1.16
• Added pylint exclusions for known but unimportant complaints
• Added workaround for NiFi 9470 by providing default inherited_parameter_contexts per advice from Chris Sampson in NiPyAPI Issue #305
• Fixes #305
• Updated NiFi and Registry versions in Docker compose configs to use latest 1.16.1 builds

NiFi and NiFi-Registry 1.15.0 support

• Moved testing to secured single user by default for NiFi
• Consolidated clients to the 1.15.0 release version for both NiFi and Registry
• Updated logic to support changed working modes in tests
• Corrected minor functional issues for 1.15 in login handling, SSL certs, Parameter Updates and Testing
• Backwards compatibility and Regression tested as requiring no breaking changes