Feature/20260115 improvements (#402)

* fix(parameters): correct parameter value removal and add rename_parameter_context

- Fix prepare_parameter to properly unset parameter values using value_removed flag
- Previously, passing None or omitting value could result in empty string or null
  instead of actually removing the parameter value
- Add _NOT_PROVIDED sentinel to distinguish omitted value from explicit None
- prepare_parameter now correctly handles: omit (preserve existing), None (remove),
  empty string (set to ""), or value (set)
- Add rename_parameter_context() for lightweight context renaming
- Use config.long_max_wait for update_parameter_context timeout
- Include description in get_parameter_ownership_map output
- Add comprehensive tests for new functionality

* feat(bulletins): add descendants parameter to get_bulletin_board

- Add descendants=True parameter to filter bulletins by process group hierarchy
- When True (default), includes bulletins from all child process groups
- When False, only returns bulletins from components directly in the specified PG
- Builds regex pattern of all PG IDs for efficient API filtering

* feat(canvas): add name lookup support to list_all_controllers

- Add greedy and identifier_type parameters for flexible PG resolution
- Supports UUID, name, or ProcessGroupEntity as pg_id input
- Uses resolve_entity for consistent behavior with other canvas functions

* feat(ci): enhance get_status with throughput stats and descendant bulletins

- Add flowfiles_in/out, bytes_in/out, bytes_read/written stats
- Use descendants=True for bulletin collection to match processor/controller scope
- Update docstring to document bulletin counts

* perf(ci): batch parameter updates to reduce cycle overhead

- Batch all parameters per context into single update_parameter_context call
- Avoids multiple stop/disable/update/enable/restart cycles per parameter
- Preserve parameter descriptions during updates
- Add tests for empty string and None value handling

* docs(layout): add PORT_QUEUE_BOX_WIDTH constant

- Document that port-to-port connections use larger queue boxes (240 vs 224)
- Clarify existing QUEUE_BOX_WIDTH is for processor-to-processor connections

* feat(cli): standardize error field convention for non-zero exit codes

- CLI now exits with code 1 when result dict contains 'error' or 'errors' key
- Update cleanup.py: change 'message' to 'error' for failure cases
- Update verify_config.py: add 'error' key with failed component names
- Add tests for error key detection and exit code behavior
- Document error field convention in cli.rst and ci.rst for CI authors

This enables scripts to rely on exit codes for operational failures,
not just Python exceptions. CI functions should include an 'error' key
when operations fail to trigger non-zero exit.

* docs: add 1.4.0 release notes to history

Author: Chaffelson

docs/ci.rst

@@ -15,6 +15,39 @@ CI operations are designed for automation scenarios:
 - **Sensible defaults**: Minimal configuration required for common use cases
 - **Structured output**: Returns plain dicts suitable for CI artifact formats
 - **Error handling**: Raises exceptions with clear error messages
+- **Exit code support**: Operations that fail include an ``error`` key, causing the CLI to exit with code 1
+
+Error Field Convention
+----------------------
+
+CI functions follow a standard convention for indicating operational failures:
+
+- **Success**: Return a dict without ``error`` or ``errors`` keys
+- **Failure**: Include an ``error`` (string) or ``errors`` (string) key with a description
+
+This enables scripts to rely on exit codes:
+
+.. code-block:: bash
+
+    # Exit code will be 1 if verification fails
+    if nipyapi ci verify_config --process_group_id "$PG_ID"; then
+        nipyapi ci start_flow --process_group_id "$PG_ID"
+    else
+        echo "Verification failed"
+        exit 1
+    fi
+
+When writing custom CI functions, follow this convention:
+
+.. code-block:: python
+
+    def my_ci_operation(...) -> dict:
+        # On success - no error key
+        if success:
+            return {"status": "complete", "count": 5}
+
+        # On failure - include error key
+        return {"status": "failed", "error": "Operation failed: reason"}
 
 Quick Start
 ===========

docs/cli.rst

@@ -341,7 +341,9 @@ Usage Examples
 Error Handling
 ==============
 
-The CLI returns structured error responses on failure:
+The CLI returns structured error responses and uses non-zero exit codes for failures.
+
+**Exception errors** (Python exceptions during execution):
 
 .. code-block:: json
 
@@ -353,10 +355,44 @@ The CLI returns structured error responses on failure:
       "logs": ["nipyapi.canvas: Fetching process group..."]
     }
 
+**Operational errors** (command completed but operation failed):
+
+CI functions include an ``error`` or ``errors`` field when an operation fails.
+The CLI detects these fields and exits with code 1.
+
+.. code-block:: json
+
+    {
+      "verified": "false",
+      "failed_count": 2,
+      "error": "Verification failed for: DBCPConnectionPool, ExecuteSQL"
+    }
+
 Exit codes:
 
-- ``0``: Success
-- ``1``: Error (check output for details)
+- ``0``: Success (no ``error``/``errors`` key in result)
+- ``1``: Error (exception raised, or result contains ``error``/``errors`` key)
+
+Error Field Convention
+----------------------
+
+When writing custom CI functions or extending nipyapi, follow this convention:
+
+- Include an ``error`` key (string) when an operation fails
+- Use ``errors`` key (string) for multiple error messages (pipe-separated)
+- Do NOT include ``error``/``errors`` keys on success
+
+This ensures the CLI exits with the correct code for scripting:
+
+.. code-block:: bash
+
+    # Script can rely on exit code
+    if nipyapi ci verify_config --process_group_id "$PG_ID"; then
+        nipyapi ci start_flow --process_group_id "$PG_ID"
+    else
+        echo "Verification failed, not starting flow"
+        exit 1
+    fi
 
 Cross-References
 ================

docs/history.rst

@@ -2,6 +2,51 @@
 History
 =======
 
+1.4.0 (2026-01-15)
+-------------------
+
+| CLI exit codes, bulletins descendants, batch parameter updates, and throughput stats
+
+**CLI Improvements**
+
+- **Standardized error exit codes**: CLI now exits with code 1 when result contains ``error`` or ``errors`` key, enabling scripts to reliably detect operational failures (not just exceptions)
+- **Error field convention**: CI functions now include ``error`` key for failures - documented convention for custom CI functions
+
+**Bulletins Module**
+
+- **get_bulletin_board()**: New ``descendants=True`` parameter to include bulletins from child process groups (default behavior matches processor/controller scope)
+
+**Canvas Module**
+
+- **list_all_controllers()**: Added ``greedy`` and ``identifier_type`` parameters for flexible process group resolution - now accepts ID, name, or ProcessGroupEntity
+
+**CI Module Enhancements**
+
+- **get_status()**: Added throughput stats (``flowfiles_in/out``, ``bytes_in/out``, ``bytes_read/written``) and fixed bulletin collection to use descendants
+- **configure_inherited_params()**: Batched parameter updates to reduce cycle overhead - all parameters per context now updated in single API call instead of one-by-one
+- **cleanup()**: Changed ``message`` key to ``error`` for failure responses (CLI exit code support)
+- **verify_config()**: Added ``error`` key with failed component names when verification fails
+
+**Parameters Module**
+
+- **prepare_parameter()**: Clarified value semantics - omitting ``value`` preserves existing, ``value=None`` explicitly unsets, ``value=""`` sets empty string
+- **list_orphaned_contexts()**: New function to find parameter contexts not bound to any process groups
+- **rename_parameter_context()**: New function to rename a parameter context
+
+**Layout Module**
+
+- **PORT_QUEUE_BOX_WIDTH**: New constant (240px) documenting port-to-port connection queue box width
+
+**Bug Fixes**
+
+- Fixed parameter value removal to correctly handle ``value=None`` vs omitted value
+- Fixed bulletin board to include controller service bulletins via group_id regex pattern
+
+**Documentation**
+
+- CLI error handling guide with exit code convention
+- CI error field convention for function authors
+
 1.3.0 (2026-01-08)
 -------------------
 

nipyapi/bulletins.py

@@ -67,7 +67,7 @@ def get_bulletins():
         return nipyapi.nifi.FlowApi().get_bulletins()
 
 
-def get_bulletin_board(pg_id=None, source_name=None, message=None, limit=None):
+def get_bulletin_board(pg_id=None, source_name=None, message=None, limit=None, descendants=True):
     """
     Retrieve bulletins from the bulletin board with optional filtering.
 
@@ -80,6 +80,9 @@ def get_bulletin_board(pg_id=None, source_name=None, message=None, limit=None):
         source_name (str, optional): Filter by source component name (regex).
         message (str, optional): Filter by message content (regex).
         limit (int, optional): Maximum number of bulletins to return.
+        descendants (bool): Include bulletins from child process groups (default True).
+            Only applies when pg_id is specified. If False, only returns bulletins
+            from components directly in the specified process group.
 
     Returns:
         list[BulletinDTO]: List of bulletin objects with direct field access.
@@ -101,15 +104,27 @@ def get_bulletin_board(pg_id=None, source_name=None, message=None, limit=None):
 
     Example::
 
+        >>> # Get bulletins from a PG and all its children (default)
         >>> bulletins = nipyapi.bulletins.get_bulletin_board(pg_id="abc-123")
         >>> for b in bulletins:
         ...     print(f"{b.source_name}: {b.message}")
-        ...     if b.stack_trace:
-        ...         print(f"  Stack: {b.stack_trace[:100]}...")
+
+        >>> # Get bulletins only from components directly in the PG
+        >>> bulletins = nipyapi.bulletins.get_bulletin_board(pg_id="abc-123", descendants=False)
     """
     kwargs = {}
+
+    # Build group_id filter - use regex to include descendants if requested
     if pg_id is not None:
-        kwargs["group_id"] = pg_id
+        if descendants:
+            # Get all child PG IDs recursively
+            all_pgs = nipyapi.canvas.list_all_process_groups(pg_id=pg_id)
+            pg_ids = [pg_id] + [pg.id for pg in all_pgs]
+            # Build regex pattern: "id1|id2|id3..."
+            kwargs["group_id"] = "|".join(pg_ids)
+        else:
+            kwargs["group_id"] = pg_id
+
     if source_name is not None:
         kwargs["source_name"] = source_name
     if message is not None:

nipyapi/canvas.py

@@ -1929,22 +1929,43 @@ def create_controller(parent_pg, controller, name=None):
         )
 
 
-def list_all_controllers(pg_id="root", descendants=True, include_reporting_tasks=False):
+def list_all_controllers(
+    pg_id="root",
+    descendants=True,
+    include_reporting_tasks=False,
+    greedy=True,
+    identifier_type="auto",
+):
     """
     Lists all controllers under a given Process Group, defaults to Root.
     Optionally recurses all child Process Groups as well.
 
     Args:
-        pg_id (str): String of the ID of the Process Group to list from
+        pg_id (str): The Process Group to list from, as a UUID string,
+            process group name, or ProcessGroupEntity object. Defaults to root.
         descendants (bool): True to recurse child PGs, False to not
         include_reporting_tasks (bool): True to include Reporting Tasks, False to not
+        greedy (bool): For name lookup, True for partial match, False for exact.
+        identifier_type (str): How to interpret string identifier:
+            "auto" (default) detects UUID vs name, "id" or "name" to force.
 
     Returns:
         None, ControllerServiceEntity, or list(ControllerServiceEntity)
 
     """
-    assert isinstance(pg_id, str)
     assert isinstance(descendants, bool)
+    assert pg_id == "root" or isinstance(pg_id, (str, nipyapi.nifi.ProcessGroupEntity))
+    # Resolve pg_id to actual ID (supports name lookup)
+    if pg_id != "root":
+        process_group = nipyapi.utils.resolve_entity(
+            pg_id,
+            get_process_group,
+            nipyapi.nifi.ProcessGroupEntity,
+            strict=True,
+            greedy=greedy,
+            identifier_type=identifier_type,
+        )
+        pg_id = process_group.id
     handle = nipyapi.nifi.FlowApi()
     # Testing shows that descendant doesn't work on NiFi-1.1.2
     # Or 1.2.0, despite the descendants option being available
@@ -2348,15 +2369,19 @@ def get_controller_service_docs(controller):
         )
 
 
-def list_all_by_kind(kind, pg_id="root", descendants=True):
+def list_all_by_kind(kind, pg_id="root", descendants=True, greedy=True, identifier_type="auto"):
     """
     Retrieves a list of all instances of a supported object type
 
     Args:
         kind (str):  one of input_ports, output_ports, funnels, controllers,
             connections, remote_process_groups
-        pg_id (str): optional, ID of the Process Group to use as search base
+        pg_id: The Process Group to list from, as a UUID string,
+            process group name, or ProcessGroupEntity object. Defaults to root.
         descendants (bool): optional, whether to collect child group info
+        greedy (bool): For name lookup, True for partial match, False for exact.
+        identifier_type (str): How to interpret string identifier:
+            "auto" (default) detects UUID vs name, "id" or "name" to force.
 
     Returns:
         list of the Entity type of the kind, or single instance, or None
@@ -2371,7 +2396,20 @@ def list_all_by_kind(kind, pg_id="root", descendants=True):
         "remote_process_groups",
     ]
     if kind == "controllers":
-        return list_all_controllers(pg_id, descendants)
+        return list_all_controllers(
+            pg_id, descendants, greedy=greedy, identifier_type=identifier_type
+        )
+    # Resolve pg_id to actual ID (supports name lookup)
+    if pg_id != "root":
+        process_group = nipyapi.utils.resolve_entity(
+            pg_id,
+            get_process_group,
+            nipyapi.nifi.ProcessGroupEntity,
+            strict=True,
+            greedy=greedy,
+            identifier_type=identifier_type,
+        )
+        pg_id = process_group.id
     handle = nipyapi.nifi.ProcessGroupsApi()
     call_function = getattr(handle, "get_" + kind)
     out = []

nipyapi/ci/cleanup.py

@@ -114,7 +114,7 @@ def cleanup(
                 "stopped": "false",
                 "deleted": "false",
                 "process_group_name": "",
-                "message": "Process group not found",
+                "error": "Process group not found",
             }
         raise
 
@@ -123,7 +123,7 @@ def cleanup(
             "stopped": "false",
             "deleted": "false",
             "process_group_name": "",
-            "message": "Process group not found",
+            "error": "Process group not found",
         }
 
     pg_name = process_group.component.name

nipyapi/ci/configure_inherited_params.py

@@ -152,7 +152,7 @@ def configure_inherited_params(
                 )
                 if ctx_id not in updates_by_context:
                     updates_by_context[ctx_id] = []
-                updates_by_context[ctx_id].append((param_name, param_value, False))
+                updates_by_context[ctx_id].append((param_name, param_value, False, None))
             else:
                 errors.append(
                     f"Parameter '{param_name}' not found in any context. "
@@ -183,20 +183,38 @@ def configure_inherited_params(
         if owner["context_id"] not in updates_by_context:
             updates_by_context[owner["context_id"]] = []
         updates_by_context[owner["context_id"]].append(
-            (param_name, param_value, owner["sensitive"])
+            (param_name, param_value, owner["sensitive"], owner["description"])
         )
 
-    # Execute if not dry run
+    # Execute if not dry run - batch all parameters per context to avoid
+    # multiple disable/enable cycles. Each update_parameter_context call
+    # triggers a full cycle (stop processors -> disable controllers -> update
+    # -> re-enable -> restart), so we must batch all params for a context.
     if not dry_run and not errors:
         for target_ctx_id, param_updates in updates_by_context.items():
-            for param_name, param_value, _sensitive in param_updates:
-                log.info("Updating %s in context %s", param_name, target_ctx_id)
-                nipyapi.parameters.update_parameter_in_context(
-                    context_id=target_ctx_id,
-                    param_name=param_name,
-                    value=param_value,
-                    create_if_missing=allow_override,
+            log.info(
+                "Updating %d parameter(s) in context %s",
+                len(param_updates),
+                target_ctx_id,
+            )
+
+            # Get current context for revision info
+            ctx = nipyapi.parameters.get_parameter_context(target_ctx_id, identifier_type="id")
+            if ctx is None:
+                errors.append(f"Parameter context not found: {target_ctx_id}")
+                continue
+
+            # Build parameter entities from planning data
+            param_entities = [
+                nipyapi.parameters.prepare_parameter(
+                    name=name, value=value, description=desc, sensitive=sens
                 )
+                for name, value, sens, desc in param_updates
+            ]
+
+            # Update all parameters in one batch
+            ctx.component.parameters = param_entities
+            nipyapi.parameters.update_parameter_context(ctx)
 
     # Build result
     result = {