feat(operator): implement per-node priority ordering

Implements per-node priority ordering where each node progresses through
skyhooks independently, preventing slow/blocked nodes from blocking
healthy nodes. Includes comprehensive e2e test documentation, bug fixes,
and validation.

Core Changes:
- Per-node priority ordering: nodes progress independently through priorities
- Nodes no longer wait for all other nodes to complete a priority level
- Added IsNodeReadyForSkyhook() to check per-node priority readiness
- Fixed CollectNodeStatus() to show in_progress if ANY node is progressing
- Updated waiting/blocked logic to be per-node, not global
- Changed runtime-required to be per node, not by skyhook.

Tests:
- Added critical assertions proving concurrent execution at different priorities
- Updated runtime-required test for per-node taint removal
- Added 7 unit tests for CollectNodeStatus per-node behavior

Documentation:
- Added comprehensive README.md for all chainsaw e2e tests
- Updated ordering_of_skyhooks.md with per-node behavior details
- Updated runtime_required.md for per-node taint removal
- Documented test scenarios, key features, and file descriptions

Infrastructure:
- Regenerated mocks for SkyhookNode with new methods
- Fixed license formatter to skip deleted files

This enables per-node parallelism while maintaining strict ordering
guarantees per node, preventing cluster-wide deadlocks from bad nodes.

Author: lockwobr

docs/ordering_of_skyhooks.md

@@ -1,12 +1,30 @@
 # Ordering of Skyhooks
 ## What
-With v0.8.0 Skyhooks now always get applied in a repeatable and specific order. This also means that all Skyhooks will now be sequential, though packages within a Skyhook can be parallel. Each custom resource now supports a `priority` field which is a non-zero positive integer. Skyhooks will be processed in order starting from 1, any Skyhooks with the same `priority` will be processed by sorting them by their `metadata.name` field.
+Skyhooks are applied in a repeatable and specific order based on their `priority` field. Each custom resource supports a `priority` field which is a non-zero positive integer. Skyhooks will be processed in order starting from 1, any Skyhooks with the same `priority` will be processed by sorting them by their `metadata.name` field.
 
 **NOTE**: Any Skyhook which does NOT provide a `priority` field will be assigned a priority value of 200.
 
-Two additional flow control features have been added with this and can be set in the annotations of each skyhook:
+## Per-Node Ordering
+
+**Important**: Priority ordering is enforced **per-node**, not globally across all nodes. This means:
+- Node A can proceed to Skyhook 2 as soon as Skyhook 1 completes on Node A
+- Node A does NOT wait for Node B to complete Skyhook 1
+- If Node B is stuck on Skyhook 1, Node A can still progress through all its skyhooks
+
+This per-node behavior prevents deadlocks where a few stuck/bad nodes would block all other healthy nodes from progressing through their skyhook sequence.
+
+### Example
+With two nodes (A, B) and two skyhooks (priority 1 and priority 2):
+- Node A completes Skyhook 1 → Node A immediately starts Skyhook 2
+- Node B is still processing Skyhook 1 → Node B shows "waiting" status on Skyhook 2
+- Node A completes Skyhook 2 → Node A is fully complete
+- Node B eventually completes Skyhook 1 → Node B starts Skyhook 2
+
+## Flow Control Annotations
+
+Two flow control features can be set in the annotations of each skyhook:
  * `skyhook.nvidia.com/disable`: bool. When `true` it will skip this Skyhook from processing and continue with any other ones further down the priority order.
- * `skyhook.nvidia.com/pause`: bool. When `true` it will NOT process this Skyhook and it WILL NOT continue to process any Skyhook's after this one. This will effectively stop all application of Skyhooks starting with this one. NOTE: This ability used to be on the Skyhook spec itself as the `pause` field and has been moved here to be consistent with `disable` and to avoid incrementing the generation of a Skyhook Custom Resource instance when changing it.
+ * `skyhook.nvidia.com/pause`: bool. When `true` it will NOT process this Skyhook and it WILL NOT continue to process any Skyhook's after this one on that node. This will effectively stop all application of Skyhooks starting with this one. NOTE: This ability used to be on the Skyhook spec itself as the `pause` field and has been moved here to be consistent with `disable` and to avoid incrementing the generation of a Skyhook Custom Resource instance when changing it.
 
 ## Why
 This solves a few problems:

docs/runtime_required.md

@@ -18,12 +18,18 @@ before the nodes that it targets are considered available for general use.
 1. It will NOT add the taint to any nodes targeted by a SCR with `runtimeRequired: true`
 
 # Details
-## When is a node considered ready
-When all of the following is true per node:
-1. All SCRs with `runtimeRequired: true` are complete (ie complete on all nodes)
+## When is the runtime-required taint removed from a node
+The taint is removed from a node when all SCRs with `runtimeRequired: true` that target that node are complete **on that specific node**.
 
-## What happens happens when a node is considered ready
-1. The runtime required taint is removed from the node if it exists.
+**Important**: Taint removal is per-node, not per-skyhook. This means:
+- Node A's taint is removed when all runtime-required skyhooks complete on Node A
+- Node A does NOT wait for Node B to complete those same skyhooks
+- If Node B is stuck or failing, Node A can still have its taint removed and become available
+
+This per-node behavior prevents deadlocks where a few bad nodes would block all other healthy nodes from becoming available.
+
+## What happens when the taint is removed
+1. The node becomes available for general workload scheduling (pods without the runtime-required toleration can now be scheduled on it).
 
 # Why would you use runtime required
 This is useful when you want to gate other work behind the successful completion of some set of Skyhook Packages. This can be for security reasons or for scheduling.

k8s-tests/chainsaw/cli/lifecycle/README.md

@@ -0,0 +1,49 @@
+# CLI Lifecycle Test
+
+## Purpose
+
+Validates all CLI lifecycle commands for controlling Skyhook processing state.
+
+## Test Scenario
+
+1. Reset state from previous runs
+2. Create a skyhook and wait for it to start processing
+3. Test pause command:
+   - Run `skyhook pause <name>`
+   - Assert the skyhook shows paused status
+4. Test resume command:
+   - Run `skyhook resume <name>`
+   - Assert the skyhook resumes processing
+5. Test disable command:
+   - Run `skyhook disable <name>`
+   - Assert the skyhook is disabled
+6. Test enable command:
+   - Run `skyhook enable <name>`
+   - Assert the skyhook is enabled and processing
+7. Test pause and disable together:
+   - Apply both pause and disable to the skyhook
+   - Assert both flags are set
+   - Run `skyhook resume` - should only remove pause, not disable
+   - Assert the skyhook is still disabled
+   - Run `skyhook enable` - should remove disable
+   - Assert the skyhook is fully enabled
+
+## Key Features Tested
+
+- `skyhook pause` - Pauses a Skyhook from processing
+- `skyhook resume` - Resumes a paused Skyhook
+- `skyhook disable` - Disables a Skyhook completely
+- `skyhook enable` - Enables a disabled Skyhook
+- Independence of pause and disable flags
+- Resume only affects pause, not disable
+
+## Files
+
+- `chainsaw-test.yaml` - Main test configuration
+- `skyhook.yaml` - Test skyhook
+- `assert-paused.yaml` - Paused state assertion
+- `assert-resumed.yaml` - Resumed state assertion
+- `assert-disabled.yaml` - Disabled state assertion
+- `assert-enabled.yaml` - Enabled state assertion
+- `assert-paused-and-disabled.yaml` - Both flags set assertion
+- `assert-still-disabled.yaml` - Resume doesn't remove disable assertion

k8s-tests/chainsaw/cli/lifecycle/chainsaw-test.yaml

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 #
@@ -20,12 +20,6 @@ kind: Test
 metadata:
   name: cli-lifecycle
 spec:
-  description: |
-    Test all CLI lifecycle commands:
-    - pause: pauses a Skyhook from processing
-    - resume: resumes a paused Skyhook
-    - disable: disables a Skyhook completely
-    - enable: enables a disabled Skyhook
   timeouts:
     assert: 60s
     exec: 30s
@@ -127,4 +121,3 @@ spec:
     - script:
         content: |
           kubectl delete skyhook cli-lifecycle-test 2>/dev/null || true
-

k8s-tests/chainsaw/cli/node/README.md

@@ -0,0 +1,41 @@
+# CLI Node Test
+
+## Purpose
+
+Validates all CLI node commands for managing node state within a Skyhook.
+
+## Test Scenario
+
+1. Reset state from previous runs
+2. Create a skyhook and wait for it to complete
+3. Test node list command:
+   - Run `skyhook node list <skyhook>`
+   - Verify output shows targeted nodes
+4. Test node status command:
+   - Run `skyhook node status [node]`
+   - Verify output shows Skyhook activity on nodes
+5. Test node ignore command:
+   - Run `skyhook node ignore <skyhook> <node>`
+   - Assert the node is excluded from processing
+6. Test node unignore command:
+   - Run `skyhook node unignore <skyhook> <node>`
+   - Assert the node is included back in processing
+7. Test node reset command:
+   - Run `skyhook node reset <skyhook> <node>`
+   - Assert the package state is reset on the node
+
+## Key Features Tested
+
+- `skyhook node list` - Shows nodes targeted by a Skyhook
+- `skyhook node status` - Shows Skyhook activity on nodes
+- `skyhook node ignore` - Excludes a node from processing
+- `skyhook node unignore` - Includes a node back in processing
+- `skyhook node reset` - Resets package state on a node
+
+## Files
+
+- `chainsaw-test.yaml` - Main test configuration
+- `skyhook.yaml` - Test skyhook
+- `assert-node-ignored.yaml` - Ignored state assertion
+- `assert-node-unignored.yaml` - Unignored state assertion
+- `assert-node-reset.yaml` - Reset state assertion

k8s-tests/chainsaw/cli/node/chainsaw-test.yaml

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 #
@@ -20,12 +20,6 @@ kind: Test
 metadata:
   name: cli-node
 spec:
-  description: |
-    Test all CLI node commands:
-    - node list: shows nodes targeted by a Skyhook
-    - node status: shows Skyhook activity on nodes  
-    - node ignore/unignore: excludes/includes nodes from processing
-    - node reset: resets package state on a node
   timeouts:
     assert: 120s
     exec: 90s
@@ -116,4 +110,3 @@ spec:
           for NODE in $(kubectl get nodes -l skyhook.nvidia.com/test-node=skyhooke2e -o jsonpath='{.items[*].metadata.name}'); do
             kubectl label node "$NODE" skyhook.nvidia.com/ignore- 2>/dev/null || true
           done
-

k8s-tests/chainsaw/cli/package/README.md

@@ -0,0 +1,32 @@
+# CLI Package Test
+
+## Purpose
+
+Validates all CLI package commands for inspecting and managing package state.
+
+## Test Scenario
+
+1. Reset state from previous runs
+2. Create a skyhook and wait for it to complete
+3. Test package status command:
+   - Run `skyhook package status <skyhook> <package>`
+   - Verify output shows package status across nodes
+4. Test package logs command:
+   - Run `skyhook package logs <skyhook> <package>`
+   - Verify logs are retrieved from package pods
+5. Test package rerun command:
+   - Run `skyhook package rerun <skyhook> <package>`
+   - Assert the package is re-run on the node
+
+## Key Features Tested
+
+- `skyhook package status` - Shows package status across nodes
+- `skyhook package logs` - Retrieves logs from package pods
+- `skyhook package rerun` - Forces a package to re-run
+
+## Files
+
+- `chainsaw-test.yaml` - Main test configuration
+- `skyhook.yaml` - Test skyhook
+- `assert-complete.yaml` - Initial completion assertion
+- `assert-package-rerun.yaml` - Rerun state assertion

k8s-tests/chainsaw/cli/package/chainsaw-test.yaml

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 #
@@ -20,11 +20,6 @@ kind: Test
 metadata:
   name: cli-package
 spec:
-  description: |
-    Test all CLI package commands:
-    - package status: shows package status across nodes
-    - package logs: retrieves logs from package pods
-    - package rerun: forces a package to re-run
   timeouts:
     assert: 120s
     exec: 90s
@@ -96,4 +91,3 @@ spec:
     - script:
         content: |
           ../skyhook-cli reset cli-package-test --confirm 2>/dev/null || true
-

k8s-tests/chainsaw/cli/readme.md

@@ -0,0 +1,53 @@
+# CLI Tests
+
+This directory contains end-to-end tests for the `kubectl-skyhook` CLI plugin. These tests validate that all CLI commands work correctly against a real Kubernetes cluster.
+
+## Prerequisites
+
+The CLI tests require:
+1. A running Kind cluster with the skyhook operator installed
+2. Nodes labeled with `skyhook.nvidia.com/test-node=skyhooke2e`
+3. The `skyhook` CLI binary built with coverage enabled
+
+## Tests
+
+| Test | Description |
+|------|-------------|
+| [lifecycle](./lifecycle/) | Pause, resume, disable, and enable commands |
+| [node](./node/) | Node list, status, ignore, unignore, and reset commands |
+| [package](./package/) | Package status, logs, and rerun commands |
+| [reset](./reset/) | Skyhook reset command |
+
+## CLI Commands Tested
+
+### Lifecycle Commands
+- `skyhook pause <skyhook>` - Pauses a Skyhook from processing
+- `skyhook resume <skyhook>` - Resumes a paused Skyhook
+- `skyhook disable <skyhook>` - Disables a Skyhook completely
+- `skyhook enable <skyhook>` - Enables a disabled Skyhook
+
+### Node Commands
+- `skyhook node list <skyhook>` - Shows nodes targeted by a Skyhook
+- `skyhook node status [node]` - Shows Skyhook activity on nodes
+- `skyhook node ignore <skyhook> <node>` - Excludes a node from processing
+- `skyhook node unignore <skyhook> <node>` - Includes a node back in processing
+- `skyhook node reset <skyhook> <node>` - Resets package state on a node
+
+### Package Commands
+- `skyhook package status <skyhook> <package>` - Shows package status across nodes
+- `skyhook package logs <skyhook> <package>` - Retrieves logs from package pods
+- `skyhook package rerun <skyhook> <package>` - Forces a package to re-run
+
+### Reset Command
+- `skyhook reset <skyhook>` - Resets all nodes for a Skyhook
+
+## Running the Tests
+
+```bash
+# Run all CLI tests
+make cli-e2e-tests
+
+# Run a specific test
+cd k8s-tests/chainsaw/cli
+chainsaw test --test-dir lifecycle
+```

k8s-tests/chainsaw/cli/reset/README.md

@@ -0,0 +1,28 @@
+# CLI Reset Test
+
+## Purpose
+
+Validates the skyhook reset command for resetting all nodes in a Skyhook.
+
+## Test Scenario
+
+1. Create a skyhook and wait for it to complete
+2. Disable the skyhook to prevent re-processing after reset
+3. Test reset command:
+   - Run `skyhook reset <name>`
+   - Assert all nodes are reset to initial state
+4. Verify node annotations are cleared
+
+## Key Features Tested
+
+- `skyhook reset` - Resets all nodes for a Skyhook
+- Node state cleanup
+- Annotation removal
+
+## Files
+
+- `chainsaw-test.yaml` - Main test configuration
+- `skyhook.yaml` - Test skyhook
+- `assert-skyhook-complete.yaml` - Initial completion assertion
+- `assert-skyhook-disabled.yaml` - Disabled state assertion
+- `assert-nodes-reset.yaml` - Reset state assertion