Skip to content

Conversation

@ramonfigueiredo
Copy link
Collaborator

@ramonfigueiredo ramonfigueiredo commented Sep 30, 2025

Link the Issue(s) this Pull Request is related to.

Summarize your change.

Implement comprehensive job management functionality and tests for cueadmin to provide better control over job operations from the command line.

Changes:

  1. Job Management Commands (cueadmin/cueadmin/common.py):
  • Add Job Options argument group to parser
  • Implement -pause/-unpause for job state control
  • Implement -kill/-kill-all for job termination with confirmation
  • Implement -retry/-retry-all for frame retry operations
  • Implement -drop-depends for dependency management
  • Implement -set-min-cores/-set-max-cores for resource control
  • Implement -priority for job priority adjustments
  • All destructive operations require confirmation unless -force flag is used
  1. Unit Tests (cueadmin/tests/test_job_commands.py):
  • 42 comprehensive tests covering all job commands
  • Tests for -lj/-lji job listing and info display
  • Tests for job state changes (pause/unpause)
  • Tests for job termination with confirmation workflows
  • Tests for frame retry operations
  • Tests for dependency removal
  • Tests for resource modifications
  • Tests for priority adjustments
  • Error handling tests for invalid job names and values
  • Tests for various job states (running, finished, failed, paused)
  1. Add integration tests for job management workflows

Extend JobManagementWorkflowTest class with comprehensive workflow tests for job management commands to verify end-to-end operation sequences.

New integration tests added:

  • Job listing and filtering (-lj, -lji with filters)
  • Priority adjustment workflow with confirmation
  • Retry workflow for dead frames across multiple jobs
  • Kill-all jobs workflow with confirmation
  • Dependency dropping for multiple jobs
  • Resource modification (min/max cores) workflow
  • Complete state transition (pause - > unpause - > kill)

These integration tests complement the unit tests in test_job_commands.py by verifying that complex command sequences maintain state consistency and work correctly together.

  1. Expand job management documentation and tutorials
  • Updated cueadmin/README.md:

    • Added job management commands to usage section
    • Expanded test suite description
    • Added job command tests to test types
  • Updated docs/_docs/reference/commands/cueadmin.md:

    • Added complete "Job Options" section with 10 job commands (-pause, -unpause, -kill, -kill-all, -retry, -retry-all, -drop-depends, -set-min-cores, -set-max-cores, -priority)
  • Updated docs/_docs/reference/tools/cueadmin.md:

    • Mentioned job management in overview and key features
    • Added "Job Operations" section
    • Added new workflow sections: Managing Problem Jobs, Cleaning Up Old Jobs
    • Updated test suite info
  • Updated docs/_docs/tutorials/cueadmin-tutorial.md:

    • Added new section: Job Management
    • Renumbered subsequent parts
    • Updated Summary and Development sections

All documentation now fully covers job management with commands, workflows, practical examples, safety notes, and best practices.

Create comprehensive integration test suite in /cueadmin/tests/integration_tests.py covering critical workflows:

- Show, allocation, and subscription creation workflow
- Job lifecycle management (pause, modify, unpause, kill)
- Host management including lock, move allocation, unlock operations
- Dependency creation and satisfaction (job, layer, frame dependencies)
- Show cleanup workflow (disable, kill jobs, delete)
- Resource reallocation during active rendering
- Error recovery scenarios for failed operations
- Permission and authorization workflows

Tests use mocked OpenCue backend to verify command execution flow and state consistency across operations without requiring live infrastructure.
Implement comprehensive job management functionality and tests for cueadmin to provide better control over job operations from the command line.

Changes:

1. Job Management Commands (cueadmin/cueadmin/common.py):
- Add Job Options argument group to parser
- Implement -pause/-unpause for job state control
- Implement -kill/-kill-all for job termination with confirmation
- Implement -retry/-retry-all for frame retry operations
- Implement -drop-depends for dependency management
- Implement -set-min-cores/-set-max-cores for resource control
- Implement -priority for job priority adjustments
- All destructive operations require confirmation unless -force flag is used

2. Unit Tests (cueadmin/tests/test_job_commands.py):
- 42 comprehensive tests covering all job commands
- Tests for -lj/-lji job listing and info display
- Tests for job state changes (pause/unpause)
- Tests for job termination with confirmation workflows
- Tests for frame retry operations
- Tests for dependency removal
- Tests for resource modifications
- Tests for priority adjustments
- Error handling tests for invalid job names and values
- Tests for various job states (running, finished, failed, paused)

All tests passing with clean pylint validation.
@ramonfigueiredo ramonfigueiredo self-assigned this Sep 30, 2025
@ramonfigueiredo
Copy link
Collaborator Author

Job Operations in CueAdmin vs Cueman

I want to address the potential concern about overlap with cueman functionality. After analyzing both tools' architecture and purposes, I conclude that job operations should be in cueadmin as well. Here's why:

Tool architecture

  • cueadmin (per ../cueadmin/README.md):

    • "command-line administration tool"
    • "provides full control over jobs, layers, frames, and hosts"
    • Primary administrative CLI - designed to be self-sufficient
  • cueman (per ../cueman/README.md):

    • "extends the functionality of cueadmin" (builds on top)
    • "user-friendly interface for managing jobs"
    • Adds advanced features: filtering, frame manipulation, stagger/reorder

Why both tools need job operations

1. Different tool purposes

  • cueadmin: Core administrative CLI with fundamental operations
  • cueman: Enhanced user-focused tool with advanced UX and filtering capabilities

2. Overlap is Intentional, not duplication

The functionality serves different use cases:

Feature cueadmin (this PR) cueman
pause/unpause Basic operation + batch wildcards + comma-separated lists
kill Job-level with confirmation Frame-level with state/memory/duration filtering
retry Dead frames for jobs Frames with state filtering and layer targeting
Frame operations X (eat, done, stagger, reorder)
Advanced filtering X (state, memory, duration, layer, range)
  • cueadmin provides direct, straightforward operations for administrators.
  • cueman provides filtered, batch-capable operations for power users.

3. Tool Independence

  • Users may install only cueadmin without cueman
  • cueadmin should be fully functional as a standalone administrative tool
  • Core job operations (pause, kill, retry, priority) are fundamental admin tasks

Conclusion

This implementation:

The overlap is by design: cueadmin provides the foundation, cueman provides the enhancement.

@ramonfigueiredo
Copy link
Collaborator Author

python3 -m pytest cueadmin/tests/test_job_commands.py

=============================================================================================================================================== test session starts ===============================================================================================================================================
cachedir: .pytest_cache
configfile: pyproject.toml
plugins: mock-3.15.1, pyfakefs-5.2.3, cov-7.0.0
collected 42 items                                                                                                                                                                                                                                                                                                

cueadmin/tests/test_job_commands.py::JobCommandTests::test_drop_depends_multiple_jobs PASSED                                                                                                                                                                                                                [  2%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_drop_depends_single_job PASSED                                                                                                                                                                                                                   [  4%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_drop_depends_with_force_flag PASSED                                                                                                                                                                                                              [  7%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_job_not_found_error_handling PASSED                                                                                                                                                                                                              [  9%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_all_jobs PASSED                                                                                                                                                                                                                             [ 11%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_all_with_force_flag PASSED                                                                                                                                                                                                                  [ 14%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_cancelled_by_user PASSED                                                                                                                                                                                                                    [ 16%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_multiple_jobs PASSED                                                                                                                                                                                                                        [ 19%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_single_job_with_confirmation PASSED                                                                                                                                                                                                         [ 21%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_with_force_flag PASSED                                                                                                                                                                                                                      [ 23%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_lists_jobs_with_multiple_filters PASSED                                                                                                                                                                                                       [ 26%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_lists_jobs_with_no_filter PASSED                                                                                                                                                                                                              [ 28%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_lists_jobs_with_substring_match PASSED                                                                                                                                                                                                        [ 30%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_with_no_results PASSED                                                                                                                                                                                                                        [ 33%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lji_displays_job_info PASSED                                                                                                                                                                                                                     [ 35%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lji_with_filter PASSED                                                                                                                                                                                                                           [ 38%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lji_with_various_job_states PASSED                                                                                                                                                                                                               [ 40%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_multiple_job_operations_sequentially PASSED                                                                                                                                                                                                      [ 42%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_pause_invalid_job_raises_exception PASSED                                                                                                                                                                                                        [ 45%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_pause_multiple_jobs PASSED                                                                                                                                                                                                                       [ 47%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_pause_single_job PASSED                                                                                                                                                                                                                          [ 50%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_invalid_value_raises_error PASSED                                                                                                                                                                                                       [ 52%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_confirmation PASSED                                                                                                                                                                                                                [ 54%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_force_flag PASSED                                                                                                                                                                                                                  [ 57%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_large_value PASSED                                                                                                                                                                                                                 [ 59%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_negative_value PASSED                                                                                                                                                                                                              [ 61%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_zero_value PASSED                                                                                                                                                                                                                  [ 64%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_all_jobs PASSED                                                                                                                                                                                                                            [ 66%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_multiple_jobs PASSED                                                                                                                                                                                                                       [ 69%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_single_job PASSED                                                                                                                                                                                                                          [ 71%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_with_force_flag PASSED                                                                                                                                                                                                                     [ 73%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_invalid_value_raises_error PASSED                                                                                                                                                                                                  [ 76%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_with_confirmation PASSED                                                                                                                                                                                                           [ 78%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_with_force_flag PASSED                                                                                                                                                                                                             [ 80%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_with_large_value PASSED                                                                                                                                                                                                            [ 83%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_invalid_value_raises_error PASSED                                                                                                                                                                                                  [ 85%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_confirmation PASSED                                                                                                                                                                                                           [ 88%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_force_flag PASSED                                                                                                                                                                                                             [ 90%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_fractional_cores PASSED                                                                                                                                                                                                       [ 92%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_integer_value PASSED                                                                                                                                                                                                          [ 95%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_unpause_multiple_jobs PASSED                                                                                                                                                                                                                     [ 97%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_unpause_single_job PASSED                                                                                                                                                                                                                        [100%]

=============================================================================================================================================== 42 passed in 0.11s ================================================================================================================================================

…-command-chains' into feature/cueadmin-job-commands-and-tests
@ramonfigueiredo
Copy link
Collaborator Author

Almost ready. Missing just update the docs/ and the cueadmin (integration tests). I will finish this PR soon.
Merged branch:

@ramonfigueiredo
Copy link
Collaborator Author

python -m pytest cueadmin/tests/integration_tests.py -v

=============================================================================================================================================== test session starts ===============================================================================================================================================
cachedir: .pytest_cache
configfile: pyproject.toml
plugins: mock-3.15.1, pyfakefs-5.2.3, cov-7.0.0
collected 36 items                                                                                                                                                                                                                                                                                                

cueadmin/tests/integration_tests.py::ShowAllocationSubscriptionWorkflowTest::test_complete_show_allocation_subscription_workflow PASSED                                                                                                                                                                     [  2%]
cueadmin/tests/integration_tests.py::ShowAllocationSubscriptionWorkflowTest::test_error_recovery_allocation_creation_failure PASSED                                                                                                                                                                         [  5%]
cueadmin/tests/integration_tests.py::ShowAllocationSubscriptionWorkflowTest::test_modify_subscription_after_creation PASSED                                                                                                                                                                                 [  8%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_batch_job_operations PASSED                                                                                                                                                                                                            [ 11%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_dependency_drop_workflow PASSED                                                                                                                                                                                                    [ 13%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_double_pause_error PASSED                                                                                                                                                                                                          [ 16%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_kill_all_workflow PASSED                                                                                                                                                                                                           [ 19%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_listing_and_info_workflow PASSED                                                                                                                                                                                                   [ 22%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_modification_without_pause PASSED                                                                                                                                                                                                  [ 25%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_pause_modify_unpause_kill_workflow PASSED                                                                                                                                                                                          [ 27%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_priority_adjustment_workflow PASSED                                                                                                                                                                                                [ 30%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_resource_modification_workflow PASSED                                                                                                                                                                                              [ 33%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_retry_workflow PASSED                                                                                                                                                                                                              [ 36%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_state_transition_workflow PASSED                                                                                                                                                                                                   [ 38%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_batch_host_allocation_transfer PASSED                                                                                                                                                                                                 [ 41%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_host_lock_move_unlock_workflow PASSED                                                                                                                                                                                                 [ 44%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_host_safe_reboot_workflow PASSED                                                                                                                                                                                                      [ 47%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_host_state_transitions PASSED                                                                                                                                                                                                         [ 50%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_create_job_dependency_workflow PASSED                                                                                                                                                                                                     [ 52%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_dependency_error_circular_detection PASSED                                                                                                                                                                                                [ 55%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_frame_dependency_workflow PASSED                                                                                                                                                                                                          [ 58%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_layer_dependency_workflow PASSED                                                                                                                                                                                                          [ 61%]
cueadmin/tests/integration_tests.py::ShowCleanupWorkflowTest::test_enable_show_after_disable PASSED                                                                                                                                                                                                         [ 63%]
cueadmin/tests/integration_tests.py::ShowCleanupWorkflowTest::test_show_delete_with_active_jobs_error PASSED                                                                                                                                                                                                [ 66%]
cueadmin/tests/integration_tests.py::ShowCleanupWorkflowTest::test_show_disable_kill_delete_workflow PASSED                                                                                                                                                                                                 [ 69%]
cueadmin/tests/integration_tests.py::ResourceReallocationWorkflowTest::test_host_reallocation_with_running_procs PASSED                                                                                                                                                                                     [ 72%]
cueadmin/tests/integration_tests.py::ResourceReallocationWorkflowTest::test_subscription_reduction_during_rendering PASSED                                                                                                                                                                                  [ 75%]
cueadmin/tests/integration_tests.py::ResourceReallocationWorkflowTest::test_subscription_resize_during_active_rendering PASSED                                                                                                                                                                              [ 77%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_host_move_with_invalid_allocation PASSED                                                                                                                                                                                               [ 80%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_job_operation_on_nonexistent_job PASSED                                                                                                                                                                                                [ 83%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_subscription_creation_with_invalid_allocation PASSED                                                                                                                                                                                   [ 86%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_subscription_modification_error_recovery PASSED                                                                                                                                                                                        [ 88%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_allocation_operations_require_authorization PASSED                                                                                                                                                                           [ 91%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_show_delete_without_authorization PASSED                                                                                                                                                                                     [ 94%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_show_operations_require_proper_authorization PASSED                                                                                                                                                                          [ 97%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_subscription_modification_authorization PASSED                                                                                                                                                                               [100%]

=============================================================================================================================================== 36 passed in 0.08s ================================================================================================================================================

@ramonfigueiredo ramonfigueiredo changed the title [cueadmin] Add unit tests and implementation for job management commands [cueadmin] Add unit tests and implementation for job management commands and update integration tests and docs Sep 30, 2025
@ramonfigueiredo ramonfigueiredo changed the title [cueadmin] Add unit tests and implementation for job management commands and update integration tests and docs [cueadmin] Add unit tests and implementation for job management commands and update integration tests and documentation Sep 30, 2025
@ramonfigueiredo ramonfigueiredo changed the title [cueadmin] Add unit tests and implementation for job management commands and update integration tests and documentation [cueadmin] Add job management commands, job unit tests, update integration tests and documentation Sep 30, 2025
Extend JobManagementWorkflowTest class with comprehensive workflow tests for job management commands to verify end-to-end operation sequences.

New integration tests added:
- Job listing and filtering (-lj, -lji with filters)
- Priority adjustment workflow with confirmation
- Retry workflow for dead frames across multiple jobs
- Kill-all jobs workflow with confirmation
- Dependency dropping for multiple jobs
- Resource modification (min/max cores) workflow
- Complete state transition (pause - > unpause - > kill)

These integration tests complement the unit tests in `test_job_commands.py` by verifying that complex command sequences maintain state consistency and work correctly together.

All 36 integration tests passing.
- Updated cueadmin/README.md:
    * Added job management commands to usage section
    * Expanded test suite description
    * Added job command tests to test types

- Updated docs/_docs/reference/commands/cueadmin.md:
    * Added complete "Job Options" section with 10 job commands (-pause, -unpause, -kill, -kill-all, -retry, -retry-all, -drop-depends, -set-min-cores, -set-max-cores, -priority)

- Updated docs/_docs/reference/tools/cueadmin.md:
    * Mentioned job management in overview and key features
    * Added "Job Operations" section
    * Added new workflow sections: Managing Problem Jobs, Cleaning Up Old Jobs
    * Updated test suite info

- Updated docs/_docs/tutorials/cueadmin-tutorial.md:
    * Added new section: Job Management
    * Renumbered subsequent parts
    * Updated Summary and Development sections

All documentation now fully covers job management with commands, workflows, practical examples, safety notes, and best practices.
@ramonfigueiredo ramonfigueiredo marked this pull request as ready for review October 1, 2025 00:45
@ramonfigueiredo
Copy link
Collaborator Author

@DiegoTavares / @lithorus
Ready for review!

@ramonfigueiredo ramonfigueiredo merged commit a7c3853 into AcademySoftwareFoundation:master Oct 1, 2025
20 checks passed
@ramonfigueiredo ramonfigueiredo deleted the feature/cueadmin-job-commands-and-tests branch October 1, 2025 20:31
@jmertic jmertic removed the devdays25 label Nov 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants