-
Notifications
You must be signed in to change notification settings - Fork 235
[cueadmin] Add job management commands, job unit tests, update integration tests and documentation #2014
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[cueadmin] Add job management commands, job unit tests, update integration tests and documentation #2014
Conversation
Create comprehensive integration test suite in /cueadmin/tests/integration_tests.py covering critical workflows: - Show, allocation, and subscription creation workflow - Job lifecycle management (pause, modify, unpause, kill) - Host management including lock, move allocation, unlock operations - Dependency creation and satisfaction (job, layer, frame dependencies) - Show cleanup workflow (disable, kill jobs, delete) - Resource reallocation during active rendering - Error recovery scenarios for failed operations - Permission and authorization workflows Tests use mocked OpenCue backend to verify command execution flow and state consistency across operations without requiring live infrastructure.
Implement comprehensive job management functionality and tests for cueadmin to provide better control over job operations from the command line. Changes: 1. Job Management Commands (cueadmin/cueadmin/common.py): - Add Job Options argument group to parser - Implement -pause/-unpause for job state control - Implement -kill/-kill-all for job termination with confirmation - Implement -retry/-retry-all for frame retry operations - Implement -drop-depends for dependency management - Implement -set-min-cores/-set-max-cores for resource control - Implement -priority for job priority adjustments - All destructive operations require confirmation unless -force flag is used 2. Unit Tests (cueadmin/tests/test_job_commands.py): - 42 comprehensive tests covering all job commands - Tests for -lj/-lji job listing and info display - Tests for job state changes (pause/unpause) - Tests for job termination with confirmation workflows - Tests for frame retry operations - Tests for dependency removal - Tests for resource modifications - Tests for priority adjustments - Error handling tests for invalid job names and values - Tests for various job states (running, finished, failed, paused) All tests passing with clean pylint validation.
Job Operations in CueAdmin vs CuemanI want to address the potential concern about overlap with Tool architecture
Why both tools need job operations1. Different tool purposes
2. Overlap is Intentional, not duplication The functionality serves different use cases:
3. Tool Independence
Conclusion This implementation:
The overlap is by design: |
|
python3 -m pytest cueadmin/tests/test_job_commands.py =============================================================================================================================================== test session starts ===============================================================================================================================================
cachedir: .pytest_cache
configfile: pyproject.toml
plugins: mock-3.15.1, pyfakefs-5.2.3, cov-7.0.0
collected 42 items
cueadmin/tests/test_job_commands.py::JobCommandTests::test_drop_depends_multiple_jobs PASSED [ 2%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_drop_depends_single_job PASSED [ 4%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_drop_depends_with_force_flag PASSED [ 7%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_job_not_found_error_handling PASSED [ 9%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_all_jobs PASSED [ 11%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_all_with_force_flag PASSED [ 14%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_cancelled_by_user PASSED [ 16%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_multiple_jobs PASSED [ 19%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_single_job_with_confirmation PASSED [ 21%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_kill_with_force_flag PASSED [ 23%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_lists_jobs_with_multiple_filters PASSED [ 26%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_lists_jobs_with_no_filter PASSED [ 28%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_lists_jobs_with_substring_match PASSED [ 30%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lj_with_no_results PASSED [ 33%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lji_displays_job_info PASSED [ 35%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lji_with_filter PASSED [ 38%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_lji_with_various_job_states PASSED [ 40%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_multiple_job_operations_sequentially PASSED [ 42%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_pause_invalid_job_raises_exception PASSED [ 45%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_pause_multiple_jobs PASSED [ 47%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_pause_single_job PASSED [ 50%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_invalid_value_raises_error PASSED [ 52%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_confirmation PASSED [ 54%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_force_flag PASSED [ 57%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_large_value PASSED [ 59%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_negative_value PASSED [ 61%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_priority_with_zero_value PASSED [ 64%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_all_jobs PASSED [ 66%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_multiple_jobs PASSED [ 69%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_single_job PASSED [ 71%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_retry_with_force_flag PASSED [ 73%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_invalid_value_raises_error PASSED [ 76%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_with_confirmation PASSED [ 78%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_with_force_flag PASSED [ 80%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_max_cores_with_large_value PASSED [ 83%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_invalid_value_raises_error PASSED [ 85%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_confirmation PASSED [ 88%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_force_flag PASSED [ 90%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_fractional_cores PASSED [ 92%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_set_min_cores_with_integer_value PASSED [ 95%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_unpause_multiple_jobs PASSED [ 97%]
cueadmin/tests/test_job_commands.py::JobCommandTests::test_unpause_single_job PASSED [100%]
=============================================================================================================================================== 42 passed in 0.11s ================================================================================================================================================ |
…-command-chains' into feature/cueadmin-job-commands-and-tests
|
Almost ready. Missing just update the docs/ and the cueadmin (integration tests). I will finish this PR soon. |
|
python -m pytest cueadmin/tests/integration_tests.py -v =============================================================================================================================================== test session starts ===============================================================================================================================================
cachedir: .pytest_cache
configfile: pyproject.toml
plugins: mock-3.15.1, pyfakefs-5.2.3, cov-7.0.0
collected 36 items
cueadmin/tests/integration_tests.py::ShowAllocationSubscriptionWorkflowTest::test_complete_show_allocation_subscription_workflow PASSED [ 2%]
cueadmin/tests/integration_tests.py::ShowAllocationSubscriptionWorkflowTest::test_error_recovery_allocation_creation_failure PASSED [ 5%]
cueadmin/tests/integration_tests.py::ShowAllocationSubscriptionWorkflowTest::test_modify_subscription_after_creation PASSED [ 8%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_batch_job_operations PASSED [ 11%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_dependency_drop_workflow PASSED [ 13%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_double_pause_error PASSED [ 16%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_kill_all_workflow PASSED [ 19%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_listing_and_info_workflow PASSED [ 22%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_modification_without_pause PASSED [ 25%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_pause_modify_unpause_kill_workflow PASSED [ 27%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_priority_adjustment_workflow PASSED [ 30%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_resource_modification_workflow PASSED [ 33%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_retry_workflow PASSED [ 36%]
cueadmin/tests/integration_tests.py::JobManagementWorkflowTest::test_job_state_transition_workflow PASSED [ 38%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_batch_host_allocation_transfer PASSED [ 41%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_host_lock_move_unlock_workflow PASSED [ 44%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_host_safe_reboot_workflow PASSED [ 47%]
cueadmin/tests/integration_tests.py::HostManagementWorkflowTest::test_host_state_transitions PASSED [ 50%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_create_job_dependency_workflow PASSED [ 52%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_dependency_error_circular_detection PASSED [ 55%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_frame_dependency_workflow PASSED [ 58%]
cueadmin/tests/integration_tests.py::DependencyWorkflowTest::test_layer_dependency_workflow PASSED [ 61%]
cueadmin/tests/integration_tests.py::ShowCleanupWorkflowTest::test_enable_show_after_disable PASSED [ 63%]
cueadmin/tests/integration_tests.py::ShowCleanupWorkflowTest::test_show_delete_with_active_jobs_error PASSED [ 66%]
cueadmin/tests/integration_tests.py::ShowCleanupWorkflowTest::test_show_disable_kill_delete_workflow PASSED [ 69%]
cueadmin/tests/integration_tests.py::ResourceReallocationWorkflowTest::test_host_reallocation_with_running_procs PASSED [ 72%]
cueadmin/tests/integration_tests.py::ResourceReallocationWorkflowTest::test_subscription_reduction_during_rendering PASSED [ 75%]
cueadmin/tests/integration_tests.py::ResourceReallocationWorkflowTest::test_subscription_resize_during_active_rendering PASSED [ 77%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_host_move_with_invalid_allocation PASSED [ 80%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_job_operation_on_nonexistent_job PASSED [ 83%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_subscription_creation_with_invalid_allocation PASSED [ 86%]
cueadmin/tests/integration_tests.py::ErrorRecoveryWorkflowTest::test_subscription_modification_error_recovery PASSED [ 88%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_allocation_operations_require_authorization PASSED [ 91%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_show_delete_without_authorization PASSED [ 94%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_show_operations_require_proper_authorization PASSED [ 97%]
cueadmin/tests/integration_tests.py::PermissionAuthorizationWorkflowTest::test_subscription_modification_authorization PASSED [100%]
=============================================================================================================================================== 36 passed in 0.08s ================================================================================================================================================ |
Extend JobManagementWorkflowTest class with comprehensive workflow tests for job management commands to verify end-to-end operation sequences. New integration tests added: - Job listing and filtering (-lj, -lji with filters) - Priority adjustment workflow with confirmation - Retry workflow for dead frames across multiple jobs - Kill-all jobs workflow with confirmation - Dependency dropping for multiple jobs - Resource modification (min/max cores) workflow - Complete state transition (pause - > unpause - > kill) These integration tests complement the unit tests in `test_job_commands.py` by verifying that complex command sequences maintain state consistency and work correctly together. All 36 integration tests passing.
- Updated cueadmin/README.md:
* Added job management commands to usage section
* Expanded test suite description
* Added job command tests to test types
- Updated docs/_docs/reference/commands/cueadmin.md:
* Added complete "Job Options" section with 10 job commands (-pause, -unpause, -kill, -kill-all, -retry, -retry-all, -drop-depends, -set-min-cores, -set-max-cores, -priority)
- Updated docs/_docs/reference/tools/cueadmin.md:
* Mentioned job management in overview and key features
* Added "Job Operations" section
* Added new workflow sections: Managing Problem Jobs, Cleaning Up Old Jobs
* Updated test suite info
- Updated docs/_docs/tutorials/cueadmin-tutorial.md:
* Added new section: Job Management
* Renumbered subsequent parts
* Updated Summary and Development sections
All documentation now fully covers job management with commands, workflows, practical examples, safety notes, and best practices.
|
@DiegoTavares / @lithorus |
a7c3853
into
AcademySoftwareFoundation:master
Link the Issue(s) this Pull Request is related to.
Summarize your change.
Implement comprehensive job management functionality and tests for cueadmin to provide better control over job operations from the command line.
Changes:
Extend JobManagementWorkflowTest class with comprehensive workflow tests for job management commands to verify end-to-end operation sequences.
New integration tests added:
These integration tests complement the unit tests in
test_job_commands.pyby verifying that complex command sequences maintain state consistency and work correctly together.Updated cueadmin/README.md:
Updated docs/_docs/reference/commands/cueadmin.md:
Updated docs/_docs/reference/tools/cueadmin.md:
Updated docs/_docs/tutorials/cueadmin-tutorial.md:
All documentation now fully covers job management with commands, workflows, practical examples, safety notes, and best practices.