Skip to content

feat: add --slice flag for worker scenario grouping#4617

Draft
cidrblock wants to merge 4 commits intomainfrom
slice
Draft

feat: add --slice flag for worker scenario grouping#4617
cidrblock wants to merge 4 commits intomainfrom
slice

Conversation

@cidrblock
Copy link
Copy Markdown

@cidrblock cidrblock commented Feb 28, 2026

Summary

Adds a --slice flag (and config file support) to control how scenarios are grouped into work units when using --workers. With --slice=1 (the default), scenarios are grouped by their top-level directory segment so all CRUD states for a resource run sequentially on one worker. With --slice=2, each leaf scenario is dispatched independently (pre-slice behavior).

This is relevant for collections with nested scenario directories (e.g. appliance_vlans/gathered, appliance_vlans/merged) where related scenarios share state and must execute in order within a worker.

Changes by file

src/molecule/click_cfg.py — New slice property on CliOptions (experimental, default "1").

src/molecule/types.pyslice: int added to both CommandArgs and ConfigData TypedDicts.

src/molecule/command/test.py, check.py, destroy.py"slice" added to @common_options and wired into command_args via int(ctx.params["slice"]). No other runtime changes.

src/molecule/config.pyslice property reads from the config dict (self.config.get("slice", 1)), following the same pattern as shared_state. _apply_cli_overrides extended so CLI --slice takes precedence over the config file value.

src/molecule/data/molecule.json — Schema entry for slice (integer, default 1, minimum 1).

src/molecule/scenarios.pyslice property on Scenarios reads from the first scenario's config, mirroring the shared_state aggregation pattern.

src/molecule/worker.py — Core implementation:

  • _slice_key(name, depth) — splits scenario name on / and joins the first depth segments to derive the grouping key.
  • _group_scenarios_by_slice(scenarios, depth) — groups Scenario objects by _slice_key, preserving original sort order.
  • run_scenario_slice(entries, ...) — executes a list of (molecule_file, name) pairs sequentially in one worker process, stopping on first failure within the slice. Returns list[SliceResult].
  • run_scenarios_parallel — refactored to group via _group_scenarios_by_slice and submit one future per group to run_scenario_slice instead of one future per scenario. Fail-fast and continue-on-failure semantics apply at the slice level. Log output reports slice count and depth.
  • validate_worker_args — validates that --slice (non-default) requires --workers > 1.
  • run_one_scenario remains unchanged and exported for backward compatibility but is no longer called internally — run_scenario_slice handles all dispatch.

pyproject.tomlverbosity_assertions changed from 2 (int) to "2" (string). Newer pytest requires this config option to be a string; the int type caused TypeError in all integration tests with assertion failures. Pre-existing issue, not introduced by this PR.

tests/fixtures/integration/test_workers/ — Fixture scenarios restructured from flat (scenario_0..5) to nested (group_a/scenario_0..2, group_b/scenario_3..5) to enable slice testing. config.yml now includes slice: 1.

tests/unit/test_worker.py — New tests for _slice_key (4 cases), _group_scenarios_by_slice (4 cases), run_scenario_slice (success + failure-stops-early), validate_worker_args slice validation (2 cases). Existing run_scenarios_parallel tests updated for the new list[SliceResult] return format.

tests/unit/test_click_cfg.pytest_slice_option verifying the new CLI option properties.

tests/integration/test_workers.py — Scenario references updated for nested names. Two new tests: test_workers_slice_groups_by_resource (verifies 2 slices at depth=1) and test_workers_slice_depth_2 (verifies 6 slices at depth=2).

docs/guides/parallel.md — Documents --slice with a behavior table, CLI examples, config file example, and precedence notes.

Runtime behavior

Existing --workers behavior is preserved. The default --slice=1 groups scenarios by their first path segment, which is a change from pre-slice behavior (individual dispatch) but produces the same results since scenarios within a group still execute sequentially. Users can opt into the previous individual-dispatch behavior with --slice=2.

Sequential mode (--workers 1 or no --workers) is completely unaffected — slice in the config is silently ignored.

Test plan

  • Unit tests pass (_slice_key, _group_scenarios_by_slice, run_scenario_slice, validation, parallel dispatch)
  • Integration tests pass (slice depth 1 and 2, parallel success, verbose, continue-on-failure, fail-fast)
  • tox -e lint clean (ruff, cspell, pydoclint, pylint, mypy, codespell)

@ssbarnea ssbarnea marked this pull request as draft March 5, 2026 13:29
@github-actions github-actions bot added feat and removed feat labels Mar 5, 2026
@ssbarnea
Copy link
Copy Markdown
Member

ssbarnea commented Mar 5, 2026

@cidrblock bring it out of draft once is passing all CI checks.

…ctations

The ../inventory.yml relative path in config.yml only works for flat
scenarios.  For group_a/scenario_0 it resolves to group_a/inventory.yml
(wrong) instead of molecule/inventory.yml.  Add inventory copies in each
group directory so ansible finds hosts and actually runs the playbooks.

Also fix slice count expectations: default scenario is included in
slicing, so depth=1 yields 3 slices and depth=2 yields 7.

Remove unused 'psql' from dictionary.txt (flagged by cspell hook).

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants