Skip to content

fix: shard subprocesses inherit CWD and profile for correct backend selection - #467

Open
rysweet wants to merge 4 commits into
mainfrom
fix/shard-backend-selection
Open

fix: shard subprocesses inherit CWD and profile for correct backend selection#467
rysweet wants to merge 4 commits into
mainfrom
fix/shard-backend-selection

Conversation

@rysweet

@rysweet rysweet commented Apr 4, 2026

Copy link
Copy Markdown
Owner

Summary

This PR contains two independent fixes to skwaq gym eval correctness.


Fix 1: Shard subprocesses inherit CWD and profile for correct backend selection

Root Cause

During skwaq gym eval, shard subprocesses were selecting the Copilot backend instead of Azure AI Foundry. Traced to two bugs in spawn_shard() in crates/cli/src/commands/gym_cmd.rs.

Config loading code path

Config::load()find_config_file() searches:

  1. skwaq.toml in current working directory
  2. .skwaq/config.toml in CWD
  3. ~/.skwaq/config.toml

If no file is found → Config::default()default_llm_backend() returns "copilot".

Bug 1: Missing .current_dir(skwaq_root)

spawn_shard did not call .current_dir(), so shard subprocesses inherited the parent's CWD. If skwaq gym eval was invoked from a directory other than the repo root, Config::load() in each shard would not find skwaq.toml and fall back to Config::default() — selecting Copilot regardless of what the config file specifies.

Fix: Set .current_dir(skwaq_root) so shards always load config from the repo root.

Bug 2: --profile not forwarded to shards

When running gym eval --profile azure, the profile's LLM backend override (reasoning = "azure") was applied in the parent process but not forwarded to shard subprocesses. Each shard called Config::load() directly from skwaq.toml without the profile overlay, potentially selecting a different backend.

Fix: Forward --profile <name> to shard subprocess args when a profile is active.

Changes (Fix 1)

  • crates/cli/src/commands/gym_cmd.rs: spawn_shard gains two new parameters (skwaq_root, profile); both call sites updated (initial spawn + retry-on-early-death).

Fix 2: CyberGym adapter — prevent source_tree_root() over-descent; WARN on patch fallback

Root Cause

CyberGym cases are often packaged as case_dir/<project-1.2.3>/src/vuln.c. The helper source_tree_root() was supposed to strip the packaging wrapper (project-1.2.3/) to find the project root. However, single_child_dir_without_sources() would descend one more level into src/ because it had no guard against conventional source-layout directories. This caused patch-relative paths like src/parser.c to fail resolution, producing false negatives.

Additionally, when both patch_affected_files and paired_case_affected_files failed, the shallow fallback scan ran silently with no log output, making false negatives invisible.

Changes (Fix 2)

  • crates/gym/src/adapters/cybergym.rs:
    • Added SOURCE_LAYOUT_DIRS constant listing conventional source directories (src, lib, include, source, Sources, core, main, libs, includes).
    • single_child_dir_without_sources() now returns None when the sole child directory name is in SOURCE_LAYOUT_DIRS, keeping source_tree_root() at the project root instead of descending into the source tree.
    • Added three regression tests:
      • test_source_tree_root_stops_at_project_root_not_src
      • test_source_tree_root_stops_at_conventional_lib
      • test_patch_affected_files_resolves_src_relative_paths

Verification

Fix 1

  • cargo build -p skwaq
  • cargo clippy -p skwaq --tests -- -D warnings

Fix 2

  • cargo test -q -p skwaq-gym ✅ (all 85 tests pass including 3 new regression tests)
  • cargo clippy -q -p skwaq-gym --tests -- -D warnings ✅ (zero warnings)

CI

CI is triggered on push. Any CI failures will be documented here as root causes are identified.

rysweet and others added 4 commits April 4, 2026 03:05
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removes the explicit 5-minute job timeout that was cancelling the PR
benchmark check before it could complete.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…election

spawn_shard was missing two things that caused Copilot to be selected
instead of Azure AI Foundry during benchmark eval runs:

1. No .current_dir(): shards inherited the parent process CWD.
   If skwaq gym eval was invoked from a directory other than the repo
   root, Config::load() in the shard subprocess could not find skwaq.toml
   and fell back to Config::default() which has reasoning = "copilot".
   Fix: set .current_dir(skwaq_root) so Config::load() always finds
   the repo-root skwaq.toml regardless of invocation directory.

2. No --profile forwarding: when gym eval --profile azure was used,
   the profile's LLM backend override was applied in the parent process
   but the shard subprocesses were spawned without --profile, so they
   used the raw skwaq.toml config rather than the profile-merged config.
   Fix: forward --profile <name> to shards when a profile is active.

Both call sites (initial spawn and retry-on-early-death) are updated.

Root-cause analysis of Copilot vs Azure AI Foundry selection:
- Config::load() searches: skwaq.toml (CWD) -> .skwaq/config.toml (CWD)
  -> ~/.skwaq/config.toml
- default_llm_backend() returns "copilot" (compile-time default)
- If no config file is found, Config::default() selects Copilot
- Shard subprocesses are the code path where CWD-dependency mattered

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on patch fallback

- Add SOURCE_LAYOUT_DIRS constant listing conventional project source
  directories (src, lib, include, source, Sources, core, main, libs, includes)
- single_child_dir_without_sources() now returns None when the sole child
  dir is a SOURCE_LAYOUT_DIR, keeping source_tree_root() at the project
  root instead of descending into the source tree
- Emit tracing::warn when patch_affected_files / paired_case_affected_files
  both fail and the shallow fallback scan is used, making false-negatives
  observable in gym logs
- Add three regression tests:
    test_source_tree_root_stops_at_project_root_not_src
    test_source_tree_root_stops_at_conventional_lib
    test_patch_affected_files_resolves_src_relative_paths

Fixes false negatives where patch-relative paths like 'src/parser.c' failed
to resolve because source_tree_root() had descended into the 'src/' child
instead of stopping at the project root wrapper directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rysweet

rysweet commented Apr 4, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #468. This older branch/PR path picked up unrelated scope and workflow residue; the focused replacement PR carries the backend-selection + CyberGym fixes with QA/docs evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant