[WIP do not review]

[eugenevinitsky]

Summary

Inline clean eval: zero perturbations and road-segment dropout, enforce red-light stops. Logs validation/* metrics under controlled conditions so progress isn't confounded by training-time noise.

• build_eval_overrides(clean=True) zeros lane_segment_dropout, boundary_segment_dropout, partner_blindness_prob, phantom_braking_{prob,trigger_prob}, and flips traffic_light_behavior to 1 (stop at red). Perturbations are always zeroed at eval regardless of clean since they don't change obs shape.
• _swap_policy_obs_counts context manager temporarily aligns the live training policy's obs_{lane,boundary}_segment_count with the clean eval env. The GigaFlow encoder is truly count-invariant (shared MLP + max-pool over segments) — only the obs buffer slicing depends on these counts. Safe swap.
• eval_multi_scenarios takes clean= and wraps the forward loop with the swap.
• Inline eval site in _train reads eval.clean_eval (new ini default: True) and plumbs it through.
• Removed stale _sys_instr.stderr.write debug lines in eval_multi_scenarios that crashed on NameError when the eval completed (was only imported in the render function).

Also noted: the inline render path was logging zero metrics because render_max_steps (e.g. 300) is much less than scenario_length (3000), so no episode completes within the cap and global_infos stays empty. The metric-only eval_multi_scenarios doesn't have this problem.

Test plan

• Short training on emerge2 with dropout=0.4, perturbations>0, multi_scenario_eval=True, clean_eval=True (default). Policy was built at dropout=0.4 (obs_lane=48) and successfully forwarded through the clean eval env (obs_lane=80) via the swap, completing all 4 scenarios in ~8s with populated metrics (avg_distance_per_infraction = 7.47, red_light_violation_rate = 0.01).
• Eval summary CSV written to experiments/.../validation/epoch_N/gigaflow/.
• Longer run on cluster with wandb to verify validation/* tab populates.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a “clean” inline evaluation mode for Drive to reduce evaluation-time noise (dropout/perturbations) and enforce red-light stops, while allowing reuse of the live training policy by temporarily aligning its road-segment slicing counts with the eval environment.

Changes:

• Introduces clean_eval plumbing from config → build_eval_overrides(..., clean=...) → eval_multi_scenarios(..., clean=...).
• Adds _swap_policy_obs_counts context manager and wraps the eval rollout loop to handle mismatched road-segment counts between training and clean eval.
• Forces robustness perturbations (partner blindness / phantom braking) to zero during eval and removes stale _sys_instr.stderr.write debug lines that could crash.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
pufferlib/pufferl.py	Adds clean-eval overrides, the policy obs-count swap context manager, and threads clean through inline + CLI evaluation paths.
pufferlib/config/ocean/drive.ini	Adds eval.clean_eval = True default and documents its intent for controlled validation metrics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

New behavior depends on _swap_policy_obs_counts preventing forward-time crashes when training/eval road-segment dropouts differ (obs buffer size changes). There are no automated tests covering this path (e.g., training policy built with dropout>0, eval env with dropout=0, clean=True), so regressions here are likely to be caught only at runtime. Adding a focused test that constructs a Drive policy/env pair with mismatched segment counts and verifies eval_multi_scenarios(clean=True) runs at least one short rollout without shape/index errors would help lock in this guarantee.

[Copilot]

The clean docstring now says clean eval is only safe when the policy is rebuilt and that inline eval will crash due to an obs-shape mismatch. In this PR, inline eval explicitly passes clean=clean_eval and relies on _swap_policy_obs_counts to make this safe, so the docstring is misleading/outdated. Please update the docstring to describe the current safety model (i.e., inline eval is safe when using the swap, and any remaining limitations).

[Copilot]

The inline comment under if clean: says "NEVER pass clean=True from an inline-eval call site", but train() now does exactly that and eval_multi_scenarios contains logic to support it via _swap_policy_obs_counts. This comment should be revised (or removed) because it contradicts the new behavior and could cause future callers to avoid the supported path or misdiagnose failures.

Suggested change

	# Dropout changes the obs shape. Only safe when the policy is
	# rebuilt from the eval env (standalone eval / render_scenario).
	# NEVER pass clean=True from an inline-eval call site — the live
	# training policy's encoder was built for the training obs shape.
	# Dropout changes the obs shape. Standalone eval / render_scenario
	# are safe because the policy is rebuilt from the eval env. Inline
	# eval may also pass clean=True, but only when the caller updates
	# the live policy's expected obs counts to match the eval env.

[Copilot]

eval_multi_scenarios has a clean parameter, but when args is None the env overrides are built using clean_from_config only (ignoring the passed clean value). This can lead to inconsistent behavior where clean=True affects swapping but not the env configuration, or callers cannot force clean=True programmatically unless they also set config. Consider making clean default to None and resolving precedence explicitly (e.g., parameter overrides config), and pass the resolved value into build_eval_overrides.

Suggested change

	eval_overrides = build_eval_overrides(
	simulation_mode=tmp_args["eval_simulation"],
	num_agents=num_agents_eval,
	num_scenarios=tmp_args["num_scenarios"],
	map_dir=map_dir,
	num_carla_maps=tmp_args.get("num_carla_maps", 8),
	clean=clean_from_config,
	)
	args = load_eval_multi_scenarios_config(env_name, model_path, eval_overrides)
	clean = clean or clean_from_config
	resolved_clean = clean or clean_from_config
	eval_overrides = build_eval_overrides(
	simulation_mode=tmp_args["eval_simulation"],
	num_agents=num_agents_eval,
	num_scenarios=tmp_args["num_scenarios"],
	map_dir=map_dir,
	num_carla_maps=tmp_args.get("num_carla_maps", 8),
	clean=resolved_clean,
	)
	args = load_eval_multi_scenarios_config(env_name, model_path, eval_overrides)
	clean = resolved_clean