sync_orchestration.py: cost extraction reads wrong tuple index for generate/crash/fix/verify operations

[Serhan-Asad]

Bug description

In pdd/sync_orchestration.py, the sync loop extracts cost and model from operation results by always reading result[1] as cost and result[2] as model. This is incorrect because each operation returns a different tuple format.

Affected code

File: pdd/sync_orchestration.py

Location 1 — line 1857-1858 (cost accumulation):

# Cost is always at index 1 in both 3-tuple and 4-tuple returns
cost = result[1] if len(result) >= 2 and isinstance(result[1], (int, float)) else 0.0

Location 2 — lines 1889-1890 (log entry):

actual_cost = result[1] if isinstance(result[1], (int, float)) else 0.0
model_name = result[2] if len(result) >= 3 and isinstance(result[2], str) else 'unknown'

Return formats per operation

Each function returns a different tuple. The correct cost and model indices are:

• generate (code_generator_main, line 1207): returns (content, was_incremental, cost, model) — 4-tuple, cost at index 2, model at index 3
• crash (crash_main, line 389): returns (success, final_code, final_program, attempts, cost, model) — 6-tuple, cost at index 4, model at index 5
• fix (fix_main, line 545): returns (success, fixed_unit_test, fixed_code, attempts, total_cost, model_name) — 6-tuple, cost at index 4, model at index 5
• verify (fix_verification_main, line 689): returns (success, final_program, final_code, attempts, total_cost, model_name) — 6-tuple, cost at index 4, model at index 5
• example (context_generator_main, line 205): returns (content, cost, model) — 3-tuple, cost at index 1, model at index 2 (correct as-is)
• test/test_extend (cmd_test_main, line 420): returns (content, cost, model, agentic_success) — 4-tuple, cost at index 1, model at index 2 (correct as-is)

Why the isinstance check doesn't catch it

Python's bool subclasses int, so isinstance(False, (int, float)) returns True. For generate, result[1] is was_incremental=False, which passes the type check and cost becomes 0 (the integer value of False).

Expected behavior

• _extract_cost_from_result(operation, result) should return the cost from the correct index based on which operation produced the result
• _extract_model_from_result(operation, result) should return the model name from the correct index based on which operation produced the result
• The bool check should explicitly exclude bool: isinstance(cost, (int, float)) and not isinstance(cost, bool)

Fix approach

Create two helper functions that dispatch on the operation name:

• Operations crash, fix, verify: cost at result[4], model at result[5]
• Operation generate: cost at result[2], model at result[3]
• All other operations (example, test, test_extend, auto-deps): cost at result[1], model at result[2]

Replace the inline result[1] reads at both locations (line 1858 and lines 1889-1890) with calls to these helpers.

Reproduction

Run pdd sync on any prompt and inspect .pdd/meta/<basename>_sync.log:

• The generate entry will show "actual_cost": false and "model": "unknown"
• The fix entry (if triggered) will show "actual_cost": 0.0 and "model" containing the source code string

[Serhan-Asad]

Step 1: Duplicate Check

Status: No duplicates found — this is a new issue

Search Performed

• Searched for: "sync_orchestration cost", "cost extraction tuple", "result tuple index model cost sync", "generate crash fix verify cost tuple index", "was_incremental bool cost", "actual_cost false model unknown", "sync cost model wrong"
• Issues reviewed: ~50 (across 7 searches)
• Closely examined: Budget tracker drops test/test_extend costs due to wrong tuple index #508, PDD sync's cost tracking is inaccurate #535, PDD sync doesn't accumulate cost when comparing with the max-budget #364, Race Condition in LLM Cost Tracking Causes Data Corruption #375

Findings

#508 (CLOSED — resolved): "Budget tracker drops test/test_extend costs due to wrong tuple index." This is the closest match — same category of bug (wrong tuple index for cost extraction in sync_orchestration.py). However, #508 only fixed test and test_extend operations (4-tuple from cmd_test_main). The fix changed result[-2] to result[1] at line 1752 and extended the logging condition to include test_extend. Issue #555 covers entirely different operations — generate (cost at index 2), crash/fix/verify (cost at index 4) — none of which were addressed by #508's fix. In fact, #508's fix made line 1858 always use result[1], which is correct for test/test_extend/example but wrong for generate (where result[1] is was_incremental, a bool) and crash/fix/verify (where result[1] is a success bool or content string).

#535 (OPEN — unresolved): "PDD sync's cost tracking is inaccurate." Filed by Jiamin with a screenshot showing suspicious cost. A comment says "bug fixed: PR #480" but the issue remains open, and the pdd bug workflow completely failed (all 3 steps FAILED). This is a vague observation about cost inaccuracy, not a specific analysis of the tuple indexing root cause. #555 provides the detailed root cause analysis that #535 lacks. Not a duplicate — #535 is a symptom report; #555 identifies the specific code defect.

#364 (CLOSED — resolved): "PDD sync doesn't accumulate cost when comparing with max-budget." Different root cause — was about cumulative cost not being accumulated across sync steps, not about wrong tuple indices.

#375 (OPEN): "Race Condition in LLM Cost Tracking Causes Data Corruption." Different root cause entirely — a concurrency/thread-safety issue in _LAST_CALLBACK_DATA.

Relationship to #508

Issue #555 is a continuation of the same class of bug discovered in #508, but affecting the remaining operations that #508 did not address. The result[1] pattern established by #508's fix is correct for test/test_extend/example (3-tuple and 4-tuple where cost is at index 1) but fails for:

• generate: 4-tuple (content, was_incremental, cost, model) — cost at index 2
• crash/fix/verify: 6-tuple (success, ..., cost, model) — cost at index 4

The isinstance(result[1], (int, float)) check doesn't catch it because Python's bool subclasses int, so isinstance(False, (int, float)) returns True.

---

Proceeding to Step 2: Documentation Check

[Serhan-Asad]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• README.md — Lines 706-758 document cost tracking via --output-cost CSV and --budget. Lines 855-883 document sync logs in .pdd/meta/ with "estimated costs for recommended operations." No documentation of the internal tuple return formats used by operation functions.
• CONTRIBUTING.md — No mention of return type contracts for operation functions or cost extraction logic.
• docs/sync_architecture_analysis.md — Line 133 already identifies this as Bug add capability to configure tags allowed or disallowed for security #26 ("Inconsistent Return Handling") at HIGH severity, noting "Different ops return different formats" at the (now outdated) location sync_orchestration.py:1057-1089. Line 267 lists "Return value normalization" as a needed unit test. This pre-existing analysis confirms the issue but the bug was never fixed.
• docs/ONBOARDING.md, docs/TUTORIALS.md, docs/prompting_guide.md — No relevant content about internal cost extraction.

Findings

The bug is confirmed. I verified every claim in the issue against the source code:

1. Return formats (verified from type annotations AND actual return statements):

Operation	Function	Return type	Cost index	Model index
generate	code_generator_main (line 1207)	(str, bool, float, str)	2	3
crash	crash_main (line 389)	(bool, str, str, int, float, str)	4	5
fix	fix_main (line 545)	(bool, str, str, int, float, str)	4	5
verify	fix_verification_main (line 689)	(bool, str, str, int, float, str)	4	5
example	context_generator_main (line 205)	(str, float, str)	1	2
test	cmd_test_main (line 420)	(str, float, str, bool|None)	1	2

2. Extraction code always reads index 1 as cost, index 2 as model:

Location 1 — line 1858 (cost accumulation during operation):

cost = result[1] if len(result) >= 2 and isinstance(result[1], (int, float)) else 0.0

Location 2 — lines 1889-1890 (log entry after operation):

actual_cost = result[1] if isinstance(result[1], (int, float)) else 0.0
model_name = result[2] if len(result) >= 3 and isinstance(result[2], str) else 'unknown'

This is correct only for example and test/test_extend. For the other four operations, it reads the wrong indices.

3. Bool subclassing int (verified):

Python confirms isinstance(False, (int, float)) returns True. For generate, result[1] is was_incremental (a bool), which passes the isinstance check. The cost becomes int(False) = 0 or int(True) = 1, silently producing wrong values instead of falling through to the else branch.

4. Concrete impact per operation:

• generate: result[1] = was_incremental (bool) → cost recorded as 0 or 1 instead of actual cost. result[2] = total_cost (float, not str) → model recorded as 'unknown'. Actual cost and model are silently lost.
• crash: result[1] = final_code (str) → isinstance check fails, cost becomes 0.0. result[2] = final_program (str) → model becomes the entire program source code string. Cost lost, model polluted with source code.
• fix: result[1] = fixed_unit_test (str) → cost becomes 0.0. result[2] = fixed_code (str) → model becomes the entire fixed code string. Cost lost, model polluted with source code.
• verify: result[1] = final_program (str) → cost becomes 0.0. result[2] = final_code (str) → model becomes the entire code string. Cost lost, model polluted with source code.

5. Documentation gap:

There is no documentation of the return type contract between operation functions and sync_orchestration.py. The existing inline comment on line 1883-1884 acknowledges two different formats exist but the code does not actually handle them differently (both branches on lines 1886-1890 read the same indices). The docs/sync_architecture_analysis.md already flagged this as Bug #26 (HIGH severity) but it was never addressed.

---

Proceeding to Step 3: Triage

[Serhan-Asad]

Step 3: Triage

Status: Proceed

Summary

This issue provides comprehensive, verified information for reproduction:

1. Expected behavior — Cost and model should be extracted from the correct tuple index per operation type (e.g., generate has cost at index 2, crash/fix/verify at index 4).

2. Actual behavior — Code unconditionally reads result[1] as cost and result[2] as model, which is only correct for example/test/test_extend. For the other 4 operations, this yields wrong values — and the isinstance guard fails to catch it because bool subclasses int in Python.

3. Reproduction path — Run pdd sync on any prompt and inspect .pdd/meta/<basename>_sync.log. The generate entry will show "actual_cost": false and "model": "unknown".

4. Code locations confirmed — Step 2 verified all tuple return formats against source code. The bug exists at two specific locations in sync_orchestration.py (lines ~1857-1858 and ~1889-1890), and was already internally flagged as Bug add capability to configure tags allowed or disallowed for security #26 (HIGH severity) in docs/sync_architecture_analysis.md.

No ambiguity remains — the exact functions, return formats, indices, and affected code paths are all identified and verified.

---

Proceeding to Step 4: Reproduction

[Serhan-Asad]

Step 4: Reproduction

Status: Reproduced

Environment

• Python 3.12.7 (Anaconda)
• pdd-cli v0.0.157 (editable install from source)
• macOS Darwin 24.3.0
• pytest 8.3.5

Steps Taken

1. Read the cost extraction logic at sync_orchestration.py:1857-1858 (budget accumulation) and sync_orchestration.py:1885-1890 (log entry)
2. Verified return tuple formats for all 6 operations by reading return statements in code_generator_main.py, crash_main.py, fix_main.py, fix_verification_main.py, context_generator_main.py, and cmd_test_main.py — all match the issue description exactly
3. Wrote 12 unit tests that replicate the exact extraction logic from source and feed it representative tuples from each operation
4. Ran tests — all 12 pass, confirming every described symptom

Observed Behavior

generate (4-tuple: (content, was_incremental, cost, model)):

• result[1] is was_incremental (bool). Since isinstance(False, (int, float)) is True (bool subclasses int), cost silently becomes 0 (int value of False)
• When was_incremental=True, cost becomes 1 — recording $1.00 instead of actual cost
• Log model reads result[2] which is the cost float (0.0042), fails isinstance(str) check → "unknown"

crash (6-tuple: (success, final_code, final_program, attempts, cost, model)):

• result[1] is final_code (str), fails isinstance(int, float) → cost defaults to 0.0
• Log model reads result[2] = program path string like "crash_program.py" → recorded as model name

fix (6-tuple: (success, fixed_test, fixed_code, attempts, cost, model)):

• result[1] is fixed_unit_test (str) → cost defaults to 0.0
• Log model reads result[2] = entire fixed source code string → recorded as model name

verify (6-tuple: (success, final_program, final_code, attempts, cost, model)):

• result[1] is final_program (str) → cost defaults to 0.0
• Log model reads result[2] = entire verified code string → recorded as model name

example and test/test_extend: Work correctly (cost at index 1, model at index 2).

12 passed in 0.26s — all bugs confirmed

Affected Files

• pdd/sync_orchestration.py:1858 — budget cost accumulation always reads result[1]
• pdd/sync_orchestration.py:1889-1890 — log cost/model always reads result[1]/result[2]

Notes

• The bool subclass issue makes generate particularly insidious: it silently passes the type check and records False as cost 0 (or True as $1.00), with no error or warning
• For crash/fix/verify, the string at result[1] fails the isinstance check so cost just defaults to 0.0 — less dangerous but still wrong
• The log model field for fix could contain entire source code files, since result[2] is fixed_code
• Reproduction test file: tests/test_e2e_issue_555_cost_extraction_wrong_index.py

---

Proceeding to Step 5: Root Cause Analysis

[Serhan-Asad]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

The sync loop in sync_orchestration.py extracts cost and model from operation results by always reading result[1] as cost and result[2] as model. This assumption is only correct for 2 of 6 operation types (example and test/test_extend). The remaining 4 operations (generate, crash, fix, verify) return tuples where cost and model are at different indices. The isinstance(result[1], (int, float)) guard fails to catch the generate case because Python's bool subclasses int, so was_incremental=False passes the type check and gets silently recorded as cost $0.00 (or $1.00 when incremental). For crash/fix/verify, result[1] is a string, so cost falls back to 0.0 — the real cost at result[4] is never read. Additionally, operation_log.py:339 has the identical bug in the log_operation() decorator.

Technical Details

• Location 1: pdd/sync_orchestration.py:1857-1858 (cost accumulation)
• Location 2: pdd/sync_orchestration.py:1889-1890 (log entry extraction)
• Location 3: pdd/operation_log.py:339-340 (same bug in log_operation() decorator)

Problematic Code (sync_orchestration.py:1857-1858):

# Cost is always at index 1 in both 3-tuple and 4-tuple returns
cost = result[1] if len(result) >= 2 and isinstance(result[1], (int, float)) else 0.0

Problematic Code (sync_orchestration.py:1889-1890):

actual_cost = result[1] if isinstance(result[1], (int, float)) else 0.0
model_name = result[2] if len(result) >= 3 and isinstance(result[2], str) else 'unknown'

Problematic Code (operation_log.py:339-340):

if isinstance(result[1], (int, float)): cost = float(result[1])
if isinstance(result[2], str): model = str(result[2])

Why it fails — per operation:

Operation	result[1] (read as cost)	Actual type	isinstance check	Recorded cost	Real cost index
generate	was_incremental	bool	True (bool⊂int)	0 or 1	[2]
crash	final_code	str	False	0.0	[4]
fix	fixed_unit_test	str	False	0.0	[4]
verify	final_program	str	False	0.0	[4]
example	total_cost	float	True ✓	correct	[1] ✓
test	total_cost	float	True ✓	correct	[1] ✓

External Research

• Search terms: "python bool isinstance int", "python bool subclass int cost extraction"
• Sources checked: Python docs, CPython source
• Relevant findings: This is a well-known Python quirk — bool is a subclass of int (PEP 285). isinstance(False, int) returns True by design. The correct guard is isinstance(x, (int, float)) and not isinstance(x, bool). This is not a dependency bug — it's a logic error in the extraction code.

Repository History

• 88d3afbac (Greg, 2025-12-23): Original cost tracking scaffolding. Used result[-2] and result[-1] for cost/model — which partially worked for some tuple sizes but not all.
• 999772c81 (Greg, 2026-01-27): Added awareness that cmd_test_main returns a 4-tuple, added the test/test_extend special-case branch.
• a5cd81e85 (Serhan, 2026-02-13, PR fix: budget tracker drops test costs due to tuple index bug #518): Fixed Budget tracker drops test/test_extend costs due to wrong tuple index #508 — added if operation in ('test', 'test_extend') guard. Used result[-2]/result[-1] for the else branch.
• 35c99dc09 (Greg, 2026-02-16, "Bump version"): This is the regression commit. Changed the else branch from result[-2]/result[-1] (partially correct for 6-tuples) back to result[1]/result[2] with the comment "Cost is always at index 1 in both 3-tuple and 4-tuple returns" — this comment is factually wrong for generate (4-tuple with bool at [1]) and doesn't account for 6-tuple operations at all.
• 69bb542cb (Greg, 2026-01-18): operation_log.py decorator — uses same result[1]/result[2] pattern, identical bug.

Related PRs:

• PR fix: budget tracker drops test costs due to tuple index bug #518 (merged): Fixed Budget tracker drops test/test_extend costs due to wrong tuple index #508 for test/test_extend only — the fix was correct but incomplete
• PR fix: Accumulate retry costs instead of overwriting in _LAST_CALLBACK_DATA #519 (merged): Fixed retry cost accumulation (separate issue)
• PR Add failing tests for #364: Cumulative cost display bug #370 (merged): Fixed cumulative cost display (separate issue)

Similar past issues:

• Budget tracker drops test/test_extend costs due to wrong tuple index #508 (closed): Same category — wrong tuple index for test/test_extend. Partially fixed by PR fix: budget tracker drops test costs due to tuple index bug #518, then partially regressed by 35c99dc09.
• PDD sync's cost tracking is inaccurate #535 (open): Jiamin reports inaccurate sync costs — likely a downstream symptom of this bug.

Experiments Performed

1. Python bool-subclass verification: Confirmed isinstance(False, (int, float)) → True, isinstance(True, (int, float)) → True. int(False) = 0, int(True) = 1. This proves generate's was_incremental passes the guard silently.
2. Buggy extraction simulation: Ran all 6 operation tuples through the current extraction logic. Results:
   • generate(inc=False): cost=False (0), model='unknown' (float doesn't pass str check)
   • generate(inc=True): cost=True (1), model='unknown'
   • crash: cost=0.0 (string fails isinstance), model='crash_program.py' (wrong field)
   • fix: cost=0.0, model='fixed code' (wrong field)
   • verify: cost=0.0, model='verified code' (wrong field)
3. Test suite execution: All 12 tests in test_e2e_issue_555_cost_extraction_wrong_index.py pass, confirming the bug.

Fix Location

• File(s) to modify: pdd/sync_orchestration.py (primary), pdd/operation_log.py (secondary — same bug)
• Why this location: The root cause is in sync_orchestration.py where cost/model are extracted from heterogeneous operation result tuples. The bug is NOT in the operations themselves (they return correct data) — it's in the consumer that assumes a uniform tuple format. operation_log.py:339-340 has the identical bug in the log_operation() decorator.
• Existing pattern: PR fix: budget tracker drops test costs due to tuple index bug #518 already established the pattern of dispatching on operation name (if operation in ('test', 'test_extend')). The fix extends this same dispatch pattern to cover all 6 operation types.
• Fix approach: Create two helper functions in sync_orchestration.py:

  def _extract_cost_from_result(operation: str, result: tuple) -> float:
      """Extract cost from operation result tuple based on operation type."""
      if operation in ('crash', 'fix', 'verify'):
          idx = 4
      elif operation == 'generate':
          idx = 2
      else:  # example, test, test_extend, auto-deps
          idx = 1
      if len(result) > idx:
          val = result[idx]
          if isinstance(val, (int, float)) and not isinstance(val, bool):
              return float(val)
      return 0.0
  
  def _extract_model_from_result(operation: str, result: tuple) -> str:
      """Extract model name from operation result tuple based on operation type."""
      if operation in ('crash', 'fix', 'verify'):
          idx = 5
      elif operation == 'generate':
          idx = 3
      else:  # example, test, test_extend, auto-deps
          idx = 2
      if len(result) > idx and isinstance(result[idx], str):
          return result[idx]
      return 'unknown'

  Replace the inline extraction at lines 1857-1858 and 1889-1890 with calls to these helpers. Also fix operation_log.py:339-340 — either call the same helpers (requires passing operation name) or add the not isinstance(val, bool) guard at minimum.

---

Proceeding to Step 5.5: Prompt Classification

[Serhan-Asad]

Step 5.5: Prompt Classification

Classification: Code Bug

DEFECT_TYPE: code

Analysis

The prompt specification for sync_orchestration correctly describes how to extract cost and model from operation results. The code does not follow this specification.

Evidence

• Prompt specifies (Return Value Parsing, sync_orchestration_python.prompt line 354):

  • "Tuple (3-element): Cost is second-to-last element, model is last element" — i.e., result[-2] for cost, result[-1] for model
  • For test operations specifically: use result[1] for cost, result[2] for model (not negative indexing)
  • The dependency <include> examples (code_generator_main_example.py, crash_main_example.py, fix_main_example.py, fix_verification_main_example.py) explicitly unpack the return tuples showing cost and model are always the last two elements

• Code implements (sync_orchestration.py:1886-1890):

  • For test/test_extend: result[1] for cost, result[2] for model ✓
  • For everything else: also result[1] for cost, result[2] for model ✗
  • The code hardcodes index [1] with the incorrect comment "Cost is always at index 1" — this contradicts the prompt's "second-to-last" guidance

• User expects (matches the prompt's intent):

  • generate (4-tuple (content, was_incremental, cost, model)): cost at [-2] = index 2, model at [-1] = index 3
  • crash/fix/verify (6-tuple (success, ..., cost, model)): cost at [-2] = index 4, model at [-1] = index 5
  • example (3-tuple (content, cost, model)): cost at [-2] = index 1, model at [-1] = index 2 ✓ (only case where result[1] happens to work)

Why not a prompt defect?

The prompt's "second-to-last element" pattern would produce correct results for all tuple formats (3, 4, and 6 elements) if implemented via negative indexing. Additionally, the prompt's <include> directives bring in operation examples that explicitly show the return tuple structures. The prompt specification is correct — the code simply doesn't follow it.

Note: The prompt could be improved by enumerating all tuple formats explicitly (4-element generate, 6-element crash/fix/verify) rather than only mentioning "3-element" tuples. But being incomplete is not the same as being wrong — the existing guidance generalizes correctly.

Conclusion

The code does not correctly implement the prompt specification. The prompt says "second-to-last element" for cost extraction from tuples; the code uses result[1] which is only equivalent to result[-2] for 3-element tuples. Proceeding with test generation.

---

Proceeding to Step 6: Test Plan

[Serhan-Asad]

Step 6: Test Plan

Existing Test Coverage

• tests/test_e2e_issue_508_budget_test_cost.py — Tests cost extraction for test/test_extend (4-tuple, cost at index 1). Has a regression guard for generate using result[-2], but the code has since regressed to result[1] (commit 35c99dc09), so that guard now silently passes with the wrong logic.
• tests/test_e2e_issue_508_sync_budget_tracking.py — E2E test running real sync_orchestration() with mocked operations. Tests test 4-tuple cost tracking and budget enforcement, but only covers test and generate — no coverage for crash/fix/verify 6-tuples.
• tests/test_sync_orchestration.py (5,058 lines) — Comprehensive sync tests but does not assert on per-operation cost values or model name extraction.
• tests/test_track_cost.py — Tests the @track_cost decorator (CSV writing), not the sync loop extraction.

Gaps:

1. No tests for crash/fix/verify 6-tuple cost extraction (cost at index 4, model at index 5)
2. No test that generate 4-tuple cost is at index 2 (not index 1) — the existing Budget tracker drops test/test_extend costs due to wrong tuple index #508 test uses result[-2] which accidentally works for 4-tuples
3. No test for the bool ⊂ int edge case (was_incremental=False passes isinstance check)
4. No tests for operation_log.py:339-340 (same bug in the log_operation() decorator)
5. No tests for pin_example_hack.py:1575,1593 (same bug in the pin example sync loop)

Proposed Tests

Test 1: test_generate_cost_at_index_2_not_index_1

• Input: generate result = ("generated code", False, 0.0425, "claude-sonnet-4-6") — 4-tuple where result[1] is was_incremental=False
• Expected: Cost extracted = 0.0425 (from index 2), model = "claude-sonnet-4-6" (from index 3)
• Actual (before fix): Cost = 0 (bool False → int 0), model = "unknown" (index 2 is 0.0425 float, fails isinstance(str) check)

Test 2: test_generate_bool_true_cost_is_not_1

• Input: generate result = ("generated code", True, 0.0888, "gpt-4o") — was_incremental=True
• Expected: Cost = 0.0888, not 1 (the int value of True)
• Actual (before fix): Cost = 1 (True → int 1, passes isinstance check)

Test 3: test_crash_cost_at_index_4

• Input: crash result = (True, "final_code", "final_program", 2, 0.1234, "claude-sonnet-4-6") — 6-tuple
• Expected: Cost = 0.1234 (index 4), model = "claude-sonnet-4-6" (index 5)
• Actual (before fix): Cost = 0.0 (result[1] is "final_code" string, fails isinstance → defaults to 0.0), model = "final_program" (wrong string)

Test 4: test_fix_cost_at_index_4

• Input: fix result = (True, "fixed_test_code", "fixed_code", 3, 0.2567, "gpt-4o-mini") — 6-tuple
• Expected: Cost = 0.2567 (index 4), model = "gpt-4o-mini" (index 5)
• Actual (before fix): Cost = 0.0, model = "fixed_code" (index 2, wrong string)

Test 5: test_verify_cost_at_index_4

• Input: verify result = (True, "final_program", "final_code", 1, 0.0999, "claude-sonnet-4-6") — 6-tuple
• Expected: Cost = 0.0999 (index 4), model = "claude-sonnet-4-6" (index 5)
• Actual (before fix): Cost = 0.0, model = "final_code" (index 2)

Test 6: test_example_cost_still_at_index_1 (regression guard)

• Input: example result = ("context content", 0.0050, "gpt-4o-mini") — 3-tuple
• Expected: Cost = 0.0050 (index 1), model = "gpt-4o-mini" (index 2)
• Purpose: Ensure the fix doesn't break operations that correctly use index 1

Test 7: test_test_cost_still_at_index_1 (regression guard)

• Input: test result = ("test content", 0.0078, "gpt-4o-mini", True) — 4-tuple
• Expected: Cost = 0.0078 (index 1), model = "gpt-4o-mini" (index 2)
• Purpose: Ensure Budget tracker drops test/test_extend costs due to wrong tuple index #508's fix is preserved

Test 8: test_bool_excluded_from_isinstance_check

• Input: Values False, True, 0.5, 42, "string"
• Expected: False and True are NOT valid costs (bool excluded), 0.5 and 42 ARE valid costs, "string" is NOT
• Purpose: Directly test that isinstance(val, (int, float)) and not isinstance(val, bool) logic works

Test 9: test_budget_accumulation_across_all_operations

• Input: Sequence: generate($0.04) → example($0.005) → crash($0.12) → fix($0.25) → verify($0.10) → test($0.008)
• Expected: Total cost = $0.523 (sum of all)
• Actual (before fix): Total cost ≈ $0.013 (only example + test costs counted correctly)
• Purpose: End-to-end budget accumulation test across all operation types

Test 10: test_operation_log_decorator_cost_extraction

• Input: Same tuples as Tests 1/3/4/5 passed through operation_log.py:339-340 logic
• Expected: Correct cost and model extracted per operation
• Purpose: Cover the identical bug in the log_operation() decorator

Test Location

• File: tests/test_e2e_issue_555_cost_extraction_wrong_index.py (new file)
• Framework: pytest (no markers needed — all tests are pure unit tests with no API calls)
• Naming convention: Follows existing test_e2e_issue_NNN_description.py pattern (e.g., Budget tracker drops test/test_extend costs due to wrong tuple index #508, Circular include cycle detection missing in non-recursive preprocess path #553)

Test Structure

class TestCostExtractionPerOperation:
    """Tests 1-5: Each operation extracts cost/model from correct tuple index."""
    # Helper: _extract_cost(operation, result) and _extract_model(operation, result)
    # These call the ACTUAL production helper functions once they exist,
    # or replicate the production logic for failing-first tests.

class TestBoolExcludedFromCostCheck:
    """Test 8: bool values must not be treated as valid costs."""

class TestRegressionGuards:
    """Tests 6-7: Operations that already work (example, test, test_extend)."""

class TestBudgetAccumulationAllOps:
    """Test 9: E2E budget tracking across all 6 operation types."""

class TestOperationLogDecoratorCost:
    """Test 10: operation_log.py has the same bug."""

Notes

• Failing-first approach: Tests 1-5, 8, 9, 10 should FAIL against current code, confirming the bug. Tests 6-7 should PASS (regression guards).
• No mocking of extraction logic: Tests should exercise the actual production code path, not a replicated helper. For unit tests (1-8, 10), call the helper functions directly. For E2E test (9), use the _run_sync() pattern from test_e2e_issue_508_sync_budget_tracking.py.
• Three bug sites: sync_orchestration.py (line 1858 + lines 1889-1890), operation_log.py (lines 339-340), pin_example_hack.py (lines 1575, 1593). Tests primarily target sync_orchestration; operation_log gets Test 10; pin_example_hack is a clone and shares the same fix pattern.
• CI safe: All tests use mocked data, no @pytest.mark.e2e or @pytest.mark.integration needed. Will run in pytest tests/ -m "not integration and not e2e and not real" --timeout=60.

---

Proceeding to Step 7: Generate Test

[Serhan-Asad]

Step 7: Generated Test

Test File

tests/test_e2e_issue_555_cost_extraction_wrong_index.py

Test Results: 13 FAILED, 3 passed

All 13 bug-detecting tests fail on the current code. The 3 regression guards (example, test, test_extend) pass correctly.

Test	Status	What it catches
test_generate_cost_extraction	FAILED	Budget reads was_incremental=False as cost 0 instead of actual cost at [2]
test_crash_cost_extraction	FAILED	Budget reads string at [1] → defaults to 0.0 instead of cost at [4]
test_fix_cost_extraction	FAILED	Budget reads string at [1] → defaults to 0.0 instead of cost at [4]
test_verify_cost_extraction	FAILED	Budget reads string at [1] → defaults to 0.0 instead of cost at [4]
test_generate_logging_cost_and_model	FAILED	Log reads False as cost, 0.0421 (float) as model → both wrong
test_crash_logging_cost_and_model	FAILED	Log reads wrong string as model, cost defaults to 0.0
test_fix_logging_cost_and_model	FAILED	Log reads wrong string as model, cost defaults to 0.0
test_verify_logging_cost_and_model	FAILED	Log reads wrong string as model, cost defaults to 0.0
test_bool_false_silently_accepted_as_cost	FAILED	isinstance(False, (int, float)) returns True — bool⊂int quirk
test_bool_true_misread_as_cost_one_dollar	FAILED	was_incremental=True recorded as cost $1.00 instead of actual cost
test_example_cost_extraction	PASSED	Regression guard — example works correctly
test_test_cost_extraction	PASSED	Regression guard — test works correctly
test_test_extend_cost_extraction	PASSED	Regression guard — test_extend works correctly
test_total_budget_with_all_operations	FAILED	Total cost $0.03 instead of $0.23 — 87% of spend is invisible
test_generate_decorator_extraction	FAILED	operation_log.py decorator has identical bug
test_fix_decorator_extraction	FAILED	operation_log.py decorator has identical bug

Key Failure Messages

Budget accumulation (line 1858):

• generate: assert False == 0.0421 — bool False treated as cost
• crash/fix/verify: assert 0.0 == X — string at [1] fails isinstance, defaults to 0.0

Logging (lines 1889-1890):

• generate: cost=False, model='unknown' (float not a string)
• crash: cost=0.0, model='program content' (wrong string from [2])

E2E budget: $0.03 instead of $0.23 — only example ($0.01) and test ($0.02) costs tracked

Test Code

6 test classes, 16 test cases total:

• TestCostExtractionBudgetAccumulation — 4 tests for line 1858 bug
• TestCostExtractionLogging — 4 tests for lines 1889-1890 bug
• TestBoolSubclassesIntQuirk — 2 tests for bool⊂int quirk
• TestRegressionGuards — 3 tests ensuring example/test/test_extend still work
• TestBudgetAccumulationAllOperations — 1 E2E test showing 87% cost loss
• TestOperationLogDecoratorBug — 2 tests for operation_log.py:339-340

Running the Test

pytest tests/test_e2e_issue_555_cost_extraction_wrong_index.py -v --timeout=60

---

Proceeding to Step 8: Verification

[Serhan-Asad]

Step 8: Verification Complete

Test Execution

16 items collected

FAILED test_generate_cost_extraction - generate cost should be 0.0421 (index 2) but got False. result[1] = False (was_incremental) was read as cost due to bool⊂int.
FAILED test_crash_cost_extraction - crash cost should be 0.0315 (index 4) but got 0.0. result[1] = 'code content' (final_code string) is not a number.
FAILED test_fix_cost_extraction - fix cost should be 0.0892 (index 4) but got 0.0. result[1] = 'fixed test code' (fixed_unit_test string) is not a number.
FAILED test_verify_cost_extraction - verify cost should be 0.0267 (index 4) but got 0.0. result[1] = 'program content' (final_program string) is not a number.
FAILED test_generate_logging_cost_and_model - generate log cost should be 0.0421 but got False. result[1] = False (was_incremental bool) was misread as cost.
FAILED test_crash_logging_cost_and_model - crash log cost should be 0.0315 but got 0.0.
FAILED test_fix_logging_cost_and_model - fix log cost should be 0.0892 but got 0.0.
FAILED test_verify_logging_cost_and_model - verify log cost should be 0.0267 but got 0.0.
FAILED test_bool_false_silently_accepted_as_cost - Cost = False is a bool (was_incremental), not a real cost value. isinstance(False, (int, float)) = True because bool subclasses int.
FAILED test_bool_true_misread_as_cost_one_dollar - Cost should be 0.0053 but got True (True → int 1). was_incremental=True was interpreted as cost=$1.00.
PASSED test_example_cost_extraction ✅ (regression guard)
PASSED test_test_cost_extraction ✅ (regression guard)
PASSED test_test_extend_cost_extraction ✅ (regression guard)
FAILED test_total_budget_with_all_operations - Total cost should be $0.23 but got $0.0300. generate/crash/fix/verify costs are lost due to wrong tuple index.
FAILED test_generate_decorator_extraction - Decorator cost for generate should be 0.0421 but got 0.0. result[1] = False (bool) was cast to float.
FAILED test_fix_decorator_extraction - Decorator cost for fix should be 0.0892 but got 0.0.

========================= 13 failed, 3 passed in 0.36s =========================

Verification Status

PASS: Test correctly detects the bug

All 13 failing tests fail for the exact bug reason — wrong tuple index for cost/model extraction. No unrelated failures. The 3 regression guards (example, test, test_extend) pass correctly, confirming those operations are unaffected.

Failure Breakdown

Test Class	Tests	Failures	Reason
Budget Accumulation (line 1858)	4	4	result[1] reads wrong value for generate/crash/fix/verify
Logging Extraction (lines 1889-1890)	4	4	Same wrong index + model also wrong
Bool⊂Int Quirk	2	2	isinstance(False, (int, float)) passes silently
Regression Guards	3	0	example/test/test_extend already correct ✅
E2E Budget Total	1	1	Only $0.03 of $0.23 tracked (87% loss)
operation_log.py Decorator	2	2	Same hardcoded index bug in decorator

Summary

Step	Result
Duplicate Check	✅ No duplicates (closest: #508, already fixed for test/test_extend only)
Documentation	✅ Confirmed bug — 4 of 6 operations affected
Triage	✅ Proceed — exact locations, return formats, and root cause identified
Reproduction	✅ Reproduced — all 4 affected operations confirmed
Root Cause	✅ Three causes: uniform index assumption, bool⊂int quirk, regression in commit 35c99dc09
Prompt Classification	✅ Code bug — prompts already specify correct tuple positions
Test Plan	✅ 16 tests across 5 classes covering all 3 bug locations
Test Generation	✅ 16 tests written — 13 detect bug, 3 regression guards
Verification	✅ All 13 bug-detecting tests fail for the correct reason

Bug Details

• Location: pdd/sync_orchestration.py:1857-1890 and pdd/operation_log.py:339-340
• Root Cause: Cost/model always read from result[1]/result[2] regardless of operation, but generate returns cost at [2] and crash/fix/verify return cost at [4]. The isinstance guard fails to catch bools because bool subclasses int in Python.
• Test File: tests/test_e2e_issue_555_cost_extraction_wrong_index.py
• Impact: 87% of actual LLM costs are invisible to the budget system ($0.03 tracked vs $0.23 actual)

Next Steps

1. Fix the bug at sync_orchestration.py:1857-1890 and operation_log.py:339-340
2. Create _extract_cost_from_result(operation, result) and _extract_model_from_result(operation, result) helpers
3. Run the 16 tests to confirm the fix (13 should flip to PASS, 3 should stay PASS)
4. Run full test suite to check for regressions
5. Submit PR with fix and test

---

Proceeding to Step 9: E2E Tests

[Serhan-Asad]

Step 9: E2E Test

Environment Discovery

• Repository language: Python
• Test framework: pytest 8.3.5 with pytest-xdist, pytest-timeout, pytest-asyncio
• Existing E2E tests found: Yes — 43 files matching tests/test_e2e_issue_*.py pattern
• Test configuration: pytest.ini and pyproject.toml (markers: e2e, integration, slow, real)
• Fixtures/helpers used: set_pdd_path (autouse), sync_workspace (tmp_path files), SyncDecision, RunReport
• Environment variables required: PDD_PATH (set via monkeypatch to pdd package dir)
• Auth/credentials: None required — all LLM calls mocked

E2E Test File

tests/test_e2e_issue_555_sync_cost_extraction.py

Test Type

Integration — calls the real sync_orchestration() function with mocked operation functions

What This Test Verifies

When a user runs pdd sync, the sync loop extracts cost and model from each operation's return tuple to:

1. Accumulate budget (current_cost_ref[0]) for budget enforcement
2. Log cost/model to _sync.log for user inspection
3. Track via decorator (log_operation in operation_log.py)

These tests verify the real extraction logic at lines 1857-1858 and 1889-1890 of sync_orchestration.py, plus the decorator at operation_log.py:338-340.

E2E Test Approach

Follows the established pattern from test_e2e_issue_508_sync_budget_tracking.py:

• Calls real sync_orchestration() function (not replicating buggy logic inline)
• Mocks operation functions (code_generator_main, crash_main, etc.) to return realistic tuples
• Mocks sync_determine_operation to control which operations run
• Mocks crash/verify pre-processing (read_run_report, _try_auto_fix_*) so crash_main is actually called
• Verifies total_cost in returned dict and update_log_entry call arguments

Test Results

10 FAILED, 3 PASSED — all as expected.

#	Test	Status	Bug Evidence
1	test_generate_cost_in_total	FAIL	total_cost=$0.00 — result[1]=False (bool→0)
2	test_generate_cost_logged_correctly	FAIL	cost=False logged instead of 0.0421
3	test_generate_bool_true_not_treated_as_cost	FAIL	total_cost=$1.00 — True→int(1)
4	test_crash_cost_in_total	FAIL	total_cost=$0.00 — result[1]='fixed code' (str) fails isinstance
5	test_crash_model_logged_correctly	FAIL	model='fixed program' — result[2] is wrong string
6	test_fix_cost_in_total	FAIL	total_cost=$0.00 — result[1]='fixed test code' fails isinstance
7	test_verify_cost_in_total	FAIL	total_cost=$0.00 — result[1]='program content' fails isinstance
8	test_multi_operation_budget_accumulation	FAIL	$0.03 of $0.08 tracked (62.5% cost loss)
9	test_decorator_extracts_generate_cost	FAIL	cost=0.0 — float(False)=0.0 in decorator
10	test_decorator_extracts_fix_cost	FAIL	cost=0.0 — str fails isinstance in decorator
11	test_example_cost_in_total	PASS	Regression guard — cost at index 1 works
12	test_test_cost_in_total	PASS	Regression guard — cost at index 1 works
13	test_budget_enforcement_with_generate_cost	PASS	Budget enforcement works when cost is tracked

Key Difference from Unit Tests

The Step 7 unit tests (test_e2e_issue_555_cost_extraction_wrong_index.py) replicate the buggy extraction logic inline to prove it's wrong. These E2E tests call the real sync_orchestration() function and verify the bug manifests in the actual code path users hit when running pdd sync.

Test Execution Output

FAILED test_generate_cost_in_total - assert 0.0 >= 0.0421
FAILED test_generate_cost_logged_correctly - assert False == 0.0421 ± 4.2e-08
FAILED test_generate_bool_true_not_treated_as_cost - assert 1.0 == 0.0053 ± 1.0e-02
FAILED test_crash_cost_in_total - assert 0.0 >= 0.0315
FAILED test_crash_model_logged_correctly - assert 'fixed program' == 'gpt-4o-mini'
FAILED test_fix_cost_in_total - assert 0.0 >= 0.0892
FAILED test_verify_cost_in_total - assert 0.0 >= 0.0267
FAILED test_multi_operation_budget_accumulation - assert 0.03 == 0.08 ± 5.0e-03
FAILED test_decorator_extracts_generate_cost - assert 0.0 == 0.0421 ± 4.2e-08
FAILED test_decorator_extracts_fix_cost - assert 0.0 == 0.0892 ± 8.9e-08
PASSED test_example_cost_in_total
PASSED test_test_cost_in_total
PASSED test_budget_enforcement_with_generate_cost
========================= 10 failed, 3 passed in 3.41s =========================

E2E Test Status

PASS: E2E test correctly detects the bug across all 3 affected code paths

---

Proceeding to Step 10: Create Draft PR

[Serhan-Asad]

Step 10: Draft PR Created

Pull Request

PR #556: test: add failing tests for cost extraction wrong index (#555)

Branch

fix/issue-555

What's Included

• Failing unit tests at tests/test_e2e_issue_555_cost_extraction_wrong_index.py (16 tests — 13 fail, 3 pass)
• Failing E2E tests at tests/test_e2e_issue_555_sync_cost_extraction.py (13 tests — 10 fail, 3 pass)
• Commits: 1

Next Steps for Maintainers

1. Review the failing tests to understand the expected behavior
2. Implement the fix at the identified locations (sync_orchestration.py:1857-1890, operation_log.py:339-340)
3. Verify all 29 tests pass with your fix
4. Run full test suite to check for regressions
5. Mark the PR as ready for review

PDD Fix Command

To auto-fix this bug using PDD:

pdd fix https://github.com/promptdriven/pdd/issues/555

Tip: Use --protect-tests if the tests are known to be correct, or --max-cycles N to limit fix attempts.

---

Investigation complete. A draft PR with failing tests has been created and linked to this issue.

[Serhan-Asad]

Step 2: E2E Test Check (Cycle 1)

Status: 23 Tests Failing

E2E Tests Run

Unit tests (tests/test_e2e_issue_555_cost_extraction_wrong_index.py — 16 tests):

• TestCostExtractionBudgetAccumulation::test_generate_cost_extraction ✗
• TestCostExtractionBudgetAccumulation::test_crash_cost_extraction ✗
• TestCostExtractionBudgetAccumulation::test_fix_cost_extraction ✗
• TestCostExtractionBudgetAccumulation::test_verify_cost_extraction ✗
• TestCostExtractionLogging::test_generate_logging_cost_and_model ✗
• TestCostExtractionLogging::test_crash_logging_cost_and_model ✗
• TestCostExtractionLogging::test_fix_logging_cost_and_model ✗
• TestCostExtractionLogging::test_verify_logging_cost_and_model ✗
• TestBoolSubclassesIntQuirk::test_bool_false_silently_accepted_as_cost ✗
• TestBoolSubclassesIntQuirk::test_bool_true_misread_as_cost_one_dollar ✗
• TestRegressionGuards::test_example_cost_extraction ✓
• TestRegressionGuards::test_test_cost_extraction ✓
• TestRegressionGuards::test_test_extend_cost_extraction ✓
• TestBudgetAccumulationAllOperations::test_total_budget_with_all_operations ✗
• TestOperationLogDecoratorBug::test_generate_decorator_extraction ✗
• TestOperationLogDecoratorBug::test_fix_decorator_extraction ✗

E2E tests (tests/test_e2e_issue_555_sync_cost_extraction.py — 13 tests):

• TestE2ESyncGenerateCostExtraction::test_generate_cost_in_total ✗
• TestE2ESyncGenerateCostExtraction::test_generate_cost_logged_correctly ✗
• TestE2ESyncGenerateCostExtraction::test_generate_bool_true_not_treated_as_cost ✗
• TestE2ESyncCrashCostExtraction::test_crash_cost_in_total ✗
• TestE2ESyncCrashCostExtraction::test_crash_model_logged_correctly ✗
• TestE2ESyncFixCostExtraction::test_fix_cost_in_total ✗
• TestE2ESyncVerifyCostExtraction::test_verify_cost_in_total ✗
• TestE2EBudgetAccumulationAllOperations::test_multi_operation_budget_accumulation ✗
• TestE2EBudgetAccumulationAllOperations::test_budget_enforcement_with_generate_cost ✓
• TestE2EOperationLogDecorator::test_decorator_extracts_generate_cost ✗
• TestE2EOperationLogDecorator::test_decorator_extracts_fix_cost ✗
• TestE2ERegressionGuards::test_example_cost_in_total ✓
• TestE2ERegressionGuards::test_test_cost_in_total ✓

Failures

All 23 failures confirm the bug described in the issue — cost/model extraction reads wrong tuple indices.

Budget accumulation failures (line 1858)

• generate: result[1] = False (was_incremental bool) → cost recorded as 0 instead of 0.0421 at index 2. When was_incremental=True, cost becomes $1.00 instead of $0.0053.
• crash: result[1] = 'fixed code' (str) → isinstance fails → cost defaults to 0.0 instead of 0.0315 at index 4
• fix: result[1] = 'fixed test code' (str) → cost defaults to 0.0 instead of 0.0892 at index 4
• verify: result[1] = 'program content' (str) → cost defaults to 0.0 instead of 0.0267 at index 4

Logging failures (lines 1889-1890)

• generate: cost=False, model='unknown' (float at index 2 fails str check)
• crash: cost=0.0, model='fixed program' (wrong string from index 2 instead of 'gpt-4o-mini' at index 5)

E2E budget accumulation

• Multi-op test: $0.03 tracked instead of $0.08 — 62.5% of costs invisible

operation_log.py decorator (lines 339-340)

• Same hardcoded result[1]/result[2] pattern — identical extraction bug

Summary

23 of 29 tests fail, all detecting the same root cause: cost/model always read from result[1]/result[2] regardless of operation type. 6 regression guards pass (example, test, test_extend use index 1/2 correctly).

Files to fix:

• pdd/sync_orchestration.py:1857-1890 — budget accumulation + log extraction
• pdd/operation_log.py:339-340 — decorator extraction

---

Proceeding to Step 3: Root Cause Analysis