Model Cost Comparison

Model Cost Comparison: Evaluating An Agentic Database Migration

Everyone (and their grandmother) is talking about "agentic workflows" these days, but few people are talking about the actual bill. To see if autonomous agents are a viable strategy or just an expensive way to generate automated syntax errors, we benchmarked three model tiers inside VSCode's GitHub Copilot extension: Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8. We had a production database migration scheduled anyway. Because the target schemas were already well-defined, it was the perfect opportunity to use the task as a guinea pig to test our potential new robot colleagues.

To understand the landscape, we have to be honest: budget dictates architecture. An AI agent that writes flawless code is completely useless if its token consumption costs more than hiring an entire team of human consultants to do the work manually. We needed to know if the credit burn could actually translate to developer hours saved.

Surprisingly, as of June 2026, the budget option wins by a landslide. Claude Haiku 4.5 hits an incredible sweet spot between cost and execution. It handles the heavy lifting inside GitHub Copilot well enough that paying for the premium tiers feels like lighting money on fire. As a side note, it also handily embarrassed the OpenAI GPT models we tested, which managed to be both more expensive and remarkably unable to even complete the baseline task.

Table of Contents

• Background & Scope
• The Testing Environment
  • Measurement Criteria
• Quantitative Analysis: Credit and Cost Breakdown
• Deep Dive: Run-by-Run Technical Evaluation
  • Run 1: Claude Haiku 4.5
  • Run 2: Claude Sonnet 4.6
  • Run 3: Claude Opus 4.8
• Final Impressions & Recommendations

Background & Scope

So, what exactly was this "walk in the park" database migration? The goal was to rename the pies.system_record table to pies.asset to align with upcoming PIES architectural changes. On paper, this is a standard, average-Tuesday database administration task. In practice, it involves a cascade of dependent changes across both the schema and application layers:

• Rename the table and update all downstream references.
• Update all foreign key constraints.
• Rename any relevant triggers, indexes, and implicit sequences.
• Hunt down and refactor every repository, service, and controller referencing the old table name.
• Pray the TypeScript compiler gods are in a good mood.

Because the scope was tight and the blast radius was well-defined, it served as the ideal benchmark. A localized table rename should theoretically be a trivial task for an autonomous agent. However, because we are dealing with non-deterministic systems, the premier engineering rule applies: never trust, always verify.

The Testing Environment

We ran the benchmarks using VSCode and the GitHub Copilot extension. To give our potential agents a fighting chance, we threw our custom skill /scaffold-db-migration at it. The test bed was a standard PostgreSQL 18 instance running on bare metal, initialized to a completely pristine migration state. For every single run, we dropped the prompt into the VSCode Chat interface verbatim:

/scaffold-db-migration rename system_record table to asset. Include all affected model changes in repository, service and controller layers.

Once we hit enter, we contemplated our life decisions, let the agent do its thing, and tracked the credit bleeding. To ensure a completely fair fight, we instituted strict sandbox rules:

1. The database schema and repository files were fully reset to baseline before each model took its turn.
2. We refused to assist, nudge, or intervene when the agent started flailing.
3. The only human interaction allowed was clicking "Approve" when Copilot requested a terminal command execution, and we only vetted those for basic shell safety, not technical correctness.

Measurement Criteria

To evaluate whether our potential robot overlords actually did a good job, we judged them across two core competencies: Database Migration Crafting and Application Layer Refactoring.

Database Migrations: The Four Horsemen of Agentic Evaluation

Each agent was evaluated against four main criteria:

• Cost & Efficiency: The final, cold, hard credit bill required to complete the scope.
• Process Velocity & Autonomy: How often did it get stuck in a loop, repeat failing tool calls, or give up entirely?
• Instruction & Architectural Adherence: Did the down function perfectly invert the up function? Did it respect referential integrity sequencing? Did it use idiomatic Kysely schema builder methods, or did it lazy-out and wrap everything in raw sql strings?
• Technical Completeness: Did it rename dependent sequences, triggers, and indexes? Did it catch the self-referencing foreign keys in the record_linkage table and update both sides of the relationship? Did it preserve explicit cascading behaviors?

Application Layer Refactoring: Completeness and Correctness

• Layer Integration: How methodically did it handle the repository, service, and controller layers?
• Compilation & Tests: Does the codebase compile without TypeScript errors, and does it pass the unit test suite after refactoring?

Quantitative Analysis: Credit and Cost Breakdown

The following table summarizes how many credits each model burned through before it triumphantly declared itself "done" with the task. Because these were long-running skill invocations, the agent would periodically pause to ask for human permission to keep thinking. This behavior is captured in the multi-phase credit splits below.

Run ID	Model Tier	Operation / Phase Split (Credits)	Total Credit Cost
haiku	Claude Haiku 4.5	84.6 + 76.8 + 15.9	177.3
sonnet	Claude Sonnet 4.6	125.4 + 130.2	255.6
opus	Claude Opus 4.8	573.8 + 28.2	602.0

What is immediately obvious from the credit distributions is that GitHub Copilot's underlying token-metering model for these tiers is aggressively exponential. The jump from Haiku to Sonnet is a noticeable step up, but the leap to Opus is astronomical.

This stark contrast raises the critical engineering question: does the premium price tag of an Opus run translate to a proportionate increase in structural performance and architectural reliability? Or are we just lighting credits on fire to enjoy the prestige of running a heavyweight model that over-thinks a straightforward table rename?

Deep Dive: Run-by-Run Technical Evaluation

Run 1: Claude Haiku 4.5 (The Budget Multi-Phase Attempt)

• Raw Haiku patch.
• Score: 27 / 30 stars

Haiku Database Migrations

• Cost & Efficiency: ⭐⭐⭐⭐⭐. Highly cost-effective. At a total of 177.3 credits across three phases, this run demonstrated the most efficient credit burn, providing massive overall value.
• Process Velocity & Autonomy: ⭐⭐⭐⭐⭐. The final diff was cohesive and fully formed, suggesting the agent required minimal environment coaxing to reach the correct conclusion.
• Instruction & Architectural Adherence: ⭐⭐⭐. The migration correctly sequenced referential integrity and provided a solid semantic inverse in the rollback function. It lost points for relying heavily on raw SQL strings instead of idiomatic Kysely schema builder methods, though it did at least make an attempt.
• Technical Completeness: ⭐⭐⭐⭐⭐. The database changes handled dependent triggers and indexes flawlessly. Using explicit native rename commands safely preserved all existing cascading behaviors and correctly updated the self-referencing foreign keys.

Haiku Application Layer Refactoring

• Repository, Service, and Controller Layers: ⭐⭐⭐⭐. The agent methodically updated the application layers and correctly adjusted type definitions. However, it chose to maintain backwards compatibility by exporting legacy aliases for the controller and service functions instead of removing them entirely.
• Unit Tests and Functionality: ⭐⭐⭐⭐⭐. The test suite was refactored perfectly to match the new asset schema and mock data structures, resulting in a completely green test suite.

Haiku Observations

• Left legacy systemRecord.ts files intact but generated new asset.ts files with the correct updated code.
• Resolved all TypeScript compilation issues reasonably well.
• Ironically, it was the only agent that did not get stuck running our npm run codegen command (which had an underlying script misconfiguration). It trusted the code output as instructed and moved on.
• Followed the instruction skill ordering very well.
• Overall Impression: Highly positive. It did not go unnecessarily deep into hypothetical edge cases, but it executed the instructions verbatim and produced a technically competent result at a very reasonable cost.

Run 2: Claude Sonnet 4.6 (The Balanced Two-Phase Execution)

• Raw Sonnet patch.
• Score: 17 / 30 stars

Sonnet Database Migrations

• Cost & Efficiency: ⭐⭐⭐⭐⭐. A decent balance of cost and execution speed. The run consumed 255.6 credits (roughly 44% more expensive than Haiku), but achieved the final result in only two phases.
• Process Velocity & Autonomy: ⭐⭐⭐⭐. The final diff was reasonably well-formed. It required minimal human intervention, but it did trap itself in a few internal tooling loops before giving up on the terminal and focusing on the core files.
• Instruction & Architectural Adherence: ⭐⭐. The migration provided a semantic inverse in the rollback function, but it lost significant points by entirely abandoning idiomatic Kysely schema builder methods in favor of raw SQL strings.
• Technical Completeness: ⭐. A massive failure here. The agent completely missed updating dependent tables. It failed to rename the foreign key column system_record_id to asset_id in process_event, on_hold_event, and the complex self-referencing record_linkage table.

Sonnet Application Layer Refactoring

• Repository, Service, and Controller Layers: ⭐⭐⭐. The agent methodically updated the core entity from systemRecord to asset, but completely forgot to update the underlying systemRecordId property mapping in the dependent event repositories and services.
• Unit Tests and Functionality: ⭐⭐. While the test suite files were renamed, the application layer still attempts to query and mutate event records using the stale, dead systemRecordId property at runtime.

Sonnet Observations

• Attempted to move some systemRecord.ts files into asset.ts, but the execution was completely inconsistent.
• Resolved superficial TypeScript errors well enough, but left deep structural type mismatches untouched.
• Strangely liked to output thoughts about "Summoning Clippy" and "Mining Diamonds" while processing.
• Got stuck running our broken npm run codegen command 3 times before giving up and moving on.
• Overall Impression: Disappointing. Despite the bump in cost, it missed critical structural nuances, left a broken application layer, and completely ignored the Kysely syntax requirements.

Run 3: Claude Opus 4.8 (The Front-Loaded Heavyweight)

• Raw Opus patch.
• Score: 22 / 30 stars

Opus Database Migrations

• Cost & Efficiency: ⭐⭐⭐. Poor cost-efficiency. With a total credit burn of 602.0 credits, this run was over three times as expensive as Haiku, tanking its efficiency ROI.
• Process Velocity & Autonomy: ⭐⭐⭐. The final diff was cohesive, but the agent was heavily prone to chasing tooling rabbit holes instead of executing the core task.
• Instruction & Architectural Adherence: ⭐⭐⭐. The migration provided a flawless semantic inverse in the down function. It utilized idiomatic Kysely schema builder methods for table and column renames, while falling back to raw SQL exclusively for complex indexes and triggers.
• Technical Completeness: ⭐⭐⭐⭐. The agent correctly identified and mutated columns in dependent tables, successfully catching both sides of the self-referencing record_linkage table. It lost a point for failing to rename the actual foreign key constraint identifiers themselves, leaving legacy names rotting in the database schema.

Opus Application Layer Refactoring

• Repository, Service, and Controller Layers: ⭐⭐⭐⭐. The agent methodically updated the core entity and successfully cascaded the systemRecordId property changes to all dependent event layers. However, it relied on a series of slightly worrisome shell commands to move files around.
• Unit Tests and Functionality: ⭐⭐⭐⭐⭐. The test suite was comprehensively refactored to match the newly renamed files and properties, resulting in a clean runtime execution.

Opus Observations

• Correctly and cleanly moved systemRecord.ts files into asset.ts.
• Resolved all TypeScript compilation issues thoroughly.
• Got stuck running the broken npm run codegen command 7 consecutive times, only moving on after the developer manually declined to approve the command execution again.
• Overall Impression: Mediocre. While it was technically the most thorough at deep code refactoring, its immense credit burn, obsession with broken terminal scripts, and high thinking latency make it completely unviable for standard day-to-day engineering tasks.

Final Impressions & Recommendations

Based on these results, we can definitively conclude that Haiku represents the optimal efficiency frontier for structured repository refactoring. It completed the task with minimal budget footprint while producing a highly competent, working result. Sonnet, despite the premium tiering, performed worse across the board, and Opus proved to be a financial black hole for a task of this scope.

That being said, there is likely still a place for heavyweight premium models in highly complex edge cases. For open-ended, unstructured tasks that lack explicit, step-by-step instructions, the deeper reasoning capabilities of an Opus-tier model may be justified. However, for a database migration with concrete application refactoring, the ROI simply isn't there.

Going forward, I recommend defaulting to the Haiku tier for the majority of our agentic workflow automation. As token-metering scaling continues to impact operational budgets, optimization is key. Haiku sits perfectly in the sweet spot for execution... provided you don't expect it to think beyond the verbatim scope you provide. And frankly, given that we probably shouldn't be trusting AI agents to fold our laundry yet, a predictable, instruction-following script engine is exactly what we want in our daily pipelines.