|
| 1 | +# Drizzle ORM beta.16 updates |
| 2 | + |
| 3 | +We've fixed a regression in migrations introduced in `beta.13` that persisted through `beta.15`, and used the opportunity to significantly improve the entire migration infrastructure going forward. |
| 4 | + |
| 5 | +After receiving github issue, we focused on not just fixing the symptoms but rethinking how migrations are tracked, validated, and applied. As we always do with Drizzle we went deeper and redesigned the underlying architecture so it can be stable for all future changes, additions, and edge cases |
| 6 | + |
| 7 | +We went through how we check migrations, how we store them, and how we help developers keep their migration flow reliable. After several iterations of rewriting the `migrate` function in both ORM and Kit, adding migration table versioning, and building a new commutativity check system: here's what changed and why. |
| 8 | + |
| 9 | +## What happened in beta.12–beta.15 and what caused the issue |
| 10 | + |
| 11 | +On `latest` (pre-beta) versions, the migrations folder used a journal-based structure. A `meta/_journal.json` file stored a timestamp for each migration in **milliseconds**. That same millis value was stored in the database's `created_at` column and used to determine which migrations had been applied. Because the journal enforced ordering, we could simply fetch the last applied migration from the database and apply everything after it. |
| 12 | + |
| 13 | +With the new v3 folder structure (introduced in beta), each migration lives in its own folder named `<YYYYMMDDHHmmss>_<name>`. This format only has **second** precision. The new structure also intentionally allows out-of-order migrations (as it should for team workflows), so we switched to reading *all* migrations from the database and comparing against all local migrations. |
| 14 | + |
| 15 | +The problem appeared after `drizzle-kit up` converted the old journal structure to v3 folders. It mapped millis timestamps to the `YYYYMMDDHHmmss_name` format, stripping away the millisecond precision. So when the new migration checker compared what was in the database (millis) to what was on disk (seconds), nothing matched causing migrations to be re-applied on every run. |
| 16 | + |
| 17 | +**This only affected beta users who upgraded from the journal-based format. Users on `latest` were not impacted.** |
| 18 | + |
| 19 | +## How we fixed it |
| 20 | + |
| 21 | +### 1. Versioned migration table |
| 22 | + |
| 23 | +We introduced an internal version number for the migrations table schema. On `beta.16`, the table will automatically upgrade itself from `version_0` to `version_1` the first time you run `migrate`. |
| 24 | + |
| 25 | +**Version 0** (old schema): |
| 26 | +| Column | Type | |
| 27 | +|---|---| |
| 28 | +| `id` | serial | |
| 29 | +| `hash` | text | |
| 30 | +| `created_at` | bigint (millis) | |
| 31 | + |
| 32 | +**Version 1** (new schema): |
| 33 | +| Column | Type | | |
| 34 | +|---|---| ---| |
| 35 | +| `id` | serial | |
| 36 | +| `hash` | text | |
| 37 | +| `created_at` | bigint | (legacy) | |
| 38 | +| `name` | text | |
| 39 | +| `applied_at` | timestamp | |
| 40 | +| `version` | integer | |
| 41 | + |
| 42 | +The `name` column stores the full folder name of the migration (e.g. `20250220153045_brave_wolverine`). The `applied_at` column records when the migration was actually executed. For pre-existing migrations that were backfilled during upgrade, `applied_at` is set to `NULL` to distinguish them from newly applied ones. |
| 43 | + |
| 44 | +This versioning system means we can add more fields in the future (like migration state for rollbacks) without any disruption for developers |
| 45 | + |
| 46 | +### 2. Matching by folder name instead of timestamps |
| 47 | + |
| 48 | +All migrations are now checked against the **full folder name**: the combination of a 14-digit UTC timestamp and a name suffix (random or custom). Even if two migrations are generated within the same second, the name suffix guarantees uniqueness. |
| 49 | + |
| 50 | +The detection logic is simple: build a set of `name` values from the database, filter local migrations whose name isn't in that set, and apply those. No more timestamp arithmetic, no more precision mismatches. |
| 51 | + |
| 52 | +### 3. Automatic upgrade with smart backfilling |
| 53 | + |
| 54 | +When `beta.16` detects a `version_0` table, it adds the new columns and backfills the `name` for each existing row using a multi-step matching strategy: |
| 55 | + |
| 56 | +1. **Millis match**: truncate the stored millis to seconds and match against local migration folder timestamps |
| 57 | +2. **Hash tiebreaker**: if multiple migrations share the same second, use the SQL hash to pick the right one |
| 58 | +3. **Hash-only fallback**: if millis matching fails entirely, fall back to matching by hash alone |
| 59 | + |
| 60 | +This means the upgrade handles all edge cases: normal single-developer projects, teams with closely-timed migrations, and even cases where the old journal data doesn't perfectly align with the new folder names. |
| 61 | + |
| 62 | +### 4. New commutativity checks (`drizzle-kit check`) |
| 63 | + |
| 64 | +When working in teams, multiple developers may generate migrations from the same base schema on different branches. These migrations can conflict in non-obvious ways, for example, two branches both adding a column to the same table, or one renaming a table that another is altering. |
| 65 | + |
| 66 | +We built a new `drizzle-kit check` command that detects these non-commutative migrations. It works by: |
| 67 | + |
| 68 | +- Building a DAG (directed acyclic graph) from snapshot `prevIds` to understand the branch structure |
| 69 | +- Finding fork points where branches diverged |
| 70 | +- Computing the DDL diff from the parent snapshot to each branch leaf |
| 71 | +- Checking for conflicting operations using a comprehensive footprint map that knows which DDL statement types can interfere with each other |
| 72 | + |
| 73 | +This is available for PostgreSQL and MySQL today. If conflicts are found, the report tells you exactly which migrations on which branches are incompatible and what statements are conflicting. |
| 74 | + |
| 75 | +The new folder structure also changed snapshot metadata from `prevId: string` to `prevIds: string[]`, enabling proper DAG representation of migration history across branches. |
| 76 | + |
| 77 | +## Upgrading to beta.16 |
| 78 | + |
| 79 | +The upgrade is automatic. When you run `migrate` for the first time on `beta.16`: |
| 80 | + |
| 81 | +1. Drizzle detects your migration table version |
| 82 | +2. If it's `version_0`, it adds the new columns and backfills `name` from your local migration files |
| 83 | +3. All future migrations are tracked by name |
| 84 | +4. No manual steps required |
| 85 | + |
| 86 | +If you're also upgrading your migration **folder structure** from the old journal format, run `drizzle-kit up` first to convert to the v3 folder layout, then `migrate` will handle the rest. |
| 87 | + |
| 88 | +## Upgrading to beta.16 is not fixing my problem |
| 89 | + |
| 90 | +If you're still hitting migration issues after upgrading, please reach out to us directly - drop a message in [Discord](https://discord.gg/7NKUQWP9c8) or open a [GitHub issue](https://github.com/drizzle-team/drizzle-orm/issues/new/choose) with your migration folder structure and the contents of your migrations table. We'll help you sort it out. |
0 commit comments