Limitations of the current migrations system

[diesieben07]

The current migration system fails to support the use case of renaming or altering fields properly. This is because migrations always run in the context of the current payload config instead of using the payload config before the migration was made.

Renaming a field

As soon as you rename the field in your configuration, there is no way to access its data anymore via the Payload API. As such your migration has no chance to rename the property, because it cannot even get at it.

Altering a field

As soon as you change the field in your configuration, Payload will try to interpret the old data in the database as if it was for the new field configuration. As such you might receive errors when trying to load the documents within your migration to update them, for example when changing a select field to a relation field.

Removing a field

The Payload API has no concept of unsetting a property. However, you can achieve the desired effect by essentially loading and then saving all documents:

payload.update({
  collection: 'my-collection',
  where: {}, /* update all documents */
  data: {} /* don't change anything */
});

Solutions

In a perfect world, this would be solved by a system similar to Django's migrations. The gist of this is that not only the database is migrated, but also the Payload configuration. As such, while executing migration N, the Payload config would be the result of applying Migration 1 through N-1. The requirement for this however is that all migrations know how to migrate the config forward. Django solves this by providing all essential building blocks, i.e. "Add a field", "Remove a field", etc.

There is a simpler solution however, which is to expose an API similar to the following (hand waving ahead):

async function doMigration(payload: Payload, migrations: PayloadMigrations, down: boolean): Promise<void> {
  await migrations.removeField({
    collection: 'myCollection',
    field: 'whatever'
  });

  await migrations.alterField({
    collection: 'myCollection',
    before: {
      // old field configuration here
    },
    after: {
      // new field configuration here
    },
    transform: (oldValue) => computeNewValue(oldValue)
  });
}

removeField is fairly obvious (although I am sure there are details to be considered, for example regarding group and similar fields).
alterField essentially provides inline configurations for the field. Payload would then look only at this field and interpret it according to the given inline configuration (doing minimal checks, e.g. not verifying that relations are valid, etc.). Then Payload would call the transform function to let the user control how to convert between the old and new value.

Alternatives

The current alternatives to this are to fall back to the underlying database driver and directly access MonogDB/Postgres. This is of course not ideal, as you need to consider the intricacies of versions, drafts, etc. etc.