Question: schema migration - durable entities

[robinfehr]

Hi,

I'm trying to wrap my head around the docs but don't yet understand how to approach schema migrations when we introduce a breaking change. For example, when having a state in v1 of an entity, we introduce a breaking change and need to deploy a v2 of it. How could we then migrate the state from the schema of v1 to the schema of v2? In a streaming world (event sourced with an event log), we would replay all recorded events over time and letting v2 rebuild the state with a new schema from scratch. What is the correct way to deal with that by design currently?

Thank you.

[olitomlinson]

@robinfehr

I would do this by putting all the migration logic in a method on the entity, and call this migration method at the start of every one of the existing entity interface methods.

public Task Foo() {
     MigrateToV1()
     // your normal 'Foo' entity code here
}

public Task<bool> Bar(string data) {
     MigrateToV1()
     // your normal 'Bar' entity code here
}

The migration method would first check a new version property (don’t forget to include this in the entity state) that is introduced in the new entity code.

[JsonProperty("version")]
public int Version {get; set; }

private void MigrateToV1() {
     if (Version < 1)
     {
           // upgrade your entity to V1
           
           Version = 1;
     }
}

If the version property is default, this entity hasn’t been upgrade yet, so run the necessary migration logic and set the version to something significant. I.e 1

You can then use the version property throughout your entity code to branch your logic where necessary if it can’t be upgraded.

---

Alternatively, if you didn't want to add it to each entity method, you could move the migration logic into an Migration class, pass the Migration class into the Entity using DI, and then invoke it, but I've not personally ever done this.

[JsonProperty("version")]
public int Version {get; set; }

public MyEntity(IDurableEntityContext ctx, IMigrateToV1 migrateV1, IMigrateToV2 migrateV2) {
     if (ctx.HasState) {
          if (Version < 1) { 
               migrateV1.Migrate(); 
               Version  = 1;
          }
          if (Version < 2) { 
               migrateV1.Migrate(); 
               Version  = 2;
          }
     }
     else {
          Version  = 2;
     }
}

Be careful about using HasState like this. There is a big assumption here that if your entity doesn't have any state it has just been activated for the first time.

---

Depending on your scenario, you may have entities that don’t get activated frequently and therefor don’t get migrated. It’s worth considering how long you are comfortable with old code cluttering the entity, waiting for all migrations to happen.

It might be worth developing a way to activate any dormant entities with some kind of noop method which can be called purely to get the migration code to run.

public void NoOp()) {
     MigrateToV1()
}

You can loop over all entities using this API and then invoke the noop using this API

Or even better, delete the entity completely if it is truly obsolete, but that’s entirely down to you to evaluate the suitability of that option.

---

It’s also worth considering that if you rollback your breaking change code for whatever reason, you will loose any new serialised data for newly introduced properties. So it’s worth considering a roll-forward plan if data loss is unacceptable.

[mpaul31]

i just ran into something similar recently but the problem we encountered was the migration also had to change how we identify our entities (EntityId). The way we are going to solve this problem is by creating an ingestion service that appends the external messages to an immutable log (document store).

I am most likely going to make this part of the "cold" path that would allow replays to rebuild our entities as opposed to a more "hot" path where we would tail the log and signal our entities to avoid the additional latency and complexity / overhead.