Improving Storage Migration with Changes to map_new_from_slices, map_unpack_to_slice, and contracttype

[leighmcculloch]

Data migration in Soroban contracts is currently more difficult than it needs to be. When a contract's data structures evolve—such as adding new optional fields—developers face significant friction.

Consider a simple upgrade scenario: a contract stores DataV1 with fields {a, b}, and needs to upgrade to a new wasm where that data will have a new field c and become {a, b, c}.

Examples

Example 1: Adding an Optional Field

I expect a developer's natural assumption would be to try using an Option<T> field. It's easy to assume based off the behaviour of many other APIs that data without the field would deserialize with c = Option::None:

#[contracttype]
pub struct DataV1 { a: i64, b: i64 }

#[contracttype]
pub struct DataV2 { a: i64, b: i64, c: Option<i64> }

#[contracttype]
pub enum DataKey {
    Data(u32),  // keyed by id
}

// Developer expects this to work when reading old V1 data:
let data: DataV2 = env.storage().persistent().get(&DataKey::Data(id)).unwrap();
// Expected: c would be None for old entries
// Actual: TRAPS with Error(Object, UnexpectedSize)

This fails because the host validates field count and names before the SDK or the developer can handle any mismatch.

Example 2: Explicit Fallback Logic

Knowing the simple approach doesn't work, a developer might try explicit fallback with try_from_val. The name suggests you can try to convert from the val, and in the vast majority of implementations try_from_val results in a guest side error if it fails. However that's not the case with contracttype structs.

#[contracttype]
pub struct DataV2 { a: i64, b: i64, c: i64 }

let raw: Val = env.storage().persistent().get(&DataKey::Data(id)).unwrap();

// This also TRAPS instead of returning Err!
if let Ok(v2) = DataV2::try_from_val(&env, &raw) {
    v2
} else {
    // Never reached - host traps before try_from_val returns
    let v1 = DataV1::try_from_val(&env, &raw).unwrap();
    DataV2 { a: v1.a, b: v1.b, c: 0 }
}

The host traps with Error(Object, UnexpectedSize) before try_from_val can return an Err.

Current Workarounds

Workaround 1: Version Marker

The best solution to work around this that I'm aware of today is to use an explicit version number and check it before reading.

#[contracttype]
pub enum DataKey {
    DataVersion(u32),  // version marker keyed by id
    Data(u32),         // data keyed by id
}

fn read_data(env: &Env, id: u32) -> DataV2 {
    let version: u32 = env.storage().persistent().get(&DataKey::DataVersion(id)).unwrap_or(1);
    match version {
        1 => {
            let v1: DataV1 = env.storage().persistent().get(&DataKey::Data(id)).unwrap();
            DataV2 { a: v1.a, b: v1.b, c: None }
        }
        _ => env.storage().persistent().get(&DataKey::Data(id)).unwrap(),
    }
}

This works but discovering this pattern is non-obvious.

This can be retrofitted on to any existing storage because the program can assume that if the version doesn't exist that it is the first version of the data.

Workaround 2: Map Inspection

I have also seen LLMs suggest work arounds to inspect the map, either its length or the symbols in the map, both both approaches are either very verbose or error prone. For example:

// Read as raw Map to inspect structure
let raw: Map<Symbol, Val> = env.storage().persistent().get(&DataKey::Data(id)).unwrap();

if raw.len() == 3 {
    // V2 format
    env.storage().persistent().get(&DataKey::Data(id)).unwrap()
} else {
    // V1 format - read again and convert
    let v1: DataV1 = env.storage().persistent().get(&DataKey::Data(id)).unwrap();
    DataV2 { a: v1.a, b: v1.b, c: None }
}

Proposal

Introduce v2 versions of the map packing/unpacking host functions with the following semantics:

1. map_unpack_to_slice_v2: When reading maps:
   a. Unpack missing keys as Val::VOID values (which convert to None for Option<T> fields).
   b. Ignore keys not requested if they are in the map being unpacked.
   c. Return an Error instead of trapping on schema mismatches.

2. map_new_from_slices_v2: When writing maps, omit key-value pairs containing 'void' values, for the main purpose of symmetry with the unpack host fn, although as a bonus as an optimisation (originally proposed in https://github.com/orgs/stellar/discussions/1750).

These changes would support a developer migrating using patterns like:

#[contracttype]
pub struct DataV1 { a: i64, b: i64 }

#[contracttype]
pub struct DataV2 { a: i64, b: i64, c: Option<i64> }

#[contracttype]
pub enum DataKey {
    Data(u32),  // keyed by id
}

// Developer expects this to work when reading old V1 data:
let data: DataV2 = env.storage().persistent().get(&DataKey::Data(id)).unwrap();

Why This Must Be Solved at the Host Level

Handling schema mismatches guest-side is impractical because validating all the fields would require iterating over the map in full and nullify any benefit of using the map unpack host fn. While in some cases a developer might optimise and only check the field count that is an incomplete check in the cases that fields have been removed or renamed.

Alternatives

1. Improve Documentation

There exists very little documentation for how to do migrations on Soroban, so I've opened an issue to improve the documentation about migrations, and if everyone needing to do migrations read that documentation, then that might suffice:

• Add how-to guide for storage data migration stellar-docs#2228

Related

• Original discussion on this topic (optimization-focused): #1750
• Test vector demonstrating migration difficulty: stellar/rs-soroban-sdk#1715

[dmkozh]

This makes sense to me. The behavior might be slightly surprising if someone tried to inspect the map manually, but we don't expect this to be a common pattern (even though LLMs suggest this currently). But I wonder if we should name this more appropriately to the use case, like unpack_contract_type_to_slice, contract_type_new_from_slices. Contract type spec is already a de facto part of the protocol and I don't see much value in obfuscating these functions (unless we believe that there are use cases for them beyond contract types, that would make names confusing).

[leighmcculloch]

if someone tried to inspect the map manually, but we don't expect this to be a common pattern

Inspecting the maps is common and easily done through several tools, such as the stellar-cli, and the lab. But I don't think this will be surprising. Omitted fields often make APIs less clear as to the full shape of an object, but it is such a common practice in APIs to omit empty, null, or none fields so I think that's fine.

unpack_contract_type_to_slice, contract_type_new_from_slices

I think the map names are more suitable. These functions are for operating on host maps, and contract type's are just one user-land use case. Contract event data bodies are a second use case that sort of look like contract types but the structs don't actually work the same way on the guest side. Another use case that was imagined at one point was us adding bulk map pack and unpack operations accessible to users, although that never eventuated.

[dmkozh]

Inspecting the maps is common and easily done through several tools, such as the stellar-cli, and the lab

I mean inspecting/interacting with the maps in code. For off-chain uses that's fine (as these just look at the storage without making assumptions about the implementation). But in code if I put ((key, None)) into map, I would expect that map.has(key) == true, which is not going to be the case here. That's why I'm not sure if it's a good idea to expose these functions as generic helpers - I feel like it might lead to footguns if developers don't read the docs. Specialized helpers would likely reduce the risks. We can call this 'sparse_map' or 'lazy_map' to keep things relatively generic. Basically I'd like the name to imply that this does not simply put the provided key-val pairs into map.

[leighmcculloch]

Ah, yes that's a really great point.

sparse_map

👍🏻 Naming them sparse_map_new_from_slices and sparse_map_unpack_to_slice would better communicate the behaviour.

To tell the narrative better, the SDK could have a SparseMap type that parallels the behaviour of these functions. Another option is a new type of Host type called SparseMap, and these new functions would be for unpacking and packing SparseMaps. That would create a single vertical where these semantics are true rather than creating different semantics at different layers. Similar to how Address can be upgraded to MuxedAddress, we could make it so that SparseMap can be converted into a Map.

In practice would anything use this other Host type? Unclear, I don't think so, so probably not worth it. The question for me is, is this change in behaviour significant enough to warrant a type incompatibility where one is not the other?

[dmkozh]

I think introducing a new host type is a bit of an overkill: it's a lot of work on our side without a clear benefit, and also it wouldn't be compatible with any existing events. Unlike the MuxedAddress case where we conveniently could modify the underlying ScAddress of ScVal::Address, this would require an explicit ScVal variant. It also would make migration to the sparse map format harder for the existing contracts. Thus I think using the more clear host fn names in combination with the existing map type is a reasonable tradeoff.

I also agree that it seems unlikely that the new host type would be too useful outside the SDK machinery.

[leighmcculloch]

Agreed, I don't think the difference in behaviour is significant enough to warrant type incompatibility.

[leighmcculloch]

@khomiakmaxim posted at stellar/stellar-docs#2228 (comment) highlighting that the enum data value pattern provides another workaround as a way to introduce multiple versions of types without the need for a separate version storage. It requires thinking ahead to add that for every data value, but for anyone who has done that there's an avenue to adding other types. It's a bit cumbersome to use for every type since every time you use the value it has to be unpacked, but it would work.