Partial State Archival

[SirTyson]

For protocol 23, I'm proposing we implement "partial" state archival. More context here.

[MonsieurNicolas]

Do you have a proposed network settings vote proposal that would need to go out with this change?
without it, the bucketlist size allocated to Soroban is going to be way too big.

[SirTyson]

I think for this particular change, we might need a different mechanism for settings upgrade. Since this is the first protocol upgrade to change the semantic meaning of a network config, we should probably change the network setting as part of the protocol upgrade. If we waited and did a proposal after the upgrade, there would be a significant delay between protocol upgrade and settings change since validators can't arm a queue of upgrades. During this delay, read and write fees would basically be free, since the "BucketList size" would change from ~12 GB to a few hundred MBs. If we did a settings upgrade before the protocol upgrade, we could mitigate the runaway growth attack, but would DOS the network in the time period between settings upgrade and p23 upgrade.

[MonsieurNicolas]

I guess it depends on how the "Soroban State Size" is calculated at the upgrade boundary. I think this is not specified in the CAP

[MonsieurNicolas]

It looks like you'd want it to be accurate from the get go. Default could be current size * 10 (arbitrary but "sane" default) until a network setting vote is applied

[MonsieurNicolas]

protocol meeting comment -- it sounds like we can just have validators vote back to back (still needs to be evaluated risk wise)

[tomerweller]

Overall this is excellent. I'd love to find a more descriptive name for this though. I think that "partial" is not a great indicator for what it is going on here and "live state prioritization" deals with the implications of this change rather than the change itself.
I think the change itself boils down to "where" the archive is located so maybe something like "validator side state archive" or something like that. Wdyt @SirTyson ?

[SirTyson]

Overall this is excellent. I'd love to find a more descriptive name for this though. I think that "partial" is not a great indicator for what it is going on here and "live state prioritization" deals with the implications of this change rather than the change itself. I think the change itself boils down to "where" the archive is located so maybe something like "validator side state archive" or something like that. Wdyt @SirTyson ?

Hmm, maybe "Validator State Archive" and "Offloaded State Archive", "Historical State Archive"? Idk about the last one, might get confused with the other History Archive.

[JFWooten4]

How about "Ember State Archival?" – Like a fire that’s still warm but not actively burning.

[MonsieurNicolas]

Re: memory limits

In the CAP:

Given that there is a limit on the maximum number of in-memory entries, and given that there is a limit on the maximum size for each entry, it seems highly unlikely that an attacker would be able to DOS the network via in-memory entry copies.

I would like more discussion on this. I didn't realize at first that this limit was removed instead of just being set to some high number.

As the implementation is likely not 0-copy, we need to look at the "worst case scenario" of a well crated transaction set:
Max number of entries a transaction can read * largest ledger entry * Max number of transactions in a ledger.

As we're likely to increase all 3 of these over time, it seems that the actual memory utilization could skyrocket without limits:

• Max number of entries a transaction can read 100, likely to grow more (but maybe not that fast).
• largest ledger entry 256kb but likely to grow some more with more complex contracts.
• Max number of transactions in a ledger we'll get to 1000 rapidly, will grow well past that.

Multiply all those things already gives a 25GB memory spike, this would be on top of everything else already residing in memory.

To me the saving of not having this network setting at this stage doesn't seem to be worth the risk.

[dmkozh]

To me the saving of not having this network setting at this stage doesn't seem to be worth the risk.

While I agree with the sentiment, I think we need to keep in mind that we have a gap in metering right now (LE serialization) that we're just compensating for with a low limit. High enough limit (which we strive to achieve) without proper accounting will still lead to the DOS potential. So even without removing the limit I think it's worth focusing on improving this (and maybe optimizing it as well).

On the other hand, there is also a concern that has nothing to do with the execution itself: the more entries we have per-transaction, the higher the cost of building and validating parallel tx sets is. And I think that this is surprisingly harder to optimize than the execution itself. So I think at least per-transaction limit should stay. As for per-ledger limit, I think with some changes above it shouldn't be necessary, but if we find these hard to implement right now, we could keep it for now as well.

[SirTyson]

Idea 2: maintain in-memory Soroban data in serialized XDR form. Then there is no serialization necessary at all, and we can trivially and correctly charge the copies on C++ side when moving the data to Rust

I think I'm most in favor of this. I think it's pretty straight forward to implement from a cache perspective and reduces the amount of redundant work we charge invocations for. Early exit metering also becomes very straight forward since we just have to worry about buffer size counting against tx memory limits.

the more entries we have per-transaction, the higher the cost of building and validating parallel tx sets is.

I hadn't considered this, I was only thinking in terms of execution, so a tx limit seems reasonable.

My final recommendation is to:

1. maintain in-memory Soroban data in serialized XDR form
2. Meter loads into Cxx buffer wrt memory tx limits
3. Keep only a per-tx limit on in-memory reads, but no ledger wide limit

[SirTyson]

One potential complication with Soroban data in serialized XDR form is the entry size wrt fees. Currently we use LedgerEntry size, is this different than the serialized size? If not, there's no issue. If it is, we might need to change to use the serialized size for fee calculation in p23. From core/env perspective this is fine, but might wreck downstream. Alternatively we can just cache pairs of <serializedXdrData, LedgerEntrySize>, which shouldn't add much memory and also be simple to implement.

[dmkozh]

Currently we use LedgerEntry size, is this different than the serialized size?

It is actually the size of the serialize XDR, so there is no need to do anything here.

FWIW we only happen to use in-memory data structure sizes for metering in Rust (and there is some code that ensures that this is actually stable and independent of the environment the code runs in). On the C++ side there is no such machinery and thus all the metering has to only be performed based on the sizes of serialized XDR.

[SirTyson]

Ah cool, that makes everything easy then

[SirTyson]

There's a potential DOS issue with how we handle state size related fees wrt module cache in CAP-65. CAP 65 caches all live WASM modules in memory. To prevent OOM based DOS, the number of live WASM modules that are required to be cached at any given time is bounded by sorobanLiveStateTargetSizeBytes in CAP 62. The issue is, sorobanLiveStateTargetSizeBytes limits the number of on-disk, raw WASM bytes of CONTRACT_CODE, whereas the module cache stores modules in memory. In the worst case, this means the size of the module cache is about 40x larger than the size of the actual CONTRACT_CODE entries being cached. In practice, this means an attacker could require core nodes to store 40 * sorobanLiveStateTargetSizeBytes bytes of memory at any given point. Note that this issue is only present in the module cache. The contract data cache in CAP 66 is not affected, since the cached data and on-disk data share a 1-1 size ratio.

I see two potential solutions.

1. For the purposes of metering and fees, we could treat CONTRACT_CODE as if it was 40x bigger than it actually is. If a 1 KB wasm is uploaded, it is charged and metered as if it is 40 KB. Since we always overcharge, sorobanLiveStateTargetSizeBytes is again an upper bound on memory usage. Actual memory usage could be significantly below since we artificially inflate code entries.

• Pros
  • Easiest implementation
  • sorobanLiveStateTargetSizeBytes bounds both memory and disk size of soroban state
• Cons
  • Contract uploads, restores, and rent bumps are 40x more expensive to end users
  • Memory is underutilized, as non-malicious contract modules use significantly less than 40x WASM size

2. sorobanLiveStateTargetSizeBytes is changed to track in-memory state as apposed to on-disk state. On startup whenever the soroban data cache is populated, we track the size of all data, and update it accordingly whenever new data is created, deleted, restored, or evicted. Similarly, when contract modules are instantiated, we track the actual, metered instantiation memory costs of the module. While metering is still an overestimate, it is significantly tighter than the 40x worst case scenario.

• Pros
  • Best memory utilization while still providing DOS protection, as sorobanLiveStateTargetSizeBytes is a more tight upper bound.
  • Fees are lower and better reflect actual utilization of resources
• Cons
  • More difficult implementation, as we'd have to track module sizes and pipe the metered results of instantiation from Rust env to Cxx core.
  • sorobanLiveStateTargetSizeBytes no longer bounds disk size, only live state/memory cache size.
    • In a way, this is both a pro and a con. From a network user perspective, it is more "fair". Network users have direct impact on whether an entry is live or archived, and sorobanLiveStateTargetSizeBytes will track exactly this. If sorobanLiveStateTargetSizeBytes is a disk based metric, it tracks many BucketList implementation details that are both opaque to network users and impossible for them to influence.

Personally, I am pro option 2. I think given cache optimizations and recent optimizations to new node startup, the size of the BucketList is less of a concern now than it was previously, with the size of live state being the primary factor we need to meter. Classic still dominates BucketList size anyway, and write limits still provide a less direct way to slow BucketList growth from the soroban size. This also gives us the lowest fees while maintaining DOS safety, and using memory resources most tightly.

It was also suggested that we break sorobanLiveStateTargetSizeBytes into two seperate limits, one for data and one for code, where we could infer that contractCodeTargetSizeBytes requires 40x bytes in memory when setting limits. This adds significantly more complexity to the fee and resource settings than what was discussed above and was not popular amonst the env devs, but can be revisted.

[dmkozh]

I also think that option 2 is better. I hope that implementation is not significantly more complex, given that the metering code is already there and we seemingly only need to plumb an additional value to the Core.
The only downside of not tracking the BL size is that we're missing the impact of ledger modifications on the state size, but I don't think it is really valuable to measure it. We had to flatten the fee rates and increase the size computation window size in order to compensate for the size fluctuations caused by the modifications and BL merges, so it doesn't seem like the current fee model works well for modifications either.

[graydon]

if I'm understanding correctly this is just the distinction between the coarse initial-parse cost and the secondary refined cost (once we have a ContractCodeCostInputs). I don't think it'll matter much in practice but if you want to remain closer to the real memory usage / not overestimate it, we can use the refined one. I agree with @dmkozh that we should probably just recycle the cost-measuring code that exists on the Rust side already. Call it with a fresh budget, ask how much memory the LE would take to parse. It should be fairly easy to wire up as a pure function.