Skip to content

Add support for custom max size per field #551

@imalygin

Description

@imalygin

Problem

We encountered the following exception during the load testing:

2025-07-30 11:39:59.993 31490    ERROR EXCEPTION        <<scheduler TransactionHandler>> TransactionHandler: error invoking ConsensusStateEventHandler.onHandleConsensusRound() [ nodeId = 1 ] with round 75353
java.lang.RuntimeException: Failed to deserialize a value from bytes
	at com.swirlds.virtualmap.datasource.VirtualLeafBytes.value(VirtualLeafBytes.java:105)
	at com.swirlds.virtualmap.VirtualMap.get(VirtualMap.java:611)
	at com.swirlds.state.merkle.disk.OnDiskQueueHelper.getFromStore(OnDiskQueueHelper.java:115)
	at com.swirlds.state.merkle.disk.OnDiskQueueHelper$QueueIterator.next(OnDiskQueueHelper.java:212)
	at com.swirlds.state.spi.WritableQueueStateBase.peek(WritableQueueStateBase.java:134)
	at com.hedera.node.app.state.recordcache.RecordCacheImpl.purgeExpiredReceiptEntries(RecordCacheImpl.java:394)
	at com.hedera.node.app.state.recordcache.RecordCacheImpl.commitRoundReceipts(RecordCacheImpl.java:351)
	at com.hedera.node.app.workflows.handle.HandleWorkflow.handleRound(HandleWorkflow.java:319)
	at com.hedera.node.app.Hedera.onHandleConsensusRound(Hedera.java:994)
	at com.hedera.node.app.state.ConsensusStateEventHandlerImpl.onHandleConsensusRound(ConsensusStateEventHandlerImpl.java:45)
	at com.swirlds.platform.state.TransactionHandler.handleRound(TransactionHandler.java:55)
	at com.swirlds.platform.state.SwirldStateManager.handleConsensusRound(SwirldStateManager.java:131)
	at com.swirlds.platform.eventhandling.DefaultTransactionHandler.handleConsensusRound(DefaultTransactionHandler.java:212)
	at com.swirlds.component.framework.component.ComponentWiring.lambda$bind$7(ComponentWiring.java:758)
	at com.swirlds.component.framework.wires.input.BindableInputWire.lambda$bind$1(BindableInputWire.java:96)
	at com.swirlds.component.framework.schedulers.internal.SequentialThreadTask.handle(SequentialThreadTask.java:19)
	at com.swirlds.component.framework.schedulers.internal.SequentialThreadTaskScheduler.run(SequentialThreadTaskScheduler.java:193)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.hedera.pbj.runtime.ParseException: RecordCacheTransactionReceiptQueue size 2404962 is greater than max 2097152
	at com.hedera.hapi.platform.state.codec.VirtualMapValueProtoCodec.parse(VirtualMapValueProtoCodec.java:1716)
	at com.hedera.hapi.platform.state.codec.VirtualMapValueProtoCodec.parse(VirtualMapValueProtoCodec.java:116)
	at com.hedera.pbj.runtime.Codec.parse(Codec.java:69)
	at com.hedera.pbj.runtime.Codec.parse(Codec.java:104)
	at com.hedera.pbj.runtime.Codec.parse(Codec.java:116)
	at com.swirlds.virtualmap.datasource.VirtualLeafBytes.value(VirtualLeafBytes.java:103)

This limit is defined in com.hedera.pbj.compiler.impl.Field as

 /** The default maximum size of a repeated or length-encoded field (Bytes, String, Message, etc.). */
    public static final long DEFAULT_MAX_SIZE = 2 * 1024 * 1024;

Currently we don't support customization of this value but the exception above suggests that we may actually need it.

Solution

Implement a feature allowing to define maxSize for a field

Alternatives

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions