bug(s3plugin): While backfilling UploadTask keeps on failing #1941
Description
Description
While the BN is catching up in autonomous backfilling the S3ArchivePlugin produces too many logs in failed attempts to upload a batch.
Steps to Reproduce
Given the following network topology:
block_nodes:
rfh01_test:
address: <redacted>
port: 40840
peers: [s01_test_chicago]
s01_test_chicago:
address: <redacted>
port: 40840
peers: [rfh01_test]
consensus_nodes:
us_central:
block_nodes: [rfh01_test, s01_test_chicago]
us_east:
block_nodes: [s01_test_chicago, rfh01_test]
BN s01_test_chicago already had blocks from 64920000 to 65017285
when block node: rfh01_test came online, with following config override:
BLOCK_NODE_EARLIEST_MANAGED_BLOCK: "65019285"
BACKFILL_START_BLOCK: "64920000"
As expected us_central CN was able to stream to rfh01_test and blocks starting to flow.
Also once Autonomous Backfill was triggered, it started backfilling from block 64920000 up to the first block it received at the time.
Additional context
Even though that everything is working as expected, the BN since it was configure to Archive to S3, is attempting to upload it's batches but it can't, since range is not complete yet for each range.
producing this logs over and over again:
2025-12-05 21:55:13.905+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,830,000 was not found! Cannot proceed to upload archive block batch: 64,830,000 - 64,839,999
2025-12-05 21:55:13.905+0000 FINER [org.hiero.block.node.backfill.BackfillPlugin handlePersisted] Received backfillType=AUTONOMOUS persisted notification for block=64928378
2025-12-05 21:55:13.905+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,830,000 - 64,839,999
2025-12-05 21:55:13.906+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,840,000 - 64,849,999
2025-12-05 21:55:13.906+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,840,000 was not found! Cannot proceed to upload archive block batch: 64,840,000 - 64,849,999
2025-12-05 21:55:13.906+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,840,000 - 64,849,999
2025-12-05 21:55:13.906+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,850,000 - 64,859,999
2025-12-05 21:55:13.906+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,850,000 was not found! Cannot proceed to upload archive block batch: 64,850,000 - 64,859,999
2025-12-05 21:55:13.906+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,850,000 - 64,859,999
2025-12-05 21:55:13.906+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,860,000 - 64,869,999
2025-12-05 21:55:13.907+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,860,000 was not found! Cannot proceed to upload archive block batch: 64,860,000 - 64,869,999
2025-12-05 21:55:13.907+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,860,000 - 64,869,999
2025-12-05 21:55:13.907+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,870,000 - 64,879,999
2025-12-05 21:55:13.907+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,870,000 was not found! Cannot proceed to upload archive block batch: 64,870,000 - 64,879,999
2025-12-05 21:55:13.907+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,870,000 - 64,879,999
2025-12-05 21:55:13.907+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,880,000 - 64,889,999
2025-12-05 21:55:13.907+0000 FINER [org.hiero.block.node.verification.VerificationServicePlugin handleBackfilled] Verified backfill block items for block=64,928,379 with success=true
2025-12-05 21:55:13.907+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,880,000 was not found! Cannot proceed to upload archive block batch: 64,880,000 - 64,889,999
2025-12-05 21:55:13.907+0000 FINER [org.hiero.block.node.messaging.BlockMessagingFacilityImpl lambda$sendBlockVerification$6] Sending block verification notification for block=64,928,379 blockSource=BACKFILL and success=true
2025-12-05 21:55:13.907+0000 FINER [org.hiero.block.node.backfill.BackfillPlugin handleVerification] Received verification notification for block 64928379
2025-12-05 21:55:13.907+0000 FINER [org.hiero.block.node.blocks.files.recent.BlockFileRecentPlugin handleVerification] Persistence Handle verification started for block 64,928,379
2025-12-05 21:55:13.907+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,880,000 - 64,889,999
2025-12-05 21:55:13.908+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,890,000 - 64,899,999
2025-12-05 21:55:13.908+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,890,000 was not found! Cannot proceed to upload archive block batch: 64,890,000 - 64,899,999
2025-12-05 21:55:13.908+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,890,000 - 64,899,999
2025-12-05 21:55:13.908+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,900,000 - 64,909,999
2025-12-05 21:55:13.908+0000 FINER [org.hiero.block.node.blocks.files.historic.BlockFileHistoricPlugin writeBlockOrFail] Wrote verified block 64,928,379 to file /opt/hiero/block-node/data/historic/staging/000/000/000/006/492/8/0000000000064928379.blk.zstd
2025-12-05 21:55:13.908+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,900,000 was not found! Cannot proceed to upload archive block batch: 64,900,000 - 64,909,999
2025-12-05 21:55:13.908+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,900,000 - 64,909,999
2025-12-05 21:55:13.909+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,910,000 - 64,919,999
2025-12-05 21:55:13.909+0000 FINER [org.hiero.block.node.blocks.files.recent.BlockFileRecentPlugin writeBlockOrFail] Wrote verified block 64,928,379 to file /opt/hiero/block-node/data/live/000/000/000/006/492/8/0000000000064928379.blk.zstd
2025-12-05 21:55:13.909+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,910,000 was not found! Cannot proceed to upload archive block batch: 64,910,000 - 64,919,999
2025-12-05 21:55:13.909+0000 FINER [org.hiero.block.node.blocks.files.recent.BlockFileRecentPlugin handleVerification] Persistence Handle verification finished for block 64928379, and it took 1466127 ns to complete
2025-12-05 21:55:13.909+0000 FINER [org.hiero.block.node.messaging.BlockMessagingFacilityImpl lambda$sendBlockPersisted$8] Sending block persisted notification: PersistedNotification[blockNumber=64928379, succeeded=true, blockProviderPriority=2000, blockSource=BACKFILL]
2025-12-05 21:55:13.909+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,910,000 - 64,919,999
2025-12-05 21:55:13.909+0000 FINER [org.hiero.block.node.backfill.BackfillPlugin handlePersisted] Received backfillType=AUTONOMOUS persisted notification for block=64928379
2025-12-05 21:55:13.909+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,920,000 - 64,929,999
2025-12-05 21:55:13.909+0000 FINER [org.hiero.block.node.backfill.BackfillPlugin backfillGap] Successfully backfilled gap 64928370->64928379
2025-12-05 21:55:14.061+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,928,380 was not found! Cannot proceed to upload archive block batch: 64,920,000 - 64,929,999
2025-12-05 21:55:14.142+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,920,000 - 64,929,999
2025-12-05 21:55:14.142+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,930,000 - 64,939,999
2025-12-05 21:55:14.142+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,930,000 was not found! Cannot proceed to upload archive block batch: 64,930,000 - 64,939,999
2025-12-05 21:55:14.142+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,930,000 - 64,939,999
2025-12-05 21:55:14.143+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,940,000 - 64,949,999
2025-12-05 21:55:14.143+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,940,000 was not found! Cannot proceed to upload archive block batch: 64,940,000 - 64,949,999
2025-12-05 21:55:14.143+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,940,000 - 64,949,999
2025-12-05 21:55:14.143+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,950,000 - 64,959,999
2025-12-05 21:55:14.143+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,950,000 was not found! Cannot proceed to upload archive block batch: 64,950,000 - 64,959,999
2025-12-05 21:55:14.143+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,950,000 - 64,959,999
2025-12-05 21:55:14.143+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,960,000 - 64,969,999
2025-12-05 21:55:14.143+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,960,000 was not found! Cannot proceed to upload archive block batch: 64,960,000 - 64,969,999
2025-12-05 21:55:14.144+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,960,000 - 64,969,999
2025-12-05 21:55:14.144+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,970,000 - 64,979,999
2025-12-05 21:55:14.144+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,970,000 was not found! Cannot proceed to upload archive block batch: 64,970,000 - 64,979,999
2025-12-05 21:55:14.144+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,970,000 - 64,979,999
2025-12-05 21:55:14.144+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,980,000 - 64,989,999
2025-12-05 21:55:14.144+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,980,000 was not found! Cannot proceed to upload archive block batch: 64,980,000 - 64,989,999
2025-12-05 21:55:14.144+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,980,000 - 64,989,999
2025-12-05 21:55:14.144+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,990,000 - 64,999,999
2025-12-05 21:55:14.144+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,990,000 was not found! Cannot proceed to upload archive block batch: 64,990,000 - 64,999,999
2025-12-05 21:55:14.145+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,990,000 - 64,999,999
2025-12-05 21:55:14.145+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 65,000,000 - 65,009,999
2025-12-05 21:55:14.145+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 65,000,000 was not found! Cannot proceed to upload archive block batch: 65,000,000 - 65,009,999
2025-12-05 21:55:14.145+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 65,000,000 - 65,009,999
2025-12-05 21:55:14.145+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,320,000 - 64,329,999
2025-12-05 21:55:14.145+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,320,000 was not found! Cannot proceed to upload archive block batch: 64,320,000 - 64,329,999
2025-12-05 21:55:14.145+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,320,000 - 64,329,999
2025-12-05 21:55:14.145+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,330,000 - 64,339,999
2025-12-05 21:55:14.145+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,330,000 was not found! Cannot proceed to upload archive block batch: 64,330,000 - 64,339,999
2025-12-05 21:55:14.146+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,330,000 - 64,339,999
2025-12-05 21:55:14.146+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,340,000 - 64,349,999
2025-12-05 21:55:14.146+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,340,000 was not found! Cannot proceed to upload archive block batch: 64,340,000 - 64,349,999
2025-12-05 21:55:14.146+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,340,000 - 64,349,999
2025-12-05 21:55:14.146+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,350,000 - 64,359,999
2025-12-05 21:55:14.146+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,350,000 was not found! Cannot proceed to upload archive block batch: 64,350,000 - 64,359,999
2025-12-05 21:55:14.146+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,350,000 - 64,359,999
2025-12-05 21:55:14.146+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,360,000 - 64,369,999
2025-12-05 21:55:14.146+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,360,000 was not found! Cannot proceed to upload archive block batch: 64,360,000 - 64,369,999
2025-12-05 21:55:14.146+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,360,000 - 64,369,999
2025-12-05 21:55:14.147+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,370,000 - 64,379,999
2025-12-05 21:55:14.147+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,370,000 was not found! Cannot proceed to upload archive block batch: 64,370,000 - 64,379,999
2025-12-05 21:55:14.147+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,370,000 - 64,379,999
2025-12-05 21:55:14.147+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,380,000 - 64,389,999
2025-12-05 21:55:14.147+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,380,000 was not found! Cannot proceed to upload archive block batch: 64,380,000 - 64,389,999
2025-12-05 21:55:14.147+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,380,000 - 64,389,999
2025-12-05 21:55:14.147+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,390,000 - 64,399,999
2025-12-05 21:55:14.147+0000 WARNING [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask gatherAccessors] Historical block 64,390,000 was not found! Cannot proceed to upload archive block batch: 64,390,000 - 64,399,999
2025-12-05 21:55:14.147+0000 INFO [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Could not upload archive block batch: 64,390,000 - 64,399,999
2025-12-05 21:55:14.147+0000 FINE [org.hiero.block.node.archive.s3.S3ArchivePlugin$UploadTask call] Uploading archive block batch: 64,400,000 - 64,409,999
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status