Testnet importer down after consensus node v0.59.2 upgrade

[chad-schwab]

I'm moving a discussion started on discord over here. Note: as of this morning our testnet importer is processing correctly. I would like to understand what happened to testnet yesterday and how the public mirror operators were able to get around the problem. I would also like to understand what we should do if this happens on mainnet.

From Cory:

Since the testnet network upgrade to v0.59.2 on 2025-03-04, our mirror importer (based on the Hiero v0.124.1 importer) has been failing due to this error ERROR scheduling-4 c.h.m.i.d.r.RecordFileDownloader None of the data files can be verified.
Here are a sample of logs

2025-03-04T22:40:11.278Z ERROR scheduling-4 c.h.m.i.d.r.RecordFileDownloader None of the data files could be verified, signatures: [StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=3, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6), StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=0, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6), StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=5, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6), StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=6, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6), StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=2, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6), StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=4, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6), StreamFileSignature(filename=2025-03-04T19_05_50.832101506Z.rcd_sig, node=1, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6)] 

[steven-sheehy]

There was an issue during consensus node upgrade that caused erroneous files to get uploaded. Our operations team fixed the issue and removed the files in the GCS bucket but forgot the AWS bucket until later. Hedera mirror node primarily uses GCS so was impacted for less time.

It's useful to have both GCS and AWS credentials available and configured as sources in importer so in case one fails it can failover. Here, there wasn't a cloud provider issue so the failover would have to be manual.