HDFS files are corrupted post stackable operator upgrade

[rajarshp]

Hi team,
We recently upgraded our AWS Rosa stackable operator to 25.3.0 (with Openshift 4.17) from 24.7.0, but after upgrading we are seeing lots of issues, the journal files are getting corrupted and application is failing.

So, could you please let me know how we can fix this ? and is this version not compatible with the older version ?

[lfrancke]

I'm afraid we'll need more information than that.
Which HDFS version are you using? Which errors are you seeing, what does your duster config look like and so on.

I'm not aware of any known issues. That said we do not generally support skipping versions during upgrades. You skipped 24.11 in this case.

[lfrancke]

It'd be helpful if we could get your YAML files to see what you deployed and we might need more information on your environment.

Also note that I'm going to be on vacation for three weeks as of Thursday. I'll tell the team about this discussion but can't make any promises.

[Jimvin]

Could you provide us some context to these error messages. Which logs do you see these messages in? Do you see this on multiple pods or just on one? Is the HDFS service not working at this point?

[rajarshp]

@Jimvin We are seeing these error in our application/datenodes logs in all the non prod env (as it is not in Prod yet)
Though the HDFS is up and running but due to these failures our application will crash.

Also we have see a trend, wherever we are seeing these UnknownHost or No more good datanode exception in the app log, we can see below container debug logs(which we asked to ignore) around that time in the datanode logs.

2025-08-19T04:15:35.522177Z WARN containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: reverse DNS lookup failed ip=fXXXXXXXXX:2150 error=no record found for Query { name: Name("0.5.1.2.3.8.e.f.f.f.a.0.8.5.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa."), query_type: PTR, query_class: IN }

This one event where we have seen this : -

<style> </style>

Timestamp	Pod	Error	Comments
2025-08-15T10:29:30	App server	2025-08-15 10:29:30,039 WARN [Thread-912467] hdfs.DataStreamer (DataStreamer.java:createBlockOutputStream(1959)) - Exception in createBlockOutputStream blk_1076531235_3069771 +++ERROR+++ : upd_server.std_err - hdfsHSync: FSDataOutputStream#hsync error: 2025-08-15 10:29:30,050 WARN [Thread-16] hdfs.DFSClient (DFSOutputStream.java:flushOrSync(748)) - Error while syncing 2025-08-15 10:29:30,043 WARN [Thread-912467] hdfs.DataStreamer (DataStreamer.java:handleBadDatanode(1731)) - Error Recovery for XX1808250458-X.X.X.X-1754595799839:blk_1076531235_3069771 in pipeline [DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,DS-441ca41e-1acf-4143-8fad-32a8f617792c,DISK], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,DS-e6a5fd38-8929-446d-b4db-6eb18579819d,DISK], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866,DS-49cf7e39-cd90-4ca0-9718-63cef849ad12,DISK]]: datanode 2(DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866,DS-49cf7e39-cd90-4ca0-9718-63cef849ad12,DISK]) is bad. IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,DS-441ca41e-1acf-4143-8fad-32a8f617792c,DISK], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,DS-e6a5fd38-8929-446d-b4db-6eb18579819d,DISK]], original=[DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,DS-441ca41e-1acf-4143-8fad-32a8f617792c,DISK], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,DS-e6a5fd38-8929-446d-b4db-6eb18579819d,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,DS-441ca41e-1acf-4143-8fad-32a8f617792c,DISK], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,DS-e6a5fd38-8929-446d-b4db-6eb18579819d,DISK]], original=[DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,DS-441ca41e-1acf-4143-8fad-32a8f617792c,DISK], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,DS-e6a5fd38-8929-446d-b4db-6eb18579819d,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866	Application was processing blk_1076531235_3069771. Current replication factor is 3.
2025-08-15T10:29:04	XXXXXhdfs-datanode-default-1	2025-08-15 10:29:04,961 INFO  DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1550)) - src: /XX.XX.XX:35678, dest: /X.X.X.X:9866, bytes: 44270616, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1910486013_1, offset: 0, srvID: 4a3542a9-00c4-42de-9cc6-3c091f46eb43, lockid: XX1808250458-X.X.X.X-1754595799839:blk_1076531235_3069771, duration(ns): 898148618 2025-08-15 10:29:04,961 INFO datanode.DataNode (BlockReceiver.java:run(1523)) - PacketResponder: XX1808250458-XXXX-1754595799839:blk_1076531235_3069771, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866, XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866] terminating	Block was written in datanode 1 successfully. It is pending to write on datanode 0,2.
2025-08-15T10:29:30	XXXXXhdfs-datanode-default-0	2025-08-15 10:29:30,038 ERROR datanode.DataNode (DataXceiver.java:run(326)) - XXXXXhdfs-datanode-default-0.XXXXXhdfs-datanode-default.Namespace.svc.cluster.local:9866:DataXceiver error processing WRITE_BLOCK operation  src: /X.X.X.X:57180 dst: /X.X.X.X:9866 at java.base/java.lang.Thread.run(Unknown Source org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:294) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:798) org.apache.hadoop.net.NetUtils.connect(NetUtils.java:563) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:605) java.net.UnknownHostException: XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local 2025-08-15 10:30:38,788 INFO  datanode.DataNode (BlockRecoveryWorker.java:logRecoverBlock(555)) - BlockRecoveryWorker: NameNode at XXXXXhdfs-namenode-default-0.XXXXXhdfs-namenode-default.Namespace.svc.cluster.local/X.X.X.X:8020 calls recoverBlock(XX1808250458-X.X.X.X-1754595799839: blk_1076531235_3069771, targets=[DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,null,null], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,null,null], DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866,null,null]], newGenerationStamp=3070066, newBlock=null, isStriped=false) 2025-08-15 10:30:38,827 INFO  datanode.DataNode (BlockRecoveryWorker.java:syncBlock(201)) - BlockRecoveryWorker: block=XX1808250458-X.X.X.X-1754595799839:blk_1076531235_3069771 (length=44270616), isTruncateRecovery=false, syncList=[block:blk_1076531235_3069825[numBytes=44270616,originalReplicaState=RBW] node:DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,null,null], block:blk_1076531235_3069825[numBytes=44270616,originalReplicaState=RBW] node:DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,null,null], block:blk_1076531235_3069771[numBytes=44270616,originalReplicaState=FINALIZED] node:DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866,null,null]] 2025-08-15 10:30:38,827 INFO  datanode.DataNode (BlockRecoveryWorker.java:syncBlock(294)) - BlockRecoveryWorker: block=XX1808250458-X.X.X.X-1754595799839:blk_1076531235_3069771 (length=44270616), bestState=FINALIZED, newBlock=XX1808250458-X.X.X.X-1754595799839:blk_1076531235_3070066 (length=44270616), participatingList=[block:blk_1076531235_3069825[numBytes=44270616,originalReplicaState=RBW] node:DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-2-listener.Namespace.svc.cluster.local:9866,null,null], block:blk_1076531235_3069825[numBytes=44270616,originalReplicaState=RBW] node:DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866,null,null], block:blk_1076531235_3069771[numBytes=44270616,originalReplicaState=FINALIZED] node:DatanodeInfoWithStorage[XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866,null,null]] 2025-08-15 10:30:51,754 INFO  impl.FsDatasetImpl (FsDatasetImpl.java:append(1252)) - Appending to FinalizedReplica, blk_1076531235_3070066, FINALIZED	We can see UnknownHostException in datanode 0 when it was trying to write data. It was trying to connect datanode 1 but couldn’t resolve it. The same time application also thrown exception about the Bad datanode. It seems the same operation of the block which above data 1 finished successfully and waiting to write in datanode 0, 2. Recovery started after 8 sec of the failure. It finalized the recovery and created a new block, which yet to get written in datanode 0, 2. HDFS creates an intermediate block during recovery process New block (recovery block of blk_1076531235_3069771) successfully writted to datanode 0 after 13 sec of the revery process.
2025-08-15T10:29:30	XXXXXhdfs-datanode-default-2	2025-08-15 10:29:30,038 INFO datanode.DataNode (BlockReceiver.java:run(1468)) - PacketResponder: XX1808250458-X.X.X.X-1754595799839:blk_1076531235_3069825, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[XXXXXhdfs-datanode-default-0-listener.Namespace.svc.cluster.local:9866, XXXXXhdfs-datanode-default-1-listener.Namespace.svc.cluster.local:9866] at java.base/java.lang.Thread.run(Unknown Source) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1398) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:521) WARN  datanode.DataNode (BlockReceiver.java:run(1503)) - IOException in PacketResponder.run(): java.base/sun.nio.ch.IOUtil.write(Unknown Source) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1495) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1585) java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source) at java.base/java.io.BufferedOutputStream.flushBuffer(Unknown Source) org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:158) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1650) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:116) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141) org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62) java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source) java.base/java.io.BufferedOutputStream.flush(Unknown Source) java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method) java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source) java.io.IOException: Connection reset by peer 2025-08-15 10:30:38,808 INFO impl.FsDatasetImpl (FsDatasetImpl.java:initReplicaRecoveryImpl(2656)) - initReplicaRecovery: blk_1076531235_3069771, recoveryId=3070066, replica=ReplicaBeingWritten, blk_1076531235_3069825, RBW	It was trying to sync the intermediate block that was created by datanode 0 after failure, but with datanode 2 also the sync failed dure to some connection issue. HDFS again started a recovery process in datanode 2 as well

[rajarshp]

I am not is this the same issue which is not resolved from a long time, the error looks similar

https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=14717137

[rajarshp]

@Jimvin did you get a chance to have a look ?

[rajarshp]

Thanks @lfrancke
On one env we upgraded it 3.4.1 , but rest of the env has 3.3.6 (we are using same template to deploy it)

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: stackble-hdfs-template
parameters:
  - name: ENV
    description: Environment
    required: true
objects:
  - apiVersion: hdfs.stackable.tech/v1alpha1
    kind: HdfsCluster
    metadata:
      name: hdfs
      namespace: dev
    spec:
      clusterConfig:
        dfsReplication: 3
        listenerClass: cluster-internal
        zookeeperConfigMapName: zk-map
      clusterOperation:
        reconciliationPaused: false
        stopped: false
      dataNodes:
        cliOverrides: {}
        config:
          affinity:
            nodeAffinity: null
            nodeSelector: null
            podAffinity: null
            podAntiAffinity: null
          listenerClass: cluster-internal
          logging:
            containers: {}
            enableVectorAgent: null
          resources:
            cpu:
              max: "2"
              min: "200m"
            memory:
              limit: 4Gi
              runtimeLimits: {}
            storage:
              data:
                capacity: 30Gi
        configOverrides: {}
        envOverrides: {}
        podOverrides: {}
        roleConfig:
          podDisruptionBudget:
            enabled: true
            maxUnavailable: null
        roleGroups:
          default:
            cliOverrides: {}
            config:
              affinity:
                nodeAffinity: null
                nodeSelector: null
                podAffinity: null
                podAntiAffinity: null
              logging:
                containers: {}
                enableVectorAgent: null
              resources:
                cpu:
                  max: "2"
                  min: "200m"
                memory:
                  limit: 4Gi
                  runtimeLimits: {}
                storage:
                  data:
                    capacity: 30Gi
            configOverrides: {}
            envOverrides: {}
            podOverrides: {}
            replicas: 3
      image:
        productVersion: 3.3.6
        pullPolicy: Always
      journalNodes:
        cliOverrides: {}
        config:
          affinity:
            nodeAffinity: null
            nodeSelector: null
            podAffinity: null
            podAntiAffinity: null
          logging:
            containers: {}
            enableVectorAgent: null
          resources:
            cpu:
              max: "1"
              min: "50m"
            memory:
              limit: 4Gi
              runtimeLimits: {}
            storage:
              data:
                capacity: 8Gi
        configOverrides: {}
        envOverrides: {}
        podOverrides: {}
        roleConfig:
          podDisruptionBudget:
            enabled: true
            maxUnavailable: null
        roleGroups:
          default:
            cliOverrides: {}
            config:
              affinity:
                nodeAffinity: null
                nodeSelector: null
                podAffinity: null
                podAntiAffinity: null
              logging:
                containers: {}
                enableVectorAgent: null
              resources:
                cpu:
                  max: "1"
                  min: "50m"
                memory:
                  limit: 4Gi
                  runtimeLimits: {}
                storage:
                  data:
                    capacity: 8Gi
            configOverrides: {}
            envOverrides: {}
            podOverrides: {}
            replicas: 2
      nameNodes:
        cliOverrides: {}
        config:
          affinity:
            nodeAffinity: null
            nodeSelector: null
            podAffinity: null
            podAntiAffinity: null
          listenerClass: cluster-internal
          logging:
            containers: {}
            enableVectorAgent: null
          resources:
            cpu:
              max: "1"
              min: "50m"
            memory:
              limit: 4Gi
              runtimeLimits: {}
            storage:
              data:
                capacity: 8Gi
        configOverrides: {}
        envOverrides: {}
        podOverrides: {}
        roleConfig:
          podDisruptionBudget:
            enabled: true
            maxUnavailable: null
        roleGroups:
          default:
            cliOverrides: {}
            config:
              affinity:
                nodeAffinity: null
                nodeSelector: null
                podAffinity: null
                podAntiAffinity: null
              logging:
                containers: {}
                enableVectorAgent: null
              resources:
                cpu:
                  max: "1"
                  min: "50m"
                memory:
                  limit: 4Gi
                  runtimeLimits: {}
                storage:
                  data:
                    capacity: 8Gi
            configOverrides: {}
            envOverrides: {}
            podOverrides: {}
            replicas: 2

[rajarshp]

Team,
it will be really great if someone can have a look on this ?

[Jimvin]

The configuration looks fine, I don't think this is the cause of the issues you are seeing.

[rajarshp]

we are seeing multiple issue and trying to find out what is the root cause, whether it s HDFS issue, infrastructure or application

• No more good datanode
• UnknownHostException
• Unexpected EOF while trying to read response from server
• Failed to APPEND_FILE because client XX is already the current lease holder