Unable to update replication factor for topic by passing json

[raje143r]

Hi Team,

I'm trying to update replication factor to 3 for already created topic by creating json file and executing the kubectl command /kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file /home/localen/demotesttopic.json --execute
but getting below error even after I have admin rights
Error: Unable to read file /home/localen/demotesttopic.json
java.io.IOException: Unable to read file /home/localen/demotesttopic.json
at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:698)
at kafka.admin.ReassignPartitionsCommand$.handleAction(ReassignPartitionsCommand.scala:256)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:215)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
Caused by: java.nio.file.NoSuchFileException: /home/localen/demotesttopic.json
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3206)
at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:695)
... 3 more

Below is my json file and its valid json
{
"version": 1,
"partitions": [{
"topic": "demotesttopic",
"partition": 0,
"replicas": [0, 1, 2]
},
{
"topic": "demotesttopic",
"partition": 1,
"replicas": [0, 1, 2]
},
{
"topic": "demotesttopic",
"partition": 2,
"replicas": [0, 1, 2]
},
{
"topic": "demotesttopic",
"partition": 3,
"replicas": [0, 1, 2]
}
]
}

Could you please suggest is there any way to update replication factor .

Thanks in advance

[scholzj]

I'm not sure how you run it, but the error says that the file probably doesn't exist. Keep in mind that if you run it using kubectl exec or kubectl run, you will need to copy the file into the pod as the command will not see any files on your local computer.

[raje143r]

Thanks for the response jakub.
You mean , first respective json file should add in the pod , and then use that pod reference in the kubectl command instead of giving filepath in the command as /home/localen/ is it?

[scholzj]

Well, you didn't shared how exactly are you running the command. But if you run it inside a Pod, it has its own filesystem. So you need to create the file directly there or copy it from local computer to the pod.

[raje143r]

I'm referring command as per the document : https://strimzi.io/docs/operators/0.22.1/deploying.html
kubectl run create -ti --image=quay.io/strimzi/kafka:0.22.1-kafka-2.7.0 --rm=true --restart=Never -- bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file /home/localen/demotesttopic.json --execute

[scholzj]

Right ... so you need to either create it in the pod or you need to copy it into the pod it creates.

[raje143r]

Hi Jakub,
As per your suggestion, we have created a file inside the temp folder(we can't keep file other than temp folder due to permission issue's) in the pod and executed kafka-reassign-partitions.sh but getting the same error. "Unable to read the file /tmp/demotesttopic.json".
It will be helpful if you can share any example for the same.
Thanks in advance

[scholzj]

Well, it looks like you did something wrong there. Maybe if you share the exact process step by step, it will show where you make the mistake. You should also check the permissions on the file etc.

[raje143r]

Below are the steps we followed

1. created the pod by using : kubectl run reassign --image=quay.io/strimzi/kafka:0.22.1-kafka-2.7.0 -n envname
2. copied json file from local machine to pod using kubectl copy : kubectl cp /path/to/file my-pod:/path/to/file
3. logged/entered into running pod using kubectl exec : kubectl exec -it reassign -n envname -- /bin/bash
4. executed ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file /tmp/reassignment.json --execute
   getting the below error:

Error: Unable to read file /tmp/reassignment.json
java.io.IOException: Unable to read file /tmp/reassignment.json
at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:698)
at kafka.admin.ReassignPartitionsCommand$.handleAction(ReassignPartitionsCommand.scala:256)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:215)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
Caused by: java.nio.file.NoSuchFileException: /tmp/reassignment.json
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3206)
at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:695)
... 3 more

[scholzj]

Hmm, I don't know, hard to say without seeing the pod etc. In general the steps are correct, although they do not seem the exact steps you executed I guess.

[raje143r]

Below are the logs when executed in kubernetes and individual docker containers

Kubernetes exception:
[kafka@create bin]$ bash ./kafka-reassign-partitions.sh --bootstrap-server :9092 --reassignment-json-file /tmp/reassignment.json --execute
Error: Unable to read file /tmp/reassignment.json
java.io.IOException: Unable to read file /tmp/reassignment.json
at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:698)
at kafka.admin.ReassignPartitionsCommand$.handleAction(ReassignPartitionsCommand.scala:256)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:215)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
Caused by: java.nio.file.NoSuchFileException: /tmp/reassignment.json
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3206)
at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:695)
... 3 more

2. kafka@create kafka]$ ls /tmp/
   hsperfdata_root ks-script-DrRL8A rfdemotesttopic.json yum.log
   these are the list of files available inside my pod

3. below are the logs when executed docker container

kafka@f58f63a9a695 kafka]$ cd bin
[kafka@f58f63a9a695 bin]$ ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file /tmp/rfdemotesttopic.json --execute
[2021-12-23 11:42:41,276] WARN [AdminClient clientId=reassign-partitions-tool] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

4. [kafka@f58f63a9a695 bin]$ ls /tmp/
   hsperfdata_kafka hsperfdata_root ks-script-DrRL8A rfdemotesttopic.json yum.log
   these are the files listed inside docker container

[scholzj]

Sorry, but without the exact steps and all of them, it is hard to help. You always share only some of them or share some dummy steps without the real values. The error says it is unable to read the file. So you need to investigate why is it not readable.

[raje143r]

attached screenshots for reference. this will be helpful

POD screenshot:

DockerContainer screenshot:

)