KafkaMirrorMaker2 does not mirror Write permission ACLs for topics

[hdjur]

Hi,

I don't know if this is a feature or bug, I expected all ACLs for a subset of topics from source cluster defined by topicsPattern to be restored at the target cluster, but I see that (at least) Write permission ACLs are missing, I didn't check Describe yet.
Looking at this: https://strimzi.io/docs/operators/latest/using.html#con-mirrormaker-acls-str , it might be a feature, but I don't quite understand the motivation behind it.
I have set sync.topic.acls.enabled to true, hoping it controls if ACLs for topics are mirrored or not, and if they are mirrored, all of them would be mirrored, and it actually only controls if ACLs changes are mirrored or not, but strictly for Read operation.
Can someone please shed some light for me on this issue?

Regards,
Hrvoje

[scholzj]

Looking at this: https://strimzi.io/docs/operators/latest/using.html#con-mirrormaker-acls-str , it might be a feature, but I don't quite understand the motivation behind it.

The motivation behind https://strimzi.io/docs/operators/latest/using.html#con-mirrormaker-acls-str is that the User Operator is used to manage the ACLs. And when it does, it will always treat the KafkaUser resources as the single source of truth. So if Mirror Maker 2 mirrors any ACLs, the User Operator would anyway overwrite them / delete them to make sure the ACLs are set according to the KafkaUser resources. So to make any sense of the ACL mirroring, you would need to disable the User Operator. Then it will not be deleting the ACLs and let MM2 mirror them.

I have set sync.topic.acls.enabled to true, hoping it controls if ACLs for topics are mirrored or not, and if they are mirrored, all of them would be mirrored, and it actually only controls if ACLs changes are mirrored or not, but strictly for Read operation.

The sync.topic.acls.enabled is just MM2 option and Strimzi will just pass it to Mirror Maker 2. So what it actually does and how it behaves depends on MM2. TBH I never used the ACL mirroring much because of the User Operator conflict, so I do not know how exactly it works.

[hdjur]

No, none of the two kafka clusters is managed by kubernetes operators, strimzi is deployed with the only specific purpose to run KafkaMirrorMaker2, because, somehow we did not manage to get rid of the undesired source cluster prefix in topic names, when we used the version of mm2 that we got with apache distribution of kafka v 2.6. With this version of mm2 we managed to avoid that prefix. I don't know if that is the same version, and we just didn't try properly prior to this, but anyway, now the only objection is this strange handling of ACL sync. The other reason was to get familiar with kubernetes operators, and we didn't deploy standalone User operator so far.

[scholzj]

Ok. In that case it is probably some issue either in MM2 it self or maybe your Kafka clusters have something similar to our User Operator

[hdjur]

Hi, I checked both by testing it, and looking into code here:
https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java

    boolean shouldReplicateAcl(AclBinding aclBinding) {
        return !(aclBinding.entry().permissionType() == AclPermissionType.ALLOW
            && aclBinding.entry().operation() == AclOperation.WRITE);
    }

that DESCRIBE ACLs are also replicated, ie only WRITE operation permission allow ACLs for topics are avoided, for some reason.
So I wrote a small java program using AdminClient class that replicates the missing ACLs for me. The alternative option was to use zoocreeper or something similar to backup/restore definitions from zookeepers of both clusters, but I made a more precisely tailored solution.

[scholzj]

Ok, I guess maybe the logic is following: The topic should be writable only in the source cluster and read only in the target cluster . So maybe not mirroring the WRITE ACLs would help to achieve that since nobody would get the write permissions for the mirrored topic? That would of course mean this cannot be used for migration from one cluster to another where at the end you delete the source cluster and use only the target cluster. But this is just assumption - you would need to confirm it with the MM2 authors I guess.

[hdjur]

Yes, I suppose so too. Thank you, it was a nice discussion.