Janusgraph try to impersonate another user on hbase

[t57root]

Hi,
I'm using the janusgraph offical docker image to run janusgraph, using a kerberized HBase as the storage backend.
But the HBase log shows that the login from janusgraph try to impersonate another user:
RpcServer.reader=0,port=29600] org.apache.hadoop.ipc.RpcServer: Connection authorization failed: User: kerberos_username@domain is not allowed to impersonate janusgraph123
org.apache.hadoop.hbase.security.AccessDeniedException: Connection from 10.119.234.9:33680 for service ClientService is unauthorized for user: janusgraph (auth:PROXY) via kerberos_username@domain (auth:KERBEROS)
the janusgraph process is exactly right run under the user ganusgraph123. Is there some configuration leading this behavior?
Thanks in advance!

[vtslab]

For janusgraph-server to access a kerberized HBase cluster, it needs:

• a keytab
• the kerberos client packages
• the kdc config file
• a klogin command using the keytab

I guess you will need an additional docker image layer on top of the janusgraph docker to realize this. After that, the hbase/hadoop client shipped with JanusGraph will pickup the kerberos config automagically (using default locations or env variables).

[t57root]

Hi, thanks for the answer. I've already done the kerberos credential part and the hbase received a request with decent kerberos auth.
But the log shows it seems that after the kerberos login, the client(janusgraph) then tries to impersonate another user.

[vtslab]

Long time ago that I used this myself (before the existence of hadoop proxyuser). Do you recognize the origin of "kerberos_username@domain" or is this some placeholder from kerberos itself? Apart from authenticating incoming connections, JanusGraph Server does not care about usernames.
So, I suspect it to be a docker thing, are you sure the klogin was done with the janusgraph123 user too? Can you exec into the container and see whether a TGT with the right UID is present?

[t57root]

"kerberos_username@domain" is the kerberos principal name which I configured in the janusgraph conf file, I replace the actual content for security reason.
It seems the ticket is not shown in klist result if the ticket is generated within janusgraph process. But I turn on the sun.security.krb5.debug flag, the debug log shows the tgt and tgs does be the right principal name(kerberos_username@domain), the name is also shown in the hbase log: Connection from 10.119.234.9:33680 for service ClientService is unauthorized for user: janusgraph123 (auth:PROXY) via kerberos_username@domain (auth:KERBEROS).

The username janusgraph123 doesn't exist anywhere in the janusgraph config files. The only origin of this name is the local linux user which the janusgraph is running under. I tried to run janusgraph under another linux user(say janusgraph124), then the hbase log will shows user: janusgraph124 (auth:PROXY) via kerberos_username@domain (auth:KERBEROS) .

[vtslab]

Ok, the principal name and username are not the same in your application. You have to tell hadoop (used in its turn by the HBase client): https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Mapping_from_Kerberos_principals_to_OS_user_accounts

The configs described in the link above can be made available to JanusGraph Server by adding the location of your hadoop config files to the JanusGraph CLASSPATH env variable. Probably you did this already given the security logs your provided.

[t57root]

I didn't do that before but after I adding auth_to_local into krb5.conf and write a hbase-site.xml with auth_to_local, the error still shows up. Event I tried to change the os user into kerberos_username, still no luck.

[vtslab]

auth_to_local belongs in the core-site.xml. I am not sure whether the hadoop/hbase clients pick up auth_to_local rules from the krb5.conf, see https://issues.apache.org/jira/browse/HADOOP-16023. You can use the krb5.conf to test the rule, though. Kinit should run as user janusgraph123 with the keytab and not as a different user.

[t57root]

It seems the configuration specified via storage.hbase.ext.hadoop.security.authentication=kerberos is not working, After I set storage.hbase.ext.hadoop.security.authentication in the core-site.xml then it works.

Thanks for your kindly and patient help!