kyuubi-1.7.1 对接华为mrs 3.2.0

[boombap9527]

当前通过kyuubi对接spark和ranger遇到一些问题 ，kyuubi对接spark正常，可正常像yarn提交spark任务，编译kyuubi-spark-authz jar包和对接ranger控制用户权限存在问题，目前集群无报错，配置未生效

环境背景：

kyuubi 版本 1.7.1
hadoop集群为华为云服务mrs 3.2.0

spark 版本为 3.1.1 （华为云mrs服务）

ranger 版本为2.0.0 （华为云mrs服务）

1. kyuubi 配置如下

[root@hw-id-bigdata-prod-kyuubi-01 conf]# cat kyuubi-defaults.conf

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

## Kyuubi Configurations

#
kyuubi.authentication                    LDAP
kyuubi.authentication.ldap.url=ldap://xxxxx.com:389
kyuubi.authentication.ldap.baseDN=cn=Users,dc=xxxx,dc=com
kyuubi.authentication.ldap.binddn=cn=admin,dc=xxxx,dc=com
kyuubi.authentication.ldap.bindpw=xxxxxxx
kyuubi.authentication.ldap.attrs=cn

#
kyuubi.frontend.bind.host                0.0.0.0
kyuubi.frontend.protocols                THRIFT_BINARY,REST
# kyuubi.frontend.thrift.binary.bind.port  10009
#
# kyuubi.engine.type                       SPARK_SQL
# kyuubi.engine.share.level                USER
# kyuubi.session.engine.initialize.timeout PT3M
#
kyuubi.ha.addresses                      10.94.0.229:2181,10.94.0.49:2181,10.94.0.74:2181
kyuubi.ha.namespace                      kyuubi
#
spark.master                             yarn
spark.yarn.queue                         default
spark.submit.deployMode                  cluster
hive.metastore.uris thrift://10.94.0.229:9083,thrift://10.94.0.49:9083

hive.exec.dynamic.partition.mode nonstrict

[root@hw-id-bigdata-prod-kyuubi-01 conf]# cat kyuubi-env.sh

export JAVA_HOME=/opt/hadoopclient/JDK/jdk1.8.0_332
export SPARK_HOME=/opt/hadoopclient/Spark2x/spark
export SPARK_CONF_DIR=/opt/hadoopclient/Spark2x/spark/conf
# export FLINK_HOME=/opt/flink
# export HIVE_HOME=/opt/hive
# export FLINK_HADOOP_CLASSPATH=/path/to/hadoop-client-runtime-3.3.2.jar:/path/to/hadoop-client-api-3.3.2.jar
# export HIVE_HADOOP_CLASSPATH=${HADOOP_HOME}/share/hadoop/common/lib/commons-collections-3.2.2.jar:${HADOOP_HOME}/share/hadoop/client/hadoop-client-runtime-3.1.0.jar:${HADOOP_HOME}/share/hadoop/client/hadoop-client-api-3.1.0.jar:${HADOOP_HOME}/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar
export HADOOP_CONF_DIR=/opt/hadoopclient/HDFS/hadoop/etc/hadoop
export YARN_CONF_DIR=/opt/hadoopclient/Yarn/config
# export KYUUBI_JAVA_OPTS="-Xmx10g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=1024m  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:./logs/kyuubi-server-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:NewRatio=3 -XX:MetaspaceSize=512m"
# export KYUUBI_BEELINE_OPTS="-Xmx2g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark"

[root@hw-id-bigdata-prod-kyuubi-01 conf]# cat /etc/profile

export JAVA_HOME=/usr/local/jdk1.8.0_221
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export MAVEN_HOME=/opt/apache-maven-3.8.8
export PATH=$MAVEN_HOME/bin:$PATH

2. 编译jar包命令为

build/mvn clean package -pl :kyuubi-spark-authz_2.12 -DskipTests -Dspark.version=3.1.1 -Dranger.version=2.0.0

将/incubator-kyuubi/extensions/spark/kyuubi-spark-authz/target/scala-2.12/jars/* 和 /incubator-kyuubi/extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_2.12-1.7.1.jar两个路径下的包拷贝到我们上面安装的spark3/jars目中,当前环境配置的spark目录为/opt/hadoopclient/Spark2x/spark/jars (此目录为kyuubi部署节点的mrs客户端的spark目录)

包含以下jar包

[root@hw-id-bigdata-prod-kyuubi-01 jars]# ls
commonj.sdo-2.1.1.jar  gethostname4j-1.0.0.jar   javax.persistence-2.1.0.jar  jersey-core-1.19.4.jar  jna-platform-5.7.0.jar          ranger-plugins-common-2.0.0.jar
eclipselink-2.5.2.jar  jackson-jaxrs-1.9.13.jar  jersey-client-1.19.4.jar     jna-5.7.0.jar           ranger-plugins-audit-2.0.0.jar

[root@hw-id-bigdata-prod-kyuubi-01 target]# ls
kyuubi-spark-authz_2.12-1.7.1.jar 

3. spark环境配置

[omm@hw-id-bigdata-prod-kyuubi-01 conf]$ cat spark-defaults.conf | grep 'spark.sql.extensions'

spark.sql.extensions = org.apache.spark.sql.hive.FISparkSessionExtension,org.apache.spark.sql.hive.CarbonInternalExtensions,org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension

[omm@hw-id-bigdata-prod-kyuubi-01 conf]$ cat ranger-spark-audit.xml

    <property>
        <name>xasecure.audit.is.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>xasecure.audit.destination.db</name>
        <value>false</value>
    </property>

    <property>
        <name>xasecure.audit.destination.db.jdbc.driver</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

    <property>
        <name>xasecure.audit.destination.db.jdbc.url</name>
        <value>jdbc:mysql://10.94.1.130/ranger_new</value>
    </property>

    <property>
        <name>xasecure.audit.destination.db.password</name>
        <value>bigdata_xx</value>
    </property>

    <property>
        <name>xasecure.audit.destination.db.user</name>
        <value>xxxxxxx</value>
    </property>

</configuration>

[omm@hw-id-bigdata-prod-kyuubi-01 conf]$ cat ranger-spark-security.xml

    <property>
        <name>ranger.plugin.spark.policy.rest.url</name>
        <value>http://hw-id-bigdata-mrs-node-master1czve.mrs-mteo.com:6080</value>
    </property>

    <property>
        <name>ranger.plugin.spark.service.name</name>
        <value>sparkServer</value>
    </property>

    <property>
        <name>ranger.plugin.spark.policy.cache.dir</name>
        <value>/opt/hadoopclient/Spark2x/spark/conf</value>
    </property>

    <property>
        <name>ranger.plugin.spark.policy.pollIntervalMs</name>
        <value>5000</value>
    </property>

    <property>
        <name>ranger.plugin.spark.policy.source.impl</name>
        <value>org.apache.ranger.admin.client.RangerAdminRESTClient</value>
    </property>

</configuration>

4. 当前现状

1.ranger 配置
新增sparkServer配置

2.内部配置如下

3.规则配置如下,拒绝此用户对所有库表的权限

4.当我使用此用户链接kyuubi以后，拒绝所有的权限并未生效

5.依然可以查看库表内容

如下

[yaooqinn]

spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension
spark.sql.extensions = org.apache.spark.sql.hive.FISparkSessionExtension,org.apache.spark.sql.hive.CarbonInternalExtensions

Late comers first?

[boombap9527]

此处已修改为
spark.sql.extensions =org.apache.spark.sql.hive.FISparkSessionExtension,org.apache.spark.sql.hive.CarbonInternalExtensions,org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension

[boombap9527]

问题并未得到解决

[BrightPter]

大佬对接上了吗，我们这边对接也碰到一些问题

[pan3793]

what's the table format? could you paste the execution plan?

[boombap9527]