跳到主要内容

ranger-hdfs-hive-测试-tips

开启 ranger hdfs audit 确认需要的 hdfs 权限​

在新环境操作hive, 需要给予各种hdfs默认权限才行. 可以开启ranger的hdfs和hive audit审计记录, 在出现错误的时候可以及时观察需要授予什么hdfs权限.

甚至在beeline连接的时候, 也会应为没有一些hdfs路径权限而连接出错, 因为hive需要读取一些默认的hdfs路径.

这时候看ranger audit审计信息就能知道细节, 不然还一直以为是kerberos的参数有误, 或是密码出现错误.

picture 0

因为没有授予/emr/hive/hive.jceks权限, 导致kerberos环境下beeline连接出现错误.

[root@172 ~]# beeline -u "jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/172.16.16.3@EMR-1MSO7OJ3"
which: no hbase in (/root/.pyenv/bin:/usr/local/service/starrocks/bin:/data/Impala/shell:/usr/local/service/kudu/bin:/usr/local/service/tez/bin:/usr/local/jdk/bin:/usr/local/service/hadoop/bin:/usr/local/service/hive/bin:/usr/local/service/hbase/bin:/usr/local/service/spark/bin:/usr/local/service/storm/bin:/usr/local/service/sqoop/bin:/usr/local/service/kylin/bin:/usr/local/service/alluxio/bin:/usr/local/service/flink/bin:/data/Impala/bin:/usr/local/service/oozie/bin:/usr/local/service/presto/bin:/usr/local/service/slider/bin:/usr/local/service/kudu/bin:/usr/local/jdk//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/service/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/172.16.16.3@EMR-1MSO7OJ3
23/08/11 16:56:04 [main]: WARN jdbc.HiveConnection: Failed to connect to 172.16.16.3:7001
Error: Could not open client transport with JDBC Uri: jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/172.16.16.3@EMR-1MSO7OJ3: Failed to open new session: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException (state=08S01,code=0)

查看ranger hdfs audit后, 给对应用户授予hdfs路径权限, beeline成功连接.


[root@172 ~]# beeline -u "jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/_HOST@EMR-1MSO7OJ3"
which: no hbase in (/root/.pyenv/bin:/usr/local/service/starrocks/bin:/data/Impala/shell:/usr/local/service/kudu/bin:/usr/local/service/tez/bin:/usr/local/jdk/bin:/usr/local/service/hadoop/bin:/usr/local/service/hive/bin:/usr/local/service/hbase/bin:/usr/local/service/spark/bin:/usr/local/service/storm/bin:/usr/local/service/sqoop/bin:/usr/local/service/kylin/bin:/usr/local/service/alluxio/bin:/usr/local/service/flink/bin:/data/Impala/bin:/usr/local/service/oozie/bin:/usr/local/service/presto/bin:/usr/local/service/slider/bin:/usr/local/service/kudu/bin:/usr/local/jdk//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/service/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/_HOST@EMR-1MSO7OJ3
Connected to: Apache Hive (version 3.1.3)
Driver: Hive JDBC (version 3.1.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.3 by Apache Hive
0: jdbc:hive2://172.16.16.3:7001/default> select current_user();
+------+
| _c0  |
+------+
| gee  |
+------+
1 row selected (0.264 seconds)
0: jdbc:hive2://172.16.16.3:7001/default>

kerberos环境下连接beeline方法​

使用kinit -kt登陆, 然后beeline连接.

hive连接路径上的kerberos principal来自hive的xml配置.

[root@tbds-172-16-16-11 ~]# klist  -kt /tmp/gee.keytab 
Keytab name: FILE:/tmp/gee.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 08/02/2023 15:55:54 gee@TBDS-HBMGJTQZ
   1 08/02/2023 15:55:54 gee@TBDS-HBMGJTQZ

[root@tbds-172-16-16-11 ~]# kinit  -kt /tmp/gee.keytab  gee
[root@tbds-172-16-16-11 ~]# 
[root@tbds-172-16-16-11 ~]# klist 
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: gee@TBDS-HBMGJTQZ

Valid starting       Expires              Service principal
08/02/2023 15:58:14  08/03/2023 15:58:14  krbtgt/TBDS-HBMGJTQZ@TBDS-HBMGJTQZ
        renew until 08/05/2023 15:58:14


beeline -u "jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/172.16.16.3@EMR-1MSO7OJ3"
which: no hbase in (/root/.pyenv/bin:/usr/local/service/starrocks/bin:/data/Impala/shell:/usr/local/service/kudu/bin:/usr/local/service/tez/bin:/usr/local/jdk/bin:/usr/local/service/hadoop/bin:/usr/local/service/hive/bin:/usr/local/service/hbase/bin:/usr/local/service/spark/bin:/usr/local/service/storm/bin:/usr/local/service/sqoop/bin:/usr/local/service/kylin/bin:/usr/local/service/alluxio/bin:/usr/local/service/flink/bin:/data/Impala/bin:/usr/local/service/oozie/bin:/usr/local/service/presto/bin:/usr/local/service/slider/bin:/usr/local/service/kudu/bin:/usr/local/jdk//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/service/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://172.16.16.3:7001/default;principal=hadoop/172.16.16.3@EMR-1MSO7OJ3
Connected to: Apache Hive (version 3.1.3)
Driver: Hive JDBC (version 3.1.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.3 by Apache Hive
0: jdbc:hive2://172.16.16.3:7001/default>
0: jdbc:hive2://172.16.16.3:7001/default> select current_user();
+------+
| _c0  |
+------+
| gee  |
+------+
1 row selected (3.372 seconds)
created at 2023-08-11