编译 ranger

docker ranger 编译

使用ranger自带的ranger_in_docker,终于首次成功编译ranger...

 ./ranger_in_docker up

编译完自动运行ranger admin, 根据命令行提供的账号密码可以直接登录使用.

WARNING: Image for service ranger-usersync was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Starting ranger-base     ... done
Creating ranger-solr     ... done
Creating ranger-zk       ... done
Creating ranger-postgres ... done
Creating ranger          ... done
Creating ranger-usersync ... done

################### LIST OF DOCKER PROCESSES EXPOSING PORTS #####################

SERVICE:                    ranger ExposedPort:       6080
SERVICE:               ranger-solr ExposedPort:       8983
SERVICE:                 ranger-zk ExposedPort:       2181

###################################################################################


Now, You can run  access RANGER portal via http://localhost:6080 (admin/rangerR0cks!)

关闭的话, 直接以下命令即可, 减少开发机器负载

./ranger_in_docker down

ranger docker 编译信息

ranger docker编译用到的docker-compose文件在如下位置

ubuntu@geetk:/data/project/ranger/dev-support/ranger-docker$ ls
config                            docker-compose.ranger-kafka.yml             docker-compose.ranger-usersync.yml  Dockerfile.ranger-hive      Dockerfile.ranger-tagsync   README.md
dist                              docker-compose.ranger-kms.yml               docker-compose.ranger.yml           Dockerfile.ranger-kafka     Dockerfile.ranger-usersync  scripts
docker-compose.ranger-base.yml    docker-compose.ranger-knox.yml              Dockerfile.ranger                   Dockerfile.ranger-kms       Dockerfile.ranger-zk
docker-compose.ranger-build.yml   docker-compose.ranger-mysql.yml             Dockerfile.ranger-base              Dockerfile.ranger-knox      download-archives.sh
docker-compose.ranger-hadoop.yml  docker-compose.ranger-postgres-mounted.yml  Dockerfile.ranger-build             Dockerfile.ranger-mysql     downloads
docker-compose.ranger-hbase.yml   docker-compose.ranger-postgres.yml          Dockerfile.ranger-hadoop            Dockerfile.ranger-postgres  patches
docker-compose.ranger-hive.yml    docker-compose.ranger-tagsync.yml           Dockerfile.ranger-hbase             Dockerfile.ranger-solr      postgres-db-mount
ubuntu@geetk:/data/project/ranger/dev-support/ranger-docker$ ls dist/
ranger-3.0.0-SNAPSHOT-admin.tar.gz                 ranger-3.0.0-SNAPSHOT-kylin-plugin.tar.gz         ranger-3.0.0-SNAPSHOT-sqoop-plugin.tar.gz
ranger-3.0.0-SNAPSHOT-atlas-plugin.tar.gz          ranger-3.0.0-SNAPSHOT-migration-util.tar.gz       ranger-3.0.0-SNAPSHOT-src.tar.gz
ranger-3.0.0-SNAPSHOT-elasticsearch-plugin.tar.gz  ranger-3.0.0-SNAPSHOT-ozone-plugin.tar.gz         ranger-3.0.0-SNAPSHOT-storm-plugin.tar.gz
ranger-3.0.0-SNAPSHOT-hbase-plugin.tar.gz          ranger-3.0.0-SNAPSHOT-presto-plugin.tar.gz        ranger-3.0.0-SNAPSHOT-tagsync.tar.gz
ranger-3.0.0-SNAPSHOT-hdfs-plugin.tar.gz           ranger-3.0.0-SNAPSHOT-ranger-tools.tar.gz         ranger-3.0.0-SNAPSHOT-usersync.tar.gz
ranger-3.0.0-SNAPSHOT-hive-plugin.tar.gz           ranger-3.0.0-SNAPSHOT-schema-registry-plugin.jar  ranger-3.0.0-SNAPSHOT-yarn-plugin.tar.gz
ranger-3.0.0-SNAPSHOT-kafka-plugin.tar.gz          ranger-3.0.0-SNAPSHOT-solr_audit_conf.tar.gz      version
ranger-3.0.0-SNAPSHOT-kms.tar.gz                   ranger-3.0.0-SNAPSHOT-solr_audit_conf.zip
ranger-3.0.0-SNAPSHOT-knox-plugin.tar.gz           ranger-3.0.0-SNAPSHOT-solr-plugin.tar.gz

构建后成功运行的docker容器列表

docker ps

CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                  PORTS                                                                                  NAMES
c24bf49ee2f0   ranger-usersync               "/home/ranger/script…"   3 hours ago    Up 3 hours                                                                                                     ranger-usersync
43f2a269e590   ranger:latest                 "/home/ranger/script…"   3 hours ago    Up 3 hours              0.0.0.0:6080->6080/tcp, :::6080->6080/tcp                                              ranger
8b765806fa19   ranger-postgres               "docker-entrypoint.s…"   3 hours ago    Up 3 hours (healthy)    5432/tcp                                                                               ranger-postgres
4047cccd9646   ranger-zk                     "/docker-entrypoint.…"   3 hours ago    Up 3 hours              2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp                ranger-zk
61d87051a006   ranger-solr                   "docker-entrypoint.s…"   3 hours ago    Up 3 hours              0.0.0.0:8983->8983/tcp, :::8983->8983/tcp                                              ranger-solr

docker-build 要求docker-compose为高版本

ubuntu18安装的docker版本位1.17, 不支持docker-compose文件network里的name属性.

docker-compose version
docker-compose version 1.17.1, build unknown
docker-py version: 2.5.1
CPython version: 2.7.17
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

ubuntu@VM-0-4-ubuntu:/data/projects/ranger$ ./ranger_in_docker up
file already in cache: hadoop-3.3.0.tar.gz
file already in cache: hbase-2.4.6-bin.tar.gz
file already in cache: kafka_2.12-2.8.1.tgz
file already in cache: apache-hive-3.1.2-bin.tar.gz
file already in cache: hadoop-3.1.1.tar.gz
file already in cache: postgresql-42.2.16.jre7.jar
file already in cache: mysql-connector-java-8.0.28.jar
file already in cache: log4jdbc-1.2.jar
file already in cache: knox-1.4.0.tar.gz
+ docker-compose -f docker-compose.ranger-base.yml build --no-cache
ERROR: The Compose file './docker-compose.ranger-base.yml' is invalid because:
networks.ranger value Additional properties are not allowed ('name' was unexpected)
+ docker-compose -f docker-compose.ranger-base.yml -f docker-compose.ranger-build.yml down --remove-orphans
ERROR: The Compose file './docker-compose.ranger-base.yml' is invalid because:
networks.ranger value Additional properties are not allowed ('name' was unexpected)
+ docker-compose -f docker-compose.ranger-base.yml -f docker-compose.ranger-build.yml up
ERROR: The Compose file './docker-compose.ranger-base.yml' is invalid because:
networks.ranger value Additional properties are not allowed ('name' was unexpected)
ERROR: Unable to complete RANGER build using DOCKER. Exiting ....

升级docker-compose方法:

https://stackoverflow.com/questions/58155523/unable-to-give-network-name-in-docker-compose

ubuntu22安装的版本为1.29, 直接运行则没有问题.

docker-compose version
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

ranger docker-compose 参考

/data/projects/ranger/dev-support/ranger-docker

cat docker-compose.ranger-base.yml

version: '3'
services:
  ranger-base:
    build:
      context: .
      dockerfile: Dockerfile.ranger-base
      args:
        - UBUNTU_VERSION=${UBUNTU_VERSION}
        - RANGER_BASE_JAVA_VERSION=${RANGER_BASE_JAVA_VERSION}
    image: ranger-base
    container_name: ranger-base
    networks:
      - ranger
    environment:
      - RANGER_VERSION

networks:
  ranger:
    name: rangernw

mvn 编译 ranger

多次重复后成功编译ranger的maven命令, 绕过了jstest和rat证书验证.

 mvn clean compile package install  -DskipJSTests -Drat.skip=true

ranger编译耗时超长, 经常出现问题, 每次解决后可以从中断处重新编译

mvn install -rf :ranger-ozone-plugin  -DskipJSTests -Drat.skip=true

其中ranger-ozone-plugin为ranger中的一个submodule. mvn编译失败的时候会提示, 比如

[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :security-admin-web

ranger 编译安装phantomjs依赖

参考这两个文档, 顺利安装phantomjs依赖, mvn可以成功编译.

How to install PhantomJS on Ubuntu

https://gist.github.com/julionc/7476620

报错Auto configuration failed, 可以参考下面连接解决:

https://github.com/ariya/phantomjs/issues/15449

Just Uncomment the following line putting # in /etc/ssl/openssl.cnf

[default_conf]
#ssl_conf = ssl_sect

compile errors

rat error

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project ranger: Too many files with unapproved license: 1 See RAT report in: /root/projects/ranger/target/rat.txt -> [Help 1]

解决方法：增加 -Drat.skip=true 参数，跳过licensing 检查。

js error

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:karma (karma dev) on project security-admin-web: Failed to run task: 'karma start ../../src/test/javascript/karma-dev.conf.js' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :security-admin-web

readme里有讲, 需要安装PhantomJS依赖.

mvn clean compile package install

mvn eclipse:eclipse

Ranger Admin UI tests depend on PhantomJS. If the build fails with npm or Karma errors you can either:

install PhantomJS dependencies for your platform (bzip2 and fontconfig)
skip JavaScript test execution: mvn -DskipJSTests ...

编译需要使用jdk1.8版本

jdk 11 版本执行ranger mvn, 报错

ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.003 s <<< FAILURE! - in org.apache.ranger.plugin.util.RangerRequestExprResolverTest
[ERROR] org.apache.ranger.plugin.util.RangerRequestExprResolverTest.testRequestAttributes  Time elapsed: 0.001 s  <<< ERROR!
java.lang.Error: 
Unresolved compilation problems: 
        NashornScriptEngineFactory cannot be resolved to a type
        NashornScriptEngineFactory cannot be resolved to a type

        at org.apache.ranger.plugin.util.NashornScriptEngineCreator.getScriptEngine(NashornScriptEngineCreator.java:44)
        at org.apache.ranger.plugin.util.ScriptEngineUtil.initScriptEngineCreator(ScriptEngineUtil.java:118)
        at org.apache.ranger.plugin.util.ScriptEngineUtil.getScriptEngineCreator(ScriptEngineUtil.java:87)
        at org.apache.ranger.plugin.util.ScriptEngineUtil.createScriptEngine(ScriptEngineUtil.java:52)
        at org.apache.ranger.plugin.util.RangerRequestExprResolver.resolveExpressions(RangerRequestExprResolver.java:102)
        at org.apache.ranger.plugin.util.RangerRequestExprResolverTest.testRequestAttributes(RangerRequestExprResolverTest.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project ranger-plugins-common: Compilation failure: Compilation failure: 
[ERROR] /home/coder/project/playground/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[26,32] error: package jdk.nashorn.api.scripting does not exist
[ERROR] /home/coder/project/playground/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[27,32] error: package jdk.nashorn.api.scripting does not exist
[ERROR] /home/coder/project/playground/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[54,54] error: cannot find symbol
[ERROR]   symbol:   class ClassFilter
[ERROR]   location: class NashornScriptEngineCreator
[ERROR] /home/coder/project/playground/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[44,12] error: cannot find symbol
[ERROR]   symbol:   class NashornScriptEngineFactory
[ERROR]   location: class NashornScriptEngineCreator
[ERROR] /home/coder/project/playground/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[44,53] error: cannot find symbol
[ERROR]   symbol:   class NashornScriptEngineFactory
[ERROR]   location: class NashornScriptEngineCreator

maven 版本要求大于等于3.6.3

mvn编译报错: 3.6.0 is not in the allowed range 3.6.3

[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ ranger ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireMavenVersion failed with message:
Detected Maven Version: 3.6.0 is not in the allowed range 3.6.3.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-versions) on
project ranger: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.
-> [Help 1]

使用sdkman安装不同版本的maven

sdk install maven 3.6.3

created at 2023-07-12

编译 ranger

docker ranger 编译​

ranger docker 编译信息​

docker-build 要求docker-compose为高版本​

ranger docker-compose 参考​

mvn 编译 ranger​

ranger 编译安装phantomjs依赖​

compile errors​

rat error​

js error​

编译需要使用jdk1.8版本​

maven 版本要求大于等于3.6.3​