Version

    Kerberos Authentication for Hadoop

    For user authentication in Hadoop, CloverDX can use the Kerberos authentication protocol.

    To use Kerberos, you have to set up your Java, project and HDFS connection. For more information, see Kerberos requirements and setting.

    Note that the following instructions are applicable for Tomcat application server and Unix-like systems.

    Java Setting

    There are several ways of setting Java for Kerberos. In the case of the first two options (configuration via system properties and via configuration file), you must modify both setenv.sh in CloverDX Server and CloverDXDesigner.ini in CloverDX Designer.

    Additionally, add the parameters in CloverDX Designer to Window  Preferences  CloverDX Runtime  VM parameters pane.

    • Configuration via system properties Set the Java system property java.security.krb5.realm to the name of your Kerberos realm, for example:

      -Djava.security.krb5.realm=EXAMPLE.COM

      Set the Java system property java.security.krb5.kdc to the hostname of your Kerberos key distribution center, for example:

      -Djava.security.krb5.kdc=kerberos.example.com

    • Configuration via config file Set the Java system property java.security.krb5.conf to point to the location of your Kerberos configuration file, for example:

      -Djava.security.krb5.conf="/path/to/krb5.conf"

    • Configuration via config file in Java installation directory Put the krb5.conf file into the %JAVA_HOME%/lib/security directory, e.g. /opt/jdk-11.0.14+9/lib/security/krb5.conf.

      Project Setting

    • Copy the .keytab file into the project, e.g. conn/clover.keytab.

    Connection Setting

    Kerberos authentication requires the hadoop-auth-*.jar library on both HDFS + MapReduce and Hive connection classpath.

    • HDFS and MapReduce Connection

      1. Set Username to the principal name, e.g. clover/clover@EXAMPLE.COM.

      2. Set the following parameters in the Hadoop Parameters pane:

        cloveretl.hadoop.kerberos.keytab=${CONN_DIR}/clover.keytab
hadoop.security.authentication=Kerberos
yarn.resourcemanager.principal=yarn/_HOST@EXAMPLE.COM
        Example 10. Properties needed to connect to a Hadoop High Availability (HA) cluster in Hadoop connection
        mapreduce.app-submission.cross-platform\=true

yarn.application.classpath\=\:$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*, $HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*, $HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*, $HADOOP_YARN_HOME/lib/*\:
yarn.app.mapreduce.am.resource.mb\=512
mapreduce.map.memory.mb\=512
mapreduce.reduce.memory.mb\=512
mapreduce.framework.name\=yarn
yarn.log.aggregation-enable\=true

mapreduce.jobhistory.address\=example.com\:port

yarn.resourcemanager.ha.enabled\=true
yarn.resourcemanager.ha.rm-ids\=rm1,rm2
yarn.resourcemanager.hostname.rm1\=example.com
yarn.resourcemanager.hostname.rm2\=example.com
yarn.resourcemanager.scheduler.address.rm1\=example.com\:port
yarn.resourcemanager.scheduler.address.rm2\=example.com\:port

fs.permissions.umask-mode\=000
fs.defaultFS\=hdfs\://nameservice1
fs.default.name\=hdfs\://nameservice1
fs.nameservices\=nameservice1
fs.ha.namenodes.nameservice1\=namenode1,namenode2
fs.namenode.rpc-address.nameservice1.namenode1\=example.com\:port
fs.namenode.rpc-address.nameservice1.namenode2\=example.com\:port
fs.client.failover.proxy.provider.nameservice1\=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

type=HADOOP
host=nameservice1
username=clover/clover@EXAMPLE.COM
hostMapred=Not needed for YARN

        The _HOST string in yarn/_HOST@EXAMPLE.COM and hive/_HOST@EXAMPLE.COM is a placeholder that will be automatically replaced with an actual hostname. This is the recommended way that will work even with high-availability Hadoop cluster setup.

      3. If you encounter an error: No common protection layer between client and server set the hadoop.rpc.protection parameter to match your Hadoop cluster configuration.

    • Hive Connection

      1. Add ;principal=hive/_HOST@EXAMPLE.COM to the URL, e.g. jdbc:hive2://hive.example.com:10000/default;principal=hive/_HOST@EXAMPLE.COM

      2. Set User to the principal name, e.g. clover/clover@EXAMPLE.COM

      3. Set cloveretl.hadoop.kerberos.keytab=${CONN_DIR}/clover.keytab in Advanced JDBC properties.