Monday, December 15, 2014

Hadoop native-hadoop library problem

If you install Hadoop using the Apache pre-built distribution, and running it on a 64-bit platform, you may get the following warning message every time you run some command. It doesn't affect the query result but quite annoying to some:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

If we set the logging level:
export HADOOP_ROOT_LOGGER=DEBUG,console

You see it’s complaining about the word width of libhadoop.so.1.0.0:

14/12/15 16:40:05 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: /local/home/mzhang/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0: /local/home/mzhang/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0: wrong ELF class: ELFCLASS32 (Possible cause: architecture word width mismatch)
14/12/15 16:40:05 DEBUG util.NativeCodeLoader: java.library.path=/local/home/mzhang/hadoop-2.4.1/lib/native
14/12/15 16:40:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


go to $HADOOP_HOME/lib/native:

[root@pool-12 native]# file *
libhadoop.a:        current ar archived
libhadooppipes.a:   current ar archive
libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'
libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
libhadooputils.a:   current ar archive
libhdfs.a:          current ar archive
libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'
libhdfs.so.0.0.0:   ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped


We can see the so files are 32-bit. I don't know why Apache still provides them in 32-bit nowadays when most production systems are running in 64-bit. But anyway, we can rebuild the hadoop source on 64 bit platform easily, then replace these two so files in your $HADOOP_HOME. We’re good to go! This time you'll see the log message as:

14/12/15 16:26:43 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
14/12/15 16:26:43 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library


Remember to setup these LIB parameters in the setup file:
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"


Finally switch back the HADOOP_ROOT_LOGGER to default:
export HADOOP_ROOT_LOGGER=INFO,console’

The default value setup can be found in: $HADOOP_HOME/etc/hadoop/log4j.properties
hadoop.root.logger=INFO,console

0 Comments:

Post a Comment

<< Home