1

I need to set LD_LIBRARY_PATH and CLASSPATH before running a job a cluster. In LD_LIBRARY_PATH i need to add location of some jars which are required while running the job, As these jars are available at my cluster, similar with CLASSPATH.

I have a 3 NODE cluster, I need to modify this LD_LIBRARY_PATH and CLASSPATH for all the 3 data nodes in such a way that jars available at my cluster node at added to classpath, so that the following jar are available while running the job as i am avoiding copying jar distribution while running the job to use all ready available jar on cluster nodes

1 Answer 1

1

To clarify some points in your question. If you need to add jars to the classpath then you can do this in a number of ways:

  • Hard bake the jars into your job jar (the so-called jar-in-jar approach), where each of the dependency jars is placed in a lib folder of the main job jar
  • Upload the jars to HDFS and amend your job configuration to include these jars on the classpath at map / reduce execution time (DistributedCache.addFileToClassPath(new Path("hdfs://namenode:8020/myapp/mylib.jar"), job);)
  • As above, but make the job client do the work of uploading the jar to HDFS and configuring the distributed cache (assuming you are using ToolRunner.run() to launch your job): hadoop jar myjar.jar org.MainClass -libjars dep1.jar,dep2.jar

LD_LIBRARY_PATH is used for native shared libraries (compiled C code). If you want to include .so libraries then you can use the -files option (similar to the previous -libjars example) to get the jobclient to upload the .so to HDFS, and configure each map / reduce task to include it on the java.library.path propeerty: hadoop jar myjar.jar org.MainClass -files libIOProcs.so

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .