Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to set LD_LIBRARY_PATH and CLASSPATH before running a job a cluster. In LD_LIBRARY_PATH i need to add location of some jars which are required while running the job, As these jars are available at my cluster, similar with CLASSPATH.

I have a 3 NODE cluster, I need to modify this LD_LIBRARY_PATH and CLASSPATH for all the 3 data nodes in such a way that jars available at my cluster node at added to classpath, so that the following jar are available while running the job as i am avoiding copying jar distribution while running the job to use all ready available jar on cluster nodes

share|improve this question

migrated from stackoverflow.com Dec 1 '12 at 9:08

This question came from our site for professional and enthusiast programmers.

1 Answer 1

To clarify some points in your question. If you need to add jars to the classpath then you can do this in a number of ways:

  • Hard bake the jars into your job jar (the so-called jar-in-jar approach), where each of the dependency jars is placed in a lib folder of the main job jar
  • Upload the jars to HDFS and amend your job configuration to include these jars on the classpath at map / reduce execution time (DistributedCache.addFileToClassPath(new Path("hdfs://namenode:8020/myapp/mylib.jar"), job);)
  • As above, but make the job client do the work of uploading the jar to HDFS and configuring the distributed cache (assuming you are using ToolRunner.run() to launch your job): hadoop jar myjar.jar org.MainClass -libjars dep1.jar,dep2.jar

LD_LIBRARY_PATH is used for native shared libraries (compiled C code). If you want to include .so libraries then you can use the -files option (similar to the previous -libjars example) to get the jobclient to upload the .so to HDFS, and configure each map / reduce task to include it on the java.library.path propeerty: hadoop jar myjar.jar org.MainClass -files libIOProcs.so

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.