Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am new with pig and I try to learn from pig tutorial on Apache website. I am working with hadoop 1.0.1 and PigVersion 0.11.1. In the tutorial, it is proposed to try pig on two examples (called script1-local.pig and script2-local.pig). But when I try to run the first example with the following CLI:

pig -x local script1-local.pig

I get the following error: (It seems to deal with Java heap size...)

2013-07-16 15:26:59,033 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2013-07-16 15:26:59,033 [main] INFO  org.apache.pig.Main - Logging error messages to: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:26:59,246 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2013-07-16 15:26:59,332 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-07-16 15:27:00,287 [main] WARN  org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 3 time(s).
2013-07-16 15:27:00,287 [main] WARN  org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 3 time(s).
2013-07-16 15:27:00,300 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,DISTINCT,FILTER
2013-07-16 15:27:00,446 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-07-16 15:27:00,469 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-07-16 15:27:00,492 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5
2013-07-16 15:27:00,493 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5
2013-07-16 15:27:00,518 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-07-16 15:27:00,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-07-16 15:27:00,536 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2013-07-16 15:27:00,537 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=208348
2013-07-16 15:27:00,537 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2013-07-16 15:27:00,557 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp directory: /tmp/1373988420563-0
2013-07-16 15:27:00,564 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting identity combiner class.
2013-07-16 15:27:00,637 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-07-16 15:27:00,648 [JobControl] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-07-16 15:27:00,652 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
****file:/root/pigtmp/excite-small.log
2013-07-16 15:27:00,681 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-07-16 15:27:00,681 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-07-16 15:27:00,688 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-07-16 15:27:00,922 [Thread-3] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2013-07-16 15:27:00,925 [Thread-3] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b4b0967
2013-07-16 15:27:00,936 [Thread-3] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/root/pigtmp/excite-small.log:0+208348
2013-07-16 15:27:00,942 [Thread-3] INFO  org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
2013-07-16 15:27:01,059 [Thread-3] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:674)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases clean1,clean2,houred,ngramed1,raw
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: raw[28,6],clean1[31,9],clean2[-1,-1],houred[39,9],ngramed1[42,11] C:  R: 
2013-07-16 15:27:01,145 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-07-16 15:27:01,148 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2013-07-16 15:27:01,148 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2013-07-16 15:27:01,149 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-07-16 15:27:01,149 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-07-16 15:27:01,149 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
2013-07-16 15:27:01,150 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
1.0.1   0.11.1  root    2013-07-16 15:27:00 2013-07-16 15:27:01 GROUP_BY,ORDER_BY,DISTINCT,FILTER

Failed!

Failed Jobs:
JobId   Alias   Feature Message Outputs
job_local_0001  clean1,clean2,houred,ngramed1,raw   DISTINCT    Message: Job failed! Error - NA 

Input(s):
Failed to read data from "file:///root/pigtmp/excite-small.log"

Output(s):

Job DAG:
job_local_0001  ->  null,
null    ->  null,
null    ->  null,
null    ->  null,
null


2013-07-16 15:27:01,150 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2013-07-16 15:27:01,152 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,153 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,154 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,154 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,155 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log

As anyone encounter the same problem?

Thank you.

share|improve this question
add comment

1 Answer 1

As is referred in this answer, you could try to increase the maximum memory allocated space:

export HADOOP_CLIENT_OPTS="-Xmx1024m"
share|improve this answer
1  
Welcome to Superuser. Please don't provide link-only answers. Linked pages can change or move and this will make your answer useless to future visitors. Instead, add in the answer contained in the link or, if it is too large, provide a precis or abridged version along with the link. –  Tog Sep 22 '13 at 9:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.