0

I have a 50 GB MapReduce job which is (basically) a WordCount app and it has the following Map/Reduce job percentages (listed at the end of the question). It seems like the reducers want to wait until the mappers are completely finished to start working. Is this normal behavior? if not, how do I go about troubleshooting why this is happening and change it?

Looking at the ending reduce percentages, it seems like it is not a huge impact to wait until the end because the Reduce portion takes about 5 minutes while the Map portion takes about 35 Minutes, but those 5 Minutes would be nice to shave off if I could get the reducers to be working while the Mappers are doing their thing.

15/02/09 09:14:38 INFO mapred.JobClient:  map 0% reduce 0%
15/02/09 09:17:08 INFO mapred.JobClient:  map 1% reduce 0%
15/02/09 09:18:04 INFO mapred.JobClient:  map 2% reduce 0%
15/02/09 09:18:34 INFO mapred.JobClient:  map 3% reduce 0%
15/02/09 09:18:51 INFO mapred.JobClient:  map 4% reduce 0%
15/02/09 09:19:10 INFO mapred.JobClient:  map 5% reduce 0%
15/02/09 09:19:30 INFO mapred.JobClient:  map 6% reduce 0%
15/02/09 09:19:48 INFO mapred.JobClient:  map 7% reduce 0%
15/02/09 09:20:02 INFO mapred.JobClient:  map 7% reduce 1%
15/02/09 09:20:07 INFO mapred.JobClient:  map 8% reduce 1%
15/02/09 09:20:37 INFO mapred.JobClient:  map 9% reduce 1%
15/02/09 09:20:49 INFO mapred.JobClient:  map 9% reduce 2%
15/02/09 09:20:54 INFO mapred.JobClient:  map 10% reduce 2%
15/02/09 09:20:58 INFO mapred.JobClient:  map 10% reduce 3%
15/02/09 09:21:08 INFO mapred.JobClient:  map 11% reduce 3%
15/02/09 09:21:25 INFO mapred.JobClient:  map 12% reduce 3%
15/02/09 09:21:47 INFO mapred.JobClient:  map 13% reduce 3%
15/02/09 09:22:09 INFO mapred.JobClient:  map 14% reduce 3%
15/02/09 09:22:23 INFO mapred.JobClient:  map 14% reduce 4%
15/02/09 09:22:30 INFO mapred.JobClient:  map 15% reduce 4%
15/02/09 09:22:47 INFO mapred.JobClient:  map 16% reduce 4%
15/02/09 09:22:57 INFO mapred.JobClient:  map 16% reduce 5%
15/02/09 09:23:09 INFO mapred.JobClient:  map 17% reduce 5%
15/02/09 09:23:19 INFO mapred.JobClient:  map 18% reduce 5%
15/02/09 09:23:36 INFO mapred.JobClient:  map 19% reduce 5%
15/02/09 09:23:55 INFO mapred.JobClient:  map 20% reduce 5%
15/02/09 09:24:19 INFO mapred.JobClient:  map 21% reduce 5%
15/02/09 09:24:38 INFO mapred.JobClient:  map 22% reduce 5%
15/02/09 09:24:57 INFO mapred.JobClient:  map 23% reduce 5%
15/02/09 09:25:10 INFO mapred.JobClient:  map 24% reduce 5%
15/02/09 09:25:27 INFO mapred.JobClient:  map 25% reduce 5%
15/02/09 09:25:51 INFO mapred.JobClient:  map 26% reduce 5%
15/02/09 09:26:09 INFO mapred.JobClient:  map 27% reduce 5%
15/02/09 09:26:19 INFO mapred.JobClient:  map 28% reduce 5%
15/02/09 09:26:35 INFO mapred.JobClient:  map 29% reduce 5%
15/02/09 09:26:49 INFO mapred.JobClient:  map 30% reduce 5%
15/02/09 09:27:06 INFO mapred.JobClient:  map 31% reduce 5%
15/02/09 09:27:18 INFO mapred.JobClient:  map 32% reduce 5%
15/02/09 09:27:42 INFO mapred.JobClient:  map 33% reduce 5%
15/02/09 09:27:51 INFO mapred.JobClient:  map 34% reduce 5%
15/02/09 09:28:07 INFO mapred.JobClient:  map 35% reduce 5%
15/02/09 09:28:26 INFO mapred.JobClient:  map 36% reduce 5%
15/02/09 09:28:53 INFO mapred.JobClient:  map 37% reduce 5%
15/02/09 09:29:10 INFO mapred.JobClient:  map 38% reduce 5%
15/02/09 09:29:19 INFO mapred.JobClient:  map 39% reduce 5%
15/02/09 09:29:37 INFO mapred.JobClient:  map 40% reduce 5%
15/02/09 09:29:57 INFO mapred.JobClient:  map 41% reduce 5%
15/02/09 09:30:13 INFO mapred.JobClient:  map 42% reduce 5%
15/02/09 09:30:26 INFO mapred.JobClient:  map 43% reduce 5%
15/02/09 09:30:47 INFO mapred.JobClient:  map 44% reduce 5%
15/02/09 09:31:03 INFO mapred.JobClient:  map 45% reduce 5%
15/02/09 09:31:12 INFO mapred.JobClient:  map 46% reduce 5%
15/02/09 09:31:30 INFO mapred.JobClient:  map 47% reduce 5%
15/02/09 09:31:40 INFO mapred.JobClient:  map 48% reduce 5%
15/02/09 09:31:59 INFO mapred.JobClient:  map 49% reduce 5%
15/02/09 09:32:15 INFO mapred.JobClient:  map 50% reduce 5%
15/02/09 09:32:28 INFO mapred.JobClient:  map 51% reduce 5%
15/02/09 09:32:45 INFO mapred.JobClient:  map 52% reduce 5%
15/02/09 09:32:56 INFO mapred.JobClient:  map 53% reduce 5%
15/02/09 09:33:18 INFO mapred.JobClient:  map 54% reduce 5%
15/02/09 09:33:38 INFO mapred.JobClient:  map 55% reduce 5%
15/02/09 09:33:40 INFO mapred.JobClient:  map 55% reduce 0%
15/02/09 09:33:51 INFO mapred.JobClient: Task Id : attempt_201306131151_3706_r_000000_0, Status : FAILED
Task attempt_201306131151_3706_r_000000_0 failed to report status for 600 seconds. Killing!
15/02/09 09:33:55 INFO mapred.JobClient:  map 56% reduce 0%
15/02/09 09:34:08 INFO mapred.JobClient:  map 57% reduce 0%
15/02/09 09:34:35 INFO mapred.JobClient:  map 58% reduce 0%
15/02/09 09:34:44 INFO mapred.JobClient:  map 58% reduce 1%
15/02/09 09:35:02 INFO mapred.JobClient:  map 59% reduce 1%
15/02/09 09:35:18 INFO mapred.JobClient:  map 60% reduce 1%
15/02/09 09:35:25 INFO mapred.JobClient:  map 60% reduce 2%
15/02/09 09:35:39 INFO mapred.JobClient:  map 61% reduce 2%
15/02/09 09:36:06 INFO mapred.JobClient:  map 62% reduce 3%
15/02/09 09:36:25 INFO mapred.JobClient:  map 63% reduce 3%
15/02/09 09:36:49 INFO mapred.JobClient:  map 63% reduce 4%
15/02/09 09:36:52 INFO mapred.JobClient:  map 64% reduce 4%
15/02/09 09:37:07 INFO mapred.JobClient:  map 65% reduce 4%
15/02/09 09:37:31 INFO mapred.JobClient:  map 66% reduce 4%
15/02/09 09:37:51 INFO mapred.JobClient:  map 67% reduce 4%
15/02/09 09:38:10 INFO mapred.JobClient:  map 68% reduce 4%
15/02/09 09:38:19 INFO mapred.JobClient:  map 69% reduce 4%
15/02/09 09:38:43 INFO mapred.JobClient:  map 70% reduce 4%
15/02/09 09:39:03 INFO mapred.JobClient:  map 71% reduce 4%
15/02/09 09:39:24 INFO mapred.JobClient:  map 72% reduce 4%
15/02/09 09:39:42 INFO mapred.JobClient:  map 73% reduce 4%
15/02/09 09:40:00 INFO mapred.JobClient:  map 74% reduce 4%
15/02/09 09:40:29 INFO mapred.JobClient:  map 75% reduce 4%
15/02/09 09:41:13 INFO mapred.JobClient:  map 76% reduce 4%
15/02/09 09:41:31 INFO mapred.JobClient:  map 77% reduce 4%
15/02/09 09:41:54 INFO mapred.JobClient:  map 78% reduce 4%
15/02/09 09:42:06 INFO mapred.JobClient:  map 79% reduce 4%
15/02/09 09:42:31 INFO mapred.JobClient:  map 80% reduce 4%
15/02/09 09:43:02 INFO mapred.JobClient:  map 81% reduce 4%
15/02/09 09:43:28 INFO mapred.JobClient:  map 82% reduce 4%
15/02/09 09:43:53 INFO mapred.JobClient:  map 83% reduce 4%
15/02/09 09:44:07 INFO mapred.JobClient:  map 84% reduce 4%
15/02/09 09:44:23 INFO mapred.JobClient:  map 85% reduce 4%
15/02/09 09:44:36 INFO mapred.JobClient:  map 86% reduce 4%
15/02/09 09:44:49 INFO mapred.JobClient:  map 87% reduce 4%
15/02/09 09:45:15 INFO mapred.JobClient:  map 88% reduce 4%
15/02/09 09:45:42 INFO mapred.JobClient:  map 89% reduce 4%
15/02/09 09:45:58 INFO mapred.JobClient:  map 90% reduce 4%
15/02/09 09:46:28 INFO mapred.JobClient:  map 91% reduce 4%
15/02/09 09:46:42 INFO mapred.JobClient:  map 92% reduce 4%
15/02/09 09:46:57 INFO mapred.JobClient:  map 93% reduce 4%
15/02/09 09:47:16 INFO mapred.JobClient:  map 94% reduce 4%
15/02/09 09:47:28 INFO mapred.JobClient:  map 95% reduce 4%
15/02/09 09:47:45 INFO mapred.JobClient:  map 96% reduce 4%
15/02/09 09:48:09 INFO mapred.JobClient:  map 97% reduce 4%
15/02/09 09:48:29 INFO mapred.JobClient:  map 98% reduce 4%
15/02/09 09:48:31 INFO mapred.JobClient:  map 98% reduce 0%
15/02/09 09:48:38 INFO mapred.JobClient:  map 99% reduce 0%
15/02/09 09:48:44 INFO mapred.JobClient: Task Id : attempt_201306131151_3706_r_000000_1, Status : FAILED
Task attempt_201306131151_3706_r_000000_1 failed to report status for 600 seconds. Killing!
15/02/09 09:49:16 INFO mapred.JobClient:  map 99% reduce 1%
15/02/09 09:49:25 INFO mapred.JobClient:  map 99% reduce 2%
15/02/09 09:49:31 INFO mapred.JobClient:  map 99% reduce 3%
15/02/09 09:49:38 INFO mapred.JobClient:  map 100% reduce 4%
15/02/09 09:49:48 INFO mapred.JobClient:  map 100% reduce 5%
15/02/09 09:50:02 INFO mapred.JobClient:  map 100% reduce 6%
15/02/09 09:50:05 INFO mapred.JobClient:  map 100% reduce 7%
15/02/09 09:50:12 INFO mapred.JobClient:  map 100% reduce 8%
15/02/09 09:50:22 INFO mapred.JobClient:  map 100% reduce 9%
15/02/09 09:50:27 INFO mapred.JobClient:  map 100% reduce 10%
15/02/09 09:50:36 INFO mapred.JobClient:  map 100% reduce 11%
15/02/09 09:50:42 INFO mapred.JobClient:  map 100% reduce 12%
15/02/09 09:50:45 INFO mapred.JobClient:  map 100% reduce 13%
15/02/09 09:50:56 INFO mapred.JobClient:  map 100% reduce 14%
15/02/09 09:51:02 INFO mapred.JobClient:  map 100% reduce 15%
15/02/09 09:51:05 INFO mapred.JobClient:  map 100% reduce 16%
15/02/09 09:51:11 INFO mapred.JobClient:  map 100% reduce 17%
15/02/09 09:51:17 INFO mapred.JobClient:  map 100% reduce 18%
15/02/09 09:51:30 INFO mapred.JobClient:  map 100% reduce 19%
15/02/09 09:51:39 INFO mapred.JobClient:  map 100% reduce 20%
15/02/09 09:51:45 INFO mapred.JobClient:  map 100% reduce 21%
15/02/09 09:51:48 INFO mapred.JobClient:  map 100% reduce 22%
15/02/09 09:51:54 INFO mapred.JobClient:  map 100% reduce 23%
15/02/09 09:52:00 INFO mapred.JobClient:  map 100% reduce 24%
15/02/09 09:52:03 INFO mapred.JobClient:  map 100% reduce 25%
15/02/09 09:52:07 INFO mapred.JobClient:  map 100% reduce 26%
15/02/09 09:52:19 INFO mapred.JobClient:  map 100% reduce 27%
15/02/09 09:52:22 INFO mapred.JobClient:  map 100% reduce 28%
15/02/09 09:52:28 INFO mapred.JobClient:  map 100% reduce 29%
15/02/09 09:52:34 INFO mapred.JobClient:  map 100% reduce 30%
15/02/09 09:52:37 INFO mapred.JobClient:  map 100% reduce 31%
15/02/09 09:52:46 INFO mapred.JobClient:  map 100% reduce 32%
15/02/09 09:52:49 INFO mapred.JobClient:  map 100% reduce 33%
15/02/09 09:53:31 INFO mapred.JobClient:  map 100% reduce 66%
15/02/09 09:53:34 INFO mapred.JobClient:  map 100% reduce 69%
15/02/09 09:53:37 INFO mapred.JobClient:  map 100% reduce 70%
15/02/09 09:53:40 INFO mapred.JobClient:  map 100% reduce 72%
15/02/09 09:53:43 INFO mapred.JobClient:  map 100% reduce 73%
15/02/09 09:53:46 INFO mapred.JobClient:  map 100% reduce 74%
15/02/09 09:53:49 INFO mapred.JobClient:  map 100% reduce 76%
15/02/09 09:53:52 INFO mapred.JobClient:  map 100% reduce 77%
15/02/09 09:53:55 INFO mapred.JobClient:  map 100% reduce 78%
15/02/09 09:53:58 INFO mapred.JobClient:  map 100% reduce 80%
15/02/09 09:54:01 INFO mapred.JobClient:  map 100% reduce 81%
15/02/09 09:54:04 INFO mapred.JobClient:  map 100% reduce 82%
15/02/09 09:54:07 INFO mapred.JobClient:  map 100% reduce 84%
15/02/09 09:54:10 INFO mapred.JobClient:  map 100% reduce 85%
15/02/09 09:54:13 INFO mapred.JobClient:  map 100% reduce 86%
15/02/09 09:54:16 INFO mapred.JobClient:  map 100% reduce 88%
15/02/09 09:54:19 INFO mapred.JobClient:  map 100% reduce 89%
15/02/09 09:54:22 INFO mapred.JobClient:  map 100% reduce 90%
15/02/09 09:54:25 INFO mapred.JobClient:  map 100% reduce 92%
15/02/09 09:54:28 INFO mapred.JobClient:  map 100% reduce 93%
15/02/09 09:54:31 INFO mapred.JobClient:  map 100% reduce 94%
15/02/09 09:54:35 INFO mapred.JobClient:  map 100% reduce 96%
15/02/09 09:54:38 INFO mapred.JobClient:  map 100% reduce 97%
15/02/09 09:54:41 INFO mapred.JobClient:  map 100% reduce 98%
15/02/09 09:54:44 INFO mapred.JobClient:  map 100% reduce 100%
Improve this question

1 Answer 1

Reset to default
1

This is by design, and is required because of the semantic guarantees required of the reduce() algorithm before it can begin (aka its preconditions). This is one of the core aspects of understanding how MapReduce works. It would be wise to learn the theory before trying to actually use MapReduce to avoid such confusions in the future.

Here is a source that directly states that the Reduce algorithm cannot begin until Map is complete.

Keep in mind, although it is theoretically possible for you to write a certain implementation of "MapReduce" (or the algorithms/functors that it leaves for the developer to write) in such a way that the reduce could begin before the map is complete, doing this would effectively break the "contract" that standard MapReduce is designed for. So you wouldn't really be using proper MapReduce at that point. And you'd have to be very, very careful to make sure that violating that contract does not cause some race conditions or locking issues.

The thing to keep in mind is that, the design contract of the MapReduce framework is there for a specific reason; it is to maximize data safety, fault tolerance, and performance all at once. Breaking the contract means that you are then responsible, from that point forwards, to do your own analysis to convince yourself that you are retaining those same guarantees that the official MapReduce promises (or to convince yourself that you don't care if those guarantees are not met). In this case, once you were done with modifying the source code of (for example) Hadoop to meet your needs, the resulting product would not really be MapReduce, since the contract of MapReduce will have been broken.

Improve this answer
5
  • Thanks for the information. I am new to mapreduce algorithms and noticed that, on smaller data sets (20 GB or so), I seem to get around 20% Reducing done by the time that the Mappers get 100% finished at which point the Reducers finish quite quickly. According to what you are saying, this should actually be avoided - is that true? If so, how do I go about avoiding this?
    – drjrm3
    Feb 9, 2015 at 16:33
  • The performance of your use of MapReduce is entirely dependent on what your Map and Reduce functions are doing. In one person's implementation, Map may take 0.1 microseconds, but Reduce might take 3 years. In a completely different scenario, the timings may be flipped. In yet another scenario, both operations may take about the same length of time. It is entirely dependent on what those functions are doing.
    – allquixotic
    Feb 9, 2015 at 16:36
  • I guess the Partitioner you are using is sending Partitions to the Reduce function before Map is complete. If that's how it's working, it is possible that the Reducer might start working before the Map phase is completely done. This is an optimization and you should be happy it is there. You aren't going to get the rest of the Reduce to complete in parallel before the Map is done, though. That's bringing the cart before the horse.
    – allquixotic
    Feb 9, 2015 at 16:41
  • Thank you - it is this optimization I am asking about. I am aware that Reduce cannot finish before the Map is done, but I am interested in optimizing how they work together (I'd rather not have 100% map finish before the reducer even starts).
    – drjrm3
    Feb 9, 2015 at 16:52
  • The reducer seems to be starting speculatively while the map is still ongoing, but when it encounters a new data point matching a key that's already been reduced, it has to throw out those results and start over. That's why your reduce % seems to be going down. It makes sense.
    – allquixotic
    Feb 9, 2015 at 18:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .