MapReduce and Hadoop: what kinds of problems are they good at solving? [closed]

Question

As I understand it already, it seems that both have similar functions (except that MapReduce is proprietary to Google, while Hadoop is open-source).

I was wondering, not "how they work", but rather some good examples on the usual kinds of problems they are deployed to solve. I know that they take parallel inputs (or makes them parallel). Are MapReduce and Hadoop used for generic parallel computations, or does a problem need to be more specific to be a better fit for the two models?

Answer 1

Don't forget Hadoop MapReduce. :)

"Uses" for MapReduce (according to Wikipedia):

MapReduce is useful in a wide range of applications including: distributed grep, distributed sort, web link-graph reversal, term-vector per host, web access log stats, inverted index construction, document clustering, machine learning, and statistical machine translation. Moreover, the MapReduce model has been adapted to several computing environments like multi-core and many-core systems, desktop grids, volunteer computing environments, dynamic cloud environments, and mobile environments.

At Google, MapReduce was used to completely regenerate Google's index of the World Wide Web.

It replaced the old ad hoc programs that updated the index and ran the various analyses.

Check out this page for a huge list of organizations using Hadoop, and what they're using it for.

A few of the "B"s, for example:

• BabaCar ◦ 4 nodes cluster (32 cores, 1TB).

◦ We use Hadoop for searching and analysis of millions of rental bookings.

• Baidu - the leading Chinese language search engine ◦ Hadoop used to analyze the log of search and do some mining work on web page database

◦ We handle about 3000TB per week

◦ Our clusters vary from 10 to 500 nodes

◦ Hypertable is also supported by Baidu

• Beebler ◦ 14 node cluster (each node has: 2 dual core CPUs, 2TB storage, 8GB RAM)

◦ We use hadoop for matching dating profiles

• Benipal Technologies - Outsourcing, Consulting, Innovation ◦ 35 Node Cluster (Core2Quad Q9400 Processor, 4-8 GB RAM, 500 GB HDD)

◦ Largest Data Node with Xeon E5420*2 Processors, 64GB RAM, 3.5 TB HDD

◦ Total Cluster capacity of around 20 TB on a gigabit network with failover and redundancy

◦ Hadoop is used for internal data crunching, application development, testing and getting around I/O limitations