How does FaceBook build servers with 12+ hard drives

Question

From The FB Blog:

... is deployed on top of commodity storage blades. The typical hardware configuration of a 2U storage blade is –

• 2 x quad-core CPUs
• 16GB – 32GB memory
• hardware raid controller with 256MB – 512MB of NVRAM cache
• 12+ 1TB SATA drives

Can someone please tell me exactly what parts they use.

I know that you can get motherboards that can take 2 cpus, I can get motherboards that can hold 16 GB of ram, but how do I connect 12 drives?

Answer 1

twelve 1TB drives? peanuts, think BIG! :)

Texas Memory Systems launches the RamSan-6200 SSD System that offers up to 100 terabytes of Flash-based storage in a 40U rack configuration that can sustain a record 5 million input/outputs per second (IOPS) with 60 gigabytes per second throughput while using a little over 6 kilowatts of power.

all this for a whopping price tag of $4.4m, you hear me, Santa? :)

(maybe RAMinSANe would be a better name)

but if you're OK with only 12 drives, you'll get a nice SATA 12 Drive Enclosure with Controller (speed > 500MB/s) for around $1.000

Answer 2

I hate to dig up such an old question, but these answers need help. There are a number of different 2U cases that feature 12x3.5" hot swap bays (3 vertically x 4 horizontally -- See Supermicro's SC827), , and even some that also have an internal bay or two (http://www.servaris.com/servers_m2100.php). It's dense, but not impossible. If you want to get a little crazy, Supermicro's SC847 supports 45 hot swap drive bays in 4U (no motherboard) or 36 with E-ATX motherboard, and someone makes a case (the RMC5D2) with 48 vertical hot swap bays with room for E-ATX board and I think 7 full-sized cards in 5U.

I believe this is for Facebook's Haystack system for photos, where they use a RAID 6 controller to connect the drives. You can buy a low profile one that still supports 12 drives or use a riser card.

Facebook uses a Hadoop system to analyze everything that happens on the site (multiple TB/day). It skips RAID disk arrays and makes the entire machine redundant by splitting large files into 64MB chunks and writing each chunk to multiple machines. For this you can get a motherboard with 8 ports and cheaply add 4 non-RAID ports. I'm pretty sure these machines don't have so many drives, but the technology is based on stuff that Google is doing. Google does have racks of 12 drive RAID free machines, and a lot of interesting computer designs. They're almost big enough for custom parts to be a volume order of commodity hardware.

Answer 3

I think that the original FB post is a bit misleading. If they're using storage blades, those are designed to connect to external storage arrays. The blade itself would run Windows Storage Server for example, but you wouldn't store your data on that blade. You would hook a 2U SAN array to that blade and that array could hold 12 hotswap bays at 3.5" or 24 bays at 2.5".

Answer 4

I work with video and since there wasn't drives of 1TB and above 3 years ago, I searched for a solution similar to yours. You have to have internal or external RAID controller for what you want. I found one company that deals in such stuff, and it is (for me at least) some kind of a paradise for creating HUGE storage systems for acceptable price.

http://www.addonics.com/

For example:

http://www.addonics.com/products/raid%5Fsystem/rack%5Foverview.asp

Answer 5

Look at servers 1U, 2U 3U 4U, those fits in rack in server room.

like those http://www.supermicro.com/products/system/2U/

You can look at Dell, HP, IBM servers and many more.

They can suport around 8 hard drive, some more.

Answer 6

Connecting 12+ disks to a machine isn't difficult -- you get a raid controller with 12+ SATA ports on it, or you use SATA expanders, or a SAS raid controller.

The difficulty is physically fitting 12 3.5 inch drives into 2U; I'm not sure you can turn a 3.5 inch drive vertical with only 2U, and you aren't going to fit 12 drives in 2U without either doing that or buring some disks in the box (making them not hot-removable).

Answer 7

As others have mentioned there are many different options to fit 12 1TB drives into 2U. While 3.5 inch drives have to be horizontal to fit in that space, 2.5 inch drives like the Western Digital Scorpio Blue WD10TPVT will fit in 2U vertically. You can get a Supermicro S216 based chassis to do up to 24 2.5 inch drives in 2U, along with two 12 port sata raid controllers (like this 3ware 9550SXU-12) to connect them all.