Does anyone know how sites hold massive numbers of pictures in general? I tried researching this, but it seems like they would need massive storage spaces to hold all these, unless there was a trick to it? I'm sure they compress them, but thats still a huge amount of data to hold for one site.
When it comes to storing large amounts of data, content providers use Storage Area Networks, also known as SANs and SAN storage hardware.
A storage area network (SAN) is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices to the operating system. A SAN typically has its own network of storage devices that are generally not accessible through the local area network by other devices. The cost and complexity of SANs dropped in the early 2000s to levels allowing wider adoption across both enterprise and small to medium sized business environments.
So what does a SAN storage device look like? Some look like the following:
Every one of those slots (the red square is the drive eject button) is a hard disk drive. The one in the picture is a relatively small SAN storage device, others can look much different and can be much larger in size.
Where I used to work, we had SAN storage which were boxes, basically the size of 3 refrigerators side-by-side filled with hard drives. We then took those drives as needed and created RAID arrays for redundancy. As we needed more space, we could order more SAN storage devices and attach them to our storage area network. This allowed us to have petabytes of redundant storage.
Sites like Flickr, Picassa, Facebook, etc, have very large SANS filling massive datacenters.
From your question I gather that you do not have a Computer Science background, so I'd avoid throwing geeky sounding terms.
Popular websites, handling extremely large amounts of data (or traffic), is nothing new or very unique. Usually there is no trickery in terms of massive compression (since most pics uploaded as JPEG are highly compressed already, and further compression can often result in loss of details). What does go in is some clever architecture, lots and lots of computers, fast & reliable network, and of course, several terabytes (or even petabytes) of storage. Actually, storage is often the least of the issues. Storage and compute power is pretty inexpensive these days.
What happens is data is often distributed (several copies) across multiple computers, for redundancy and faster-retrieval, and seek/search of data happens in parallel. Keeping frequently used data closer to the edge of the network or users and keeping such data updated based on usage, are some of the techniques.
Some geeky keywords that are often used, and might be seen as wizardry are:
- Multi-level caching
- Distributed storage
- Data Warehousing
- Data sharding (mostly in the SQL world)
- Parallel processing
- CDN (Content Delivery Networks)
How to do this well, more efficiently, effectively is a field of study and also research, in the area of computer science and computer architecture. Different technique are used based on nature of data, nature / frequency of access (more writes, versus more reads), kind of reliability required etc.
Edited: This picture of google's server rack (from 1999) is epic:! Note the exposed harddrives (3-4 of them) in the middle of each server "tray" (especially the one labelled "g61").
and the full journey is captured in this post here:
They can't compress the photos, because photos are almost certainly already compressed, either with JPEG or PNG compression, and it's not possible to compress already-compressed data. (That's oversimplifying it a bit, but unless you want to get deep into information theory, just accept that as a given.)
There's really no shortcut. A site that holds massive amounts of data has massive amounts of computers to hold it on.
Let's say an image weighs in at 1 MB. There are plenty that are bigger, and plenty that are smaller, but just for simplicity's sake let's say the average image is 1 MB. It's not hard to find affordable 2 TB drives these days, which means that each drive could theoretically hold around 2 million images. (Obviously there will be some space lost to overhead, but you get the idea.)
A server can have a RAID configuration set up with multiple hard drives. Some of the data is lost to redundancy, but even so you can have several TB worth of drives per computer. And a server farm can hold dozens, hundreds, or even thousands of servers. That's how sites like Pinterest and Facebook manage so much content.
They tend to have massive server farms, with computers in front of them that route requests from Web browsers, looking up the content in the appropriate place in the server farm and serving it back to the user. It's a really big topic to try to cover here, but that's the basic idea.