Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Please do not ask me why (they made me) but I have to copy 500GB data to the local drive every 200 node/instances that I am launching in EC2. For reasons beyond this post, this data must by on the local drive and not EBS drive so I can not benefit from snapshots.

What is the fastest way that I can manage to this? Copying from S3 to each node takes a long time. I trying to attached an EBS volume to every node with the data and then copy the data from EBS to the local drive but that also take a long time (several hours_)

Now, I am also thinking to use bit torrent but not sure how well it is going to be. What is the best way to copy 500GB of static data to each local drive of 200 ec2 instances?

The 500Gb of data is composed several hundred of file with varying size but the biggest file is 20GB.

share|improve this question

migrated from stackoverflow.com Dec 12 '12 at 2:43

This question came from our site for professional and enthusiast programmers.

    
i'd be interested to know this –  popnoodles Dec 11 '12 at 23:28
    
What form does the data take? For instance, one large file, several tiny files, or data from a database? –  user1158559 Dec 11 '12 at 23:48
    
Despite the fact that you don't want us to ask 'why', and why can't you just mount the 500Gb on an EBS drive or a few for redundancy, I would seriously question your boss as to whether this is a good idea. P.S. I very much like your idea of using Torrents. They are secure, robust, highly configurable, and I think that would mimimize the total cost of that kind of transfer. –  user1158559 Dec 11 '12 at 23:56
    
What sort of data is it? I'm thinking in terms of compression. –  user1158559 Dec 11 '12 at 23:57
    
The reason that EBS is not an option is because the files are used by legacy code that use memory mapping and that is very slow on EBS –  iCode Dec 12 '12 at 0:05

2 Answers 2

Your reason for not wanting to use EBS is that it is slow. You may want to test EBS optimized instances plus Provisioned IOPS EBS volumes (which can even be RAIDed for higher IOPS). This will simplify making data available to new instances.

Note that an EBS volume takes a while to make all the data available at highest performance. I.e., the performance you get on a fresh EBS volume is slower than the performance after the volume blocks have been populated.

Here's an article I wrote which talks about this process, including one way to identify when an EBS volume has completed initialization from a snapshot (though it basically includes transferring the entire volume through the network anyways):

http://alestic.com/2010/03/ebs-volume-initialization-from-snapshot

If your application wants to start right away at the expense of being a bit slow when it needs to access previously unread data, then I would recommend EBS optimized instances plus Provisioned IOPS EBS volumes, possibly in RAID-0. Once the volume(s) populate, the application speed should ramp up considerably.

Otherwise, the name of the game when loading data from S3 is parallelization. You can have a hundred concurrent connections downloading pieces of the data from the super-scalable S3 service, as long as you are using an instance type with sufficiently high IO.

Even on a 1Gbps interface, however, it will still take over an hour to download 500GB.

Here's a possible trick to consider: With an EBS optimized instance you get a dedicated network interface for the EBS subsystem, separate from the standard network interface. You might be able to cut your data load time in half if you download half the data from S3 and the other half from an EBS volume.

Just in case you hadn't considered it: Make sure your data is compressed in storage to reduce transmission time.

share|improve this answer

Disclosure: I'm with Zadara Storage

I suggest you take a look at Zadara Storage. With Zadara Storage you can have the central repository in a NFS mount that will be accessible from all EC2 machines. Zadara has very high bandwidth and low latency compared to S3, and you can copy to the local drives each time. (or even use directly from Zadara Storage) You can mount Zadara Storage from EC2 via simple NFS or iSCSI if you need a block device.

You can have a free trial at http://www.zadarastorage.com

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.