3

Please do not ask me why (they made me) but I have to copy 500GB data to the local drive every 200 node/instances that I am launching in EC2. For reasons beyond this post, this data must by on the local drive and not EBS drive so I can not benefit from snapshots.

What is the fastest way that I can manage to this? Copying from S3 to each node takes a long time. I trying to attached an EBS volume to every node with the data and then copy the data from EBS to the local drive but that also take a long time (several hours_)

Now, I am also thinking to use bit torrent but not sure how well it is going to be. What is the best way to copy 500GB of static data to each local drive of 200 ec2 instances?

The 500Gb of data is composed several hundred of file with varying size but the biggest file is 20GB.

Improve this question
7
  • i'd be interested to know this
    – popnoodles
    Dec 11, 2012 at 23:28
  • What form does the data take? For instance, one large file, several tiny files, or data from a database?
    – user1158559
    Dec 11, 2012 at 23:48
  • 1
    Despite the fact that you don't want us to ask 'why', and why can't you just mount the 500Gb on an EBS drive or a few for redundancy, I would seriously question your boss as to whether this is a good idea. P.S. I very much like your idea of using Torrents. They are secure, robust, highly configurable, and I think that would mimimize the total cost of that kind of transfer.
    – user1158559
    Dec 11, 2012 at 23:56
  • What sort of data is it? I'm thinking in terms of compression.
    – user1158559
    Dec 11, 2012 at 23:57
  • The reason that EBS is not an option is because the files are used by legacy code that use memory mapping and that is very slow on EBS
    – iCode
    Dec 12, 2012 at 0:05

3 Answers 3

Reset to default
4

Your reason for not wanting to use EBS is that it is slow. You may want to test EBS optimized instances plus Provisioned IOPS EBS volumes (which can even be RAIDed for higher IOPS). This will simplify making data available to new instances.

Note that an EBS volume takes a while to make all the data available at highest performance. I.e., the performance you get on a fresh EBS volume is slower than the performance after the volume blocks have been populated.

Here's an article I wrote which talks about this process, including one way to identify when an EBS volume has completed initialization from a snapshot (though it basically includes transferring the entire volume through the network anyways):

http://alestic.com/2010/03/ebs-volume-initialization-from-snapshot

If your application wants to start right away at the expense of being a bit slow when it needs to access previously unread data, then I would recommend EBS optimized instances plus Provisioned IOPS EBS volumes, possibly in RAID-0. Once the volume(s) populate, the application speed should ramp up considerably.

Otherwise, the name of the game when loading data from S3 is parallelization. You can have a hundred concurrent connections downloading pieces of the data from the super-scalable S3 service, as long as you are using an instance type with sufficiently high IO.

Even on a 1Gbps interface, however, it will still take over an hour to download 500GB.

Here's a possible trick to consider: With an EBS optimized instance you get a dedicated network interface for the EBS subsystem, separate from the standard network interface. You might be able to cut your data load time in half if you download half the data from S3 and the other half from an EBS volume.

Just in case you hadn't considered it: Make sure your data is compressed in storage to reduce transmission time.

Improve this answer
0

Disclosure: I'm with Zadara Storage

I suggest you take a look at Zadara Storage. With Zadara Storage you can have the central repository in a NFS mount that will be accessible from all EC2 machines. Zadara has very high bandwidth and low latency compared to S3, and you can copy to the local drives each time. (or even use directly from Zadara Storage) You can mount Zadara Storage from EC2 via simple NFS or iSCSI if you need a block device.

You can have a free trial at http://www.zadarastorage.com

Improve this answer
0

This is a very old question, but for those with a similar issue, the fastest way to get this done would be to copy it to one EBS volume, snapshot that volume, then create volumes as needed from that one snapshot and attach them to your instances. This is probably a good use case for something almost no one seems to use - placement groups. Placement groups are limited to a single AZ but put you on the 10G network, meaning your 500GB file copy would boosted significantly.

Or, you could dump it in S3 and copy it from there.

Improve this answer
1
  • EBS volumes created from snapshots are actually incredibly slow, I get about 3.5MB/s read rate from them using dd, until all of the blocks of the volume have been populated from S3 (where Amazon stores EBS snapshots). After an initial read of every block, the IO speed returns to normal.
    – thenickdude
    May 23, 2017 at 4:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .