Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am currently running a few perl and python scripts on a windows pc and would like to port over to the Amazon EC2 servers running 64-bit LINUX. The scripts are basic web scrapers that go to a variety of websites, get data and then save daily as csv files.

I would like to install these in the cloud and get them running in an automated way so that they will run without my intervention. Also given that I don't want to lose all the data if the instance crashes, I should also upload the csv files to Amazon S3.

Any idea how I can do this? I am not terribly versed in LINUX nor do I know Perl/Python well. What is the best way for me to tackle thi

share|improve this question

migrated from Jan 3 '11 at 6:36

This question came from our site for professional and enthusiast programmers.

If all you are doing is scraping websites, I would suggest an alternate solution:

Do not use Amazon EC2. EC2 requires a skill level beyond simply being able to administer a unix-style operating system. EC2 also assumes that you can handle situations by yourself, there is no one to bail you out. I understand that currently EC2 is free to new signups for individuals, but you are hitting a finishing nail with a pneumatic jackhammer.

Instead, use a shared web hosting service that supports perl, those described at the helpful would be good. The shared hosting providers listed there are all relatively cheap (< $10/month) and give you the opportunity to use CPAN. This way you have somebody else worried about the server operating system, and you only worry about your scripts. Then learn about using unix crontab to run your script on a schedule, or learn something like POE or AnyEvent to make your scraper a service (search CPAN for the last two).

As for storage, you can buy a backup package for your shared hosting and you should have plenty of free storage with your package (usually at least 10GB). If you need to backup to S3 for some specific reason you can do that as well, there are even perl modules on CPAN to help with this. For bonus points I suggest you explore Dancer or Catalyst to make your scarpings available from a web browser and learn perl webdev in the process. Keep in mind that these shared hosting packages often include unlimited transfer and almost unlimited storage, EC2 does not include these things.

If shared hosting won't work and you want to spend more money I suggest as an intermediate step. You are responsible for your install of linux, but you have more support backup than you would with EC2.

The bottom line is that EC2/S3 are VERY useful tools, but they're really only for production deployments or very serious individuals who have had experience with production deployments. Because it sounds like you don't have this experience I would highly recommend you go another route.

Good Luck.

share|improve this answer
I second this, with the caveat that you verify your shared hosting allows you to run scheduled scripts (some do not). Amazon provides new customers a free year for most EC2-related things, with overage charges for some usage categories. Even aside from that, it's worth noting that a continuously running EC2 micro instance, reserved for 3 years, can work out to under $8/month. – ysth Jan 2 '11 at 22:46
Thanks for the Perl Shared Hosting link! – Hugmeir Jan 2 '11 at 22:50
I understand the thought, but I figured I would try to learn the EC2 platform by doing various simple things. I know basic Unix command line but don't know how to manipulate instances, storage, etc... Is there a way I can start through some baby steps? Like how do I save a file to EBS? – racket99 Jan 2 '11 at 22:56
My answer is firmly rooted in the idea of baby steps. In terms of learning curve required for hosted platforms I would rank this way: shared hosting, managed VPS (mediatemple), VPS (linode), rackspace cloudservers, Amazon EC2. EC2 requires a fairly advanced understanding of unix operating system management. I understand that it is a hot topic right now, and that Amazon is offering it free for a year, but it is definately jumping into the deep end. If you cannot install a copy of Debian on an old PC, install CPAN and setup a crontab to run the scripts you have written, then EC2 is too much. – huntar Jan 2 '11 at 23:28
Fair enough. But can I run python scripts on this perl shared hosting? – racket99 Jan 3 '11 at 0:05
  1. Find someone who does know Linux, Perl, and Python.
  2. Give them money.
share|improve this answer
That's probably the right approach but thought I'd see if I could learn by trying this task. – racket99 Jan 2 '11 at 21:58

I don't know if EC2 is the right tool for the job. Maybe have a look at it basically allows you to just run single methods in the Cloud.

share|improve this answer

If you actually want the data in S3, uploading to S3 makes sense. If your goal is just to keep the existing data, an alternative is using Amazon's Elastic Block Storage to have a persistent mounted drive available.

share|improve this answer

You could use one of the Cloud Management Apps to either do this for you through the UI or submit a support request to get a sysadmin to do it for you - if you want to use AWS that is. Check out and (disclaimer: I work in Digital Mines).

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .