Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am in ongoing development of a simple but processor intensive computer program that I use for scientific research. It would be really helpful to have more processing speed. Right now I'm running Mac OS X 10.5.7 on a 2 GHz Intel Core Duo and my program runs pretty slow. For a while I was running it on a considerably older laptop with Ubuntu and it ran ~20 times faster.

What I'm thinking is that I could hook up over LAN a bunch of cheap second-hand PCs running linux, and use threaded programing to distribute the processing load across the computers. My program is embarrassingly parallel, ie linear increases in the number of cores should lead to near linear increases in computing speed. Also, program size, memory requirements, and HD requirements are all practically nil, and the amount of information to be passed to and from the main routine (between computers) is basically zero.

So what I'm wondering is, what sort of road blocks are likely to stand in the way of implementing this? Should it be pretty straightforward or no? Will it even work? Also, in purchasing the used computers, what factors should I take into account? For my purposes, is a dual core 2GHz ~= to a single core 4GHz? I figure that as a rule of thumb, I should go with the cheapest multicore-processor machines I can find (best processing to cost ratio).

share|improve this question
Did you consider a Beowulf cluster? – Piskvor Oct 24 '11 at 8:41
Watch out for your fuses! – Jens Erat Oct 24 '11 at 10:33
up vote 3 down vote accepted

Warning: Most knowledge here is theoretical, and needs to be backed up with facts based off specific needs

You probably could/should set up a cluster. If you're going for new systems, the cheapest multicore systems sounds like a good bet (if you go for old ones AVOID PENTIUM IVs, unless you want to heat the room more efficiently than you process. I would consider a dual core 2 ghz system slightly better than a single 4 ghz processor, unless you had a single thread that requires a lot of power (which you don't). I'd note that in general, newer versions of mainstream processors do generally tend to be better than the ones they replace, especially recently since power use has become an issue.If i wanted to get down to the nitty gritty of it, i'd look at the various generations of processors, and benchmarks for them to get a clearer idea of efficiency and power.

You'd probably want to consider booting the worker systems with a minimal os, off standard images via network using PXE to save on storage for each individual system - since your ram requirements are minimal, you could probably have the entire OS in ram, since you're only using it for one thing

You would need to probably have software that is capable of being run on a cluster (IE, take that into account when designing it - maybe with MPI, have the cluster software setup and so on. That being said, i have seen MASSIVE arrays of commodity PCs linked together at a university for a cluster so its plausible

On the other hand, considering massive parallelisation is your main goal, and you're going for massively parallel workloads , you might want to consider alternatives to a x86 cluster - ubuntu has a 48 core cluster that they built out of arm development boards (I'm thinking a cluster of raspberry pi systems would be... interesting, and quite efficient powerwise, if you didn't mind a massive array of odd little inexpensive systems).

share|improve this answer
I love the idea of a rack of a couple hundred Raspberry Pi machines crunching numbers in a cluster (a "bushel" maybe?). – Shinrai Oct 24 '11 at 14:20

Journeyman Geek has very good points about setting up your own cluster but I would also like to add, depending on your workload it may be cheaper to use something like Elastic Map-Reduce. If your workload can be turned in to a Map-Reduce problem you should look in to it. After factoring in the cost of buying the computers, electric, cooling, and maintenance costs it may be cheaper/hr to just run it on amazon ($0.10/hr for the lowest standard tier from Amazon, $0.045/hr if you use the current spot instance price))

If you are willing to do a little more effort you can use your own parallel processing engine and use the micro instance for even cheaper ($0.02/hr for a full instance, $0.007/hr current spot price). Amazon's own Map-Reduce package will not run on it. This can be very useful if your process is cpu bound instead if memory bound as micro instances are faster than the small standard image but have no storage and less ram.

share|improve this answer
So, in general, if I use the the amazon service, do I get a shell account so that I can compile and debug my program on their cloud system, or how exactly does that work? I assume they provide specs for developing software to take advantage of their distributed system? – Matt Munson Oct 24 '11 at 18:49
Depends on if you use a full instance or their managed MapReduce system. If you use their map reduce you get a command line app you run locally to debug If you are going the "Other processing engine" route you get a address and private key you SSH in to and have full control of the box. Like I said, the full control way is cheaper but takes more work to setup. – Scott Chamberlain Oct 24 '11 at 19:42
They have a free tier of their normal EC2 instances to play around with and try it out, if you keep below a threshold of usage. But they do not have a free Map-Reduce tier to try it out. – Scott Chamberlain Oct 24 '11 at 19:45
My program is coded in C++. Does mean that it will be a lot of extra to work to get it working with MapReduce? How much work would I be looking at to setup my own engine? In terms of the minimum amount of work to get a minimally complex C++ program modified for threaded programming running under a cloud computing service, which would be the best way to go? should I consider services other than amazon? I'll try out the free tier and see what I'm up against. – Matt Munson Oct 24 '11 at 20:03
let us continue this discussion in chat – Scott Chamberlain Oct 24 '11 at 22:33

Another option you may want to consider is running your program inside of BOINC. This has the advantage of an existing system that already has the hard part of distributing and collecting "jobs". This also allows you to get more people to run your program. You can get friends or even random people on the internet to run your program if you advertise it enough. This allows you to have a (potentially) massive cluster of computers to run programs on without the startup cost of lots of small computers and ongoing electricity costs.

Note that I've never programmed on BOINC, so difficulty of doing so is something your going to have to find out. For a starting platform though, I would think the time saved not having to reinvent BOINC would balance the complexity of using it

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .