Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've seen a couple of questions on here that are similar to what I'm thinking, but not quite. So, I figured I'd ask and see what falls out.

Basically, I have 2 or 3 old, slower-ish laptops laying around. What I would like to do is cluster these computers in order to perform some "embarrassingly parallel" data analysis procedures on them (for example, training a machine learning algorithm on many different cores). To be more specific, I'm wanting to run Python code (scikit-learn, which has the ability to run on many different cores) on this cluster.

Ideally what I would like to do is have a cluster with the 3 laptops tied together, and a storage device (1-2 TB) tied into the network. I would then like to write the scripts to do my data analysis on my current laptop, access the cluster somehow (ssh?), and run the scripts on the data stored on the connected device while splitting the analysis amongst the different laptops. Thoughts?

I've read some about Beowulf clusters, clustering with Ubuntu, and other things. I have 0 experience with networking, clustering, Unix commands, etc, however, and don't really understand any of it. I'm willing to learn but really have no idea where to start, so tips, resources, suggestions, would be greatly appreciated.

share|improve this question
up vote 2 down vote accepted

Maybe you can find Beowulf clustering an interesting solution for your case.

share|improve this answer

If you don't want to get into the low-level details of clustering I would suggest you to take a look at BOINC . (Relatively) easy to set up, lot of fun :)

share|improve this answer

If the laptops are extremely old, it might just be faster to buy a newish single machine which would get the job done a lot faster. Or buy a whole bunch of the new tiny ARM powered Linux machines (RaspberryPi et. al.) which are fairly cheap, and wire them all up for a lot of CPU's in a pretty compact size.

Or, if you just want to play around and learn, see the other answers on clustering.

share|improve this answer

In your situation I would learn about sharing directories with NFS so each machine has access to the same data. For this example we will call the directory /data. Then you would want to setup ssh to work without requiring a password using ssh keys(ssh-keygen). From there you would want to do some scripting, probably in Python since you will be using it anyway, to automate the running of your jobs. Your script will end up running something along the lines of "ssh otherlaptop doexecutable /data/datafile". You could also use something like the Torque resource manager, or Slurm(Simple Linux Utility for Resource Management) to do the execution on the laptops but you would still need a script to create the jobs that you would hand off to torque/slurm.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .