Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 0 down vote favorite
2

I need to run ~1500 jobs on my lab's server. The program I am using (blastpgp if anyone cares) is capable of running in parallel mode, the "-a" option sets how many threads it will launch. The server in question has 40 64bit Opteron CPUs available.

I have the following choices:

  1. Run the jobs one by one, sequentially, allowing blastpgp to occupy all 40 cores by launching 40 threads (-a 40).

  2. Run 40 jobs in parallel, allowing each to launch one thread only.

Which of these two options will be faster? Are they equivalent? I imagine it may depend on the particular hardware/software/job details but is there a general rule when running a natively parallel program on a cluster?

share|improve this question
I would expect threads to be slightly faster, but why dont you try it out? – Karthik T Jan 7 at 13:58
@KarthikT because each job takes a while to run and results may vary depending on the server's current load, and I really don't have the time to do an exhaustive test. I was hoping someone here would know. Also, this is a shared server so I am trying to limit my footprint so as not to bother other users. – terdon Jan 7 at 14:12
If your results are dependent on the servers load, then it may not matter! – Dave Rook Jan 7 at 14:34
@DaveRook. results will always depend on processor load. If a higher priority process is running, my job will not be able to use 100% of a CPU. In any case, I am searching for a general rule. – terdon Jan 7 at 15:59
@terdon - sorry. In that case, and I'm only talking about my experience with .NET where the parallel class is very powerful and more powerful than something I could write (as it's optimized for efficiency), therefore my answer is use the -a and make it all parallel (although technically it's still serial, as each process has nothing to do with each other). – Dave Rook Jan 7 at 16:09

1 Answer

up vote 0 down vote accepted

I asked a similar (not identical) question on stackoverflow and the general consensus seems to be that multi-threading will be faster because:

  1. Threads are lighter than processes, place less of a strain on the system
  2. Threading means launching the program only once, so any initialization tasks need only be performed once.
  3. The OS is usually better than userland in managing memory and scheduling so it is better to launch multiple threads and let the OS sort them out.

I am marking this as community wiki because it is just a summary of the answers I received at stackoverflow.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.