torque in Cluster

Question

I am learning and have some questions about using PBS to submit jobs in a cluster.

(1) If a job is parallelized by MPI and run by mpirun without using qsub, it cab be running on different nodes. So what is the point of using qsub?

Is the job not submitted by qsub queued or not?

(2) In a pbs script that is submitted by qsub, are all the commands executed one after the other?

If I want to run several executables at the same time, is it to make these calls background by adding "&" at the end?

(3) In the pbs script, if the several calls to run the executables are running the same time by running background, are the number of nodes and processors per nodes specified to be the total needed by all those calls?

If yes, and the number of nodes and processors per nodes specified are not completely available but enough to run some of the calls, will some of these calls be run first or delayed until the requested total resources are completely available?

Thanks!

---

UPDATE:

For you better understanding my questions above, Our cluster is a linux cluster with Torque and Maui. I am reading tutorials online and find that the compilation is only on the head node, and after submitting the job by qsub, the other nodes are running the executable compiled on the head node. So that is why my questions arise.

Answer 1

I haven't used PBS for a while, only SGE so these answers could be somewhat SGE-centric.

The reason for using a queue-system is to utilize your resources better. If you have a cluster of 1000 nodes and 100 users you don't want all of them logging in and try to run mpirun to use all of the 1000 nodes. Instead you let them submit jobs asking for resources (for example 10-20 cpus, 4 GB memory and so on) and the queue system takes care of dispatching jobs on available machines. Also, if you don't have that much users and that big cluster you could perhaps have 100 job that you need to run. Just submit al of them to the queue system, let it work in the background, over the night and over the weekend and all your resources would be used in a efficient way. One cool thing you can do is to let the queue system start and stop your machines depending on demand so that if you have no jobs it shuts down your nodes and when you submit jobs it starts the right amount of machines. Could potentially save a lot of power!

The script is running the same way it would have if you start it by hand, it just has an environment that knows which nodes to use, what directory to run in and such. If you really want jobs running in background you can start them with an & at the end, but remember to wait for running jobs at the end, otherwise your could get a miss-match between resources that your queue system knows of and the real resources available.

Answer 2

To answer your first question. MPI should be configured with the nodes available and number of processor cores available and MPI should handle all the distribution if it is coded correctly. At least that was how I remember MPI working when I used it. I used the LAM-MPI distribution though, so maybe some things were different.

As for PBS, I have not used that yet, but I did find this university page explaining some basics of runs and specification:

http://euler.phys.cmu.edu/cluster/pbs.html