Torque Job Submission
R2D2, the star wars cluster, consists of 30 dual processor, 64 bit, Opteron, computational nodes, with routing to 3 of the jedi computers as well. These jedis, 3, 4, and 5 are similar to the other computational nodes, except that they have 4 processors instead of 2, so that they are exceptional for cases where the software is multithreaded.
Interactive use of the star wars cluster is prohibited, with a few exceptions, such as cluster monitoring software and CCP4i_6.0.1, which has the queue submission commands built in. All other jobs should be submitted with the qsub command. The qsub command has a different implementation on r2d2 than on queen, which necessitates these instructions. It is advisable to review the instructions for the hive, and use it when the star wars cluster is busy.
Using the Torque implementation of qsub, which we have on r2d2, jobs can now be routed to the machines which are best suited in terms of memory requirements and multi-threading capabilities, as in the following examples.
qsub -q lowmem qsub -q medmem qsub -q himem qsub -q @jedi_master qsub -q @jedi_master3 qsub -q @jedi_master4 qsub -q @jedi_master5
Add the name of your command script to one of those commands in order to submit it to the appropriate queue, and feel free to let me know if you hit any snags. We are ready to help you get underway with the cluster.
The high priority default queue is lowmem, so if you have special requirements, other queues should be selected. Here are examples of some commonly used queue queries which are adapted to the new situation.
qstat -q qstat -q midmem qstat -q @midmem qstat -q @jedi_master5
Note the server line in the output. The -q option works with qsub too, but I think qdel must be run on the destination server. Entries with an @ in front are destination queues. Master is now a dedicated routing queue server, so jobs disappear from the master queue and are passed along to destination queues.
Here are some job query commands.
qstat -na @lowmem qstat -na @midmem qstat -na @himem qstat -na @jedi_master3 qstat -na @jedi_master4 qstat -na @jedi_master5
Here are some pbsnodes examples.
pbsnodes -a -s midmem pbsnodes -a -s jedi_master4
Together with our recent gigabit and parallel file access upgrades, now beowulf now provides a fast and up to date clustering tool. It was quite some work, but if you have any further questions or suggestions, please feel free to pass them along.
document created by Michael L. Love, and maintained by the Crystallography facility