Status Monitoring

To get an idea of queue usage, you can try the "qstat", "qhost", "ql", and "qb" commands. Each displays the same data, derived from SGE's monitoring system, but presents it in different views. The "qstat" and "qhost" programs have numerous options but by default tend to present too much or too little information. This same information is processed by "qb" and "ql" to show a more user-friendly view of the data. "qb" shows the 'block' view of the current cluster usage, showing each machine with letters indicating what jobs are running on it. "ql" shows a very summarized view, giving the number of machines that are loaded or unloaded. See the page on SGE Job Monitoring for more info.

We have installed the Ganglia monitoring system on all machines in the cluster. This tool produces a web page HERE that lists the overall cluster usage, as well as per-sub-cluster, and per-machine statistics. If you are finding that your jobs are taking a long time to run, or that they are sitting in the queues for a while, you may want to check this web page to see if the cluster is experiencing unusually high demand.

We also have MRTG running HERE which shows router/switch traffic.