I work for Research Computing at RIT. When I first came on board I was tasked with extracting data from our monitoring solution, Zabbix, and pushing into graphite. So I wrote Clustertop. Clustertop works by periodically polling zabbix for data and then pushing that data somewhere else using plugins. Its completely configured using a simple config file which looks something like this.
slurm.jobs.running=system.run[squeue -t R --noheader | wc -l]
slurm.jobs.pending=system.run[squeue -t PD --noheader | wc -l]
slurm.jobs.suspended=system.run[squeue -t S --noheader | wc -l]
host = localhost
port = 2004
It uses a simple ini file. You can configure how to talk to zabbix, as well as
what keys/hosts to grab data for. The poller option allows for the usage of
customer pollers. In the above example the GraphitePoller is used. Below that
there is another example of how to specify what data to grab from zabbix.
[special:<hostname>] blocks allow you to do one off checks for a specific host.
In the above case it is used to get special data off of a cluster head node.
The rest of the config can be used to configure the custom pollers. Here
the graphite poller is told how to contact the graphite server.
Clustertop is built on python 2.7 and its dependencies are listed below.
By default clustertop only comes with 2 different pollers. The first is the
clustertop.poller:Poller. This poller prints out the data
it grabbed from zabbix. It is useful for debugging, and is also the base on
which all custom pollers are built. The other builtin poller is
clustertop.poller:GraphitePoller. This is also a pretty simple poller, it just pushes data into graphite.
Clustertop has two commands, check and run. check does the same thing as run, except instead of executing in a loop, it does it once and exists. Useful for debugging and verifying the configuration.
$> clustertop check --config /etc/clustertop
$> clustertop run --config /etc/clustertop