Figures - Statistics Gathering and Reporting

Overview

Figures is a collection of small programs that gathers floating point data from sources and aggregates it, providing views and alerts.

You need some of the programs from conserv to make this work.

Data interchange

Figures data is simply a stream of 4-tuples of type (string, float, float, float). The floats are 32-bit IEEE floats in little endian format (because I started that way and it's just continued). The strings consist of a length byte followed by that many ASCII values encoded as single bytes. Nearly all Figures utilities talk this protocol.

Most of the time the 3 floats are minimum, maximum and average. However, in alarm data they are severity, trigger value and current value.

The string is a key string and has a free form. However, it generally should be of the form hostname.section.subsection.

Utilities

figures_merge

stdin: a dgram socket over which file descriptors are passed. So make it a child of conguardian or mtcpserver. These file descriptors should have figures data comming in over them

stdout: a figures data stream containing all the value from all the inputs

figures_split

stdin: a figures data stream (e.g. from figures_merge)

fd 3: a dgram socket over which file descriptors are passed. You can set the CONGUARDIAN_FD environment variable to 3 to make conguardian set this fd. The input is written to every file descriptor passed.

figures_average.py

arguments: a single number which should be the number of seconds to average over

stdin: a figures data stream

stdout: a figures data stream in which the min, max and average are calculated from the input for each different key string.

figures_show.py

stdin: a figures data stream

stdout: a printed representation of the data.

figures_headings.py

arguments: a directory name. The directory should be a multilog directory where the data is from figures_show with multilog timestamps. Every 30 seconds all the multilog data files are scanned and a list of unique key strings is output in a file called headings in that directory. There is a second, optional, argument that can be used to set the timeout to something other than 30 seconds.

figures_tcptime.py

arguments: a tabbed filename and sleep time. The file should have a number of lines with two, whitespace separated, fields like thus:

www             80
www2            80
mail            25
mail            110

The program will wake up every sleep_time seconds and try to connect to the given ports on the given hosts. It outputs figures data with key strings like mail.ports.110 and the floats being the time taken to connect. It times out the connection after 5 seconds. It sets the time to connect to 10 seconds if the connection fails.

stdin: a figures data stream

figures_mon.py

arguments: a file name. This file should be whitespace and line separated with 4 columns like thus:

# RegExp        >/<     Value           Severity
#------------------------------------------------
.*\.load        >       10              1
.*\.load        >       15              2
.*\.port\..*    >       1               3
.*\.port\..*    >       4               4

The regexp is matched against the key string and if the value is less/greater than the trigger value, an alarm of severity whatever is triggered.

stdin: figures data with (min, max, av)

stdout: figures data with (severity, trigger value, current value)

figures_htmlmon.py

arguments: a file name to write to

stdin: alarm-mode figures data (e.g. from figures_mon.py). The file name given in the command line argument is overridden with a very simple, dull looking, HTML page showing the alarms.

figures_tiniread.py

arguments: a file name with a flie like:

1700080057E0E110        room225.testtemp
1700080057E0E265	machineroom.lhr.t4.top

stdin: line-by-line, whitespace separated, two column, key(string)-value(float as string) data. Generally this is going to be TINI data with the key being the temperature sensor ID and the value being the temperature. The keys are mapped via the file given as a command line argument and figures data produced.

stdout: figures data

sysinfo

stdout: figures data gathered from /proc every 5 seconds

Plumbing it all together

Something like this:

#!/bin/sh

mkdir -p /var/figures
conguardian /var/figures/live_in   figures_merge | CONGUARDIAN_FD=3 conguardian /var/figures/live   figures_split &
conguardian /var/figures/minute_in figures_merge | CONGUARDIAN_FD=3 conguardian /var/figures/minute figures_split  &
conguardian /var/figures/hour_in   figures_merge | CONGUARDIAN_FD=3 conguardian /var/figures/hour   figures_split  &
conguardian /var/figures/day_in    figures_merge | CONGUARDIAN_FD=3 conguardian /var/figures/day    figures_split  &
conguardian /var/figures/alarm_in  figures_merge | CONGUARDIAN_FD=3 conguardian /var/figures/alarm  figures_split  &
sleep 1

nothing | domcat /var/figures/live   zero | python figures_show.py | multilog t n50 ./live   &
nothing | domcat /var/figures/minute zero | python figures_show.py | multilog t n50 ./minute &
nothing | domcat /var/figures/hour   zero | python figures_show.py | multilog t n50 ./hour   &
nothing | domcat /var/figures/day    zero | python figures_show.py | multilog t n50 ./day    &
nothing | domcat /var/figures/alarm  zero | python figures_show.py | multilog t n50 ./alarm  &

nothing | domcat /var/figures/live   zero | python figures_average.py 60    | domcat /var/figures/minute_in zero &
nothing | domcat /var/figures/minute zero | python figures_average.py 3600  | domcat /var/figures/hour_in   zero &
nothing | domcat /var/figures/hour   zero | python figures_average.py 86400 | domcat /var/figures/day_in    zero &

python figures_headings.py ./live   &
python figures_headings.py ./minute &
python figures_headings.py ./hour   &
python figures_headings.py ./day    &
python figures_headings.py ./alarm  &

nothing | domcat /var/figures/live   zero | python figures_mon.py mon_live   | domcat /var/figures/alarm_in zero &
nothing | domcat /var/figures/minute zero | python figures_mon.py mon_minute | domcat /var/figures/alarm_in zero &

python figures_tcptime.py  tcptime_cfg 10 | domcat /var/figures/live_in zero &

Graphing

The graph.cgi and show.cgi programs can be used (with gnuplot) to graph the data. These programs source data from multilog directories of figures_show.py output (with multilog timestamps).

Example graph: