Performance reporting with bash, gnuplot and LaTex

Performance reporting with bash, gnuplot and LaTex

In an earlier earlier blog post I demonstrated how to create a simple CPU performance graph using the TiKz LaTeX class. I soon found out that generating graphs for longer periods and for many servers using TiKz took a very long time to compile and I started to look for other solutions - GNUplot with the gnuplottex class proved to be the right solution.

It all starts out by collecting data on the individual servers using the following bash script as a cronjob, which saves the data to a NFS share:

#!/bin/bash
# Sarcopy script by Rasmus Edgar Joergensen 2012.05
sardir=/path/to/nfsshare/sardata/$HOSTNAME
cpudir=$sardir/cpu
iodir=$sardir/io
memdir=$sardir/mem
# Check if host dir has been created, create if not present
if [ ! -d $sardir ]; then
    mkdir -p $sardir/{cpu,mem,io}
fi
saday=$(ls -c /var/log/sa/sa[0-9][0-9] | head -n 1)
sadf -d -- -u $saday | cut -d';' -f3- > $cpudir/$(uname -n)-$(date '+%Y-%m-%d')-cpu.csv
sadf -d -- -r $saday | cut -d';' -f3- > $memdir/$(uname -n)-$(date '+%Y-%m-%d')-mem.csv
sadf -d -- -b $saday | cut -d';' -f3- > $iodir/$(uname -n)-$(date '+%Y-%m-%d')-io.csv

The reason it is run locally on the server is because sar data files are not backwards compatible with earlier versions of sadf and vice-versa.

This cron job runs every day a 23:55:

# Copy sardata for perf reports - Rasmus 2012.05
55 23 * * * /path/to/script/sarcopy > /dev/null 2>&1

Since I need to create graphs for many servers using SAR data extracted from the servers I created the following template:

% REGJ - 20120113
\documentclass{article}
\usepackage[danish]{babel}
\usepackage[utf8]{inputenc}
\usepackage{palatino}
\pdfcompresslevel=0
\pdfobjcompresslevel=0
\usepackage{fancyhdr}
\usepackage[left=1.5in, right=1in, top=1in, bottom=1in, includefoot, headheight=13.6pt]{geometry}
\setlength{\headheight}{0.6in}
\fancyheadoffset[LE,RO]{1cm}
\fancyhead[L]{Performance plots for SERVERNAME - TIME}% empty left
\fancyhead[R]{ % right
   \includegraphics[height=0.43in]{$HOME/bitmaps/logo.eps}
}
\fancyfoot[L] {\tiny Written by Rasmus Edgar Fink Jørgensen in \LaTeX/bash.}
\usepackage{tikz,pgfplots}
\usepackage{eurosym}
\usepackage{gnuplottex}
\usepgfplotslibrary{dateplot}
\pagestyle{fancy}
\begin{document}
\begin{figure}
\begin{gnuplot}[terminal=epslatex,terminaloptions={color colortext solid size 15.4cm,9.35cm}]
set datafile separator ";"
set ylabel 'CPU/wait time \% used'
set xdata time
set autoscale x
set yrange [0:100]
set timefmt "%Y-%m-%d %H:%M:%S"
set xrange ["DATEMIN":"DATEMAX"]
set style line 11 lc rgb '#808080' lt 1
set border 3 ls 11
set key box opaque vert right top
set tics
set style line 12 lc rgb'#808080' lt 0 lw 1
set grid back ls 12
set style line 1 lc rgb '#F33600' lt 1 lw 2.5 # Orange
set style line 2 lc rgb '#0DF300' lt 1 lw 2.5 # Green
set style line 3 lc rgb '#0005FD' lt 1 lw 2.5 # Blue
plot "/tmp/SERVERNAME-concatcpu.csv" using 1:3 title '\%user' w lines ls 1, '' using 1:5 title '\%sys' w lines ls 2, '' 
using 1:6 title '\%iowait' w lines ls 3
\end{gnuplot}
\end{figure}
\begin{itemize}
\item{user:}
Percentage of CPU utilization that occurred while executing at the user level (application).
\item{iowait:}
Percentage  of  time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
\end{itemize}
\newpage
\begin{figure}
\begin{gnuplot}[terminal=epslatex,terminaloptions={color colortext solid size 15.4cm,9.35cm}]
set datafile separator ";"
set autoscale ymax
set format y "%.f"
set xdata time
#set autoscale x
set ylabel "Mem. in KB"
set timefmt "%Y-%m-%d %H:%M:%S"
set xrange ["DATEMIN":"DATEMAX"]
set style line 11 lc rgb '#808080' lt 1
set border front 3 ls 11
set style line 12 lc rgb'#808080' lt 0 lw 1
set grid back ls 12
set key box opaque vert right top
set style fill
set style line 1 lc rgb '#F33600' lt 1 lw 2.5 # Orange
set style line 2 lc rgb '#0DF300' lt 1 lw 2.5 # Green
set style line 3 lc rgb '#0005FD' lt 1 lw 2.5 # Blue
set style line 4 lc rgb '#9B438E' lt 1 lw 2.5 # Purple
set tics
plot "/tmp/SERVERNAME-concatmem.csv" using 1:3 title 'kbmemused' w filledcurve x1 ls 1, '' using 1:6 title 'kbcached' w 
filledcurve x1 ls 2, '' using 1:2 title 'kbmemfree' ls 3 with filledcurve x1, '' using 1:5 title 'kbbuffers' w filledcur
ve x1 ls 4
\end{gnuplot}
\end{figure}
\begin{itemize}
\item{kbmemused:}
Amount  of used memory in kilobytes. This does not take into account memory used by the kernel itself.
\item{kbcached:}
Amount of memory used to cache data by the kernel in kilobytes.
\item{kbmemfree:}
Amount of free memory available in kilobytes.
\item{kbbuffers:}
Amount of memory used as buffers by the kernel in kilobytes.
\item{Extra info:}
Linux treats unused memory as a wasted resource and so uses as much RAM as it can to cache process/kernel information.
\end{itemize}
\newpage
\begin{figure}
\begin{gnuplot}[terminal=epslatex,terminaloptions={color colortext solid size 15.4cm,9.35cm}]
set datafile separator ";"
set ylabel "Transfers/request /s"
set format y "%.f"
set xdata time
set autoscale x
set yrange [0:*]
set timefmt "%Y-%m-%d %H:%M:%S"
set xrange ["DATEMIN":"DATEMAX"]
set style line 11 lc rgb '#808080' lt 1
set border 3 front ls 11
set key box opaque vert right top
set tics
set style line 12 lc rgb'#808080' lt 0 lw 1
set grid back ls 12
set style line 1 lc rgb '#F33600' lt 1 lw 2.5 # Orange
set style line 2 lc rgb '#0DF300' lt 1 lw 2.5 # Green
set style line 3 lc rgb '#0005FD' lt 1 lw 2.5 # Blue
plot "/tmp/SERVERNAME-concatio.csv" using 1:2 title 'tps' w lines ls 1, '' using 1:2 title 'rtps' w lines ls 2, '' using
 1:3 title 'wtps' w lines ls 3
\end{gnuplot}
\end{figure}
\begin{itemize}
\item{tps:}
Total number of transfers per second that were issued to physical devices.  A  transfer
is  an I/O request to a physical device. Multiple logical requests can be combined into
a single I/O request to the device.  A transfer is of indeterminate size.
\item{rtps:}
Total number of read requests per second issued to physical devices.
\item{wtps:}
Total number of write requests per second issued to physical devices.
\item{bwrtn/s:}
Total amount of data written to devices in blocks per second.
\end{itemize}
\newpage
\begin{figure}
\begin{gnuplot}[terminal=epslatex,terminaloptions={color colortext solid size 15.4cm,9.35cm}]
set datafile separator ";"
set ylabel "Transfers,Blocks /s"
set format y "%.f"
set xdata time
set autoscale x
set yrange [0:*]
set timefmt "%Y-%m-%d %H:%M:%S"
set xrange ["DATEMIN":"DATEMAX"]
set style line 11 lc rgb '#808080' lt 1
set border 3 front ls 11
set key box opaque vert right top
set tics
set style line 12 lc rgb'#808080' lt 0 lw 1
set grid back ls 12
set style line 1 lc rgb '#F33600' lt 1 lw 2.5 # Orange
set style line 2 lc rgb '#0DF300' lt 1 lw 2.5 # Green
plot "/tmp/SERVERNAME-concatio.csv" using 1:5 title 'bread/s' w lines ls 1, '' using 1:6 title 'bwrtn/s' w lines ls 2
\end{gnuplot}
\end{figure}
\begin{itemize}
\item{bread/s:}
Total number of transfers per second that were issued to physical devices.  A  transfer
is  an I/O request to a physical device. Multiple logical requests can be combined into
a single I/O request to the device.  A transfer is of indeterminate size.
\item{bwrtn/s:}
Total amount of data written to devices in blocks per second.
\end{itemize}
\end{document}

This template gets filled out, the data handled and the PDFs compiled by the following script:

#!/bin/bash
basedir=$(pwd)
csvpath=${basedir}/csv
outpath=${basedir}/out
texpath=${basedir}/texfiles
pdfpath=${basedir}/pdfs
datetex=$(date +'%Y-%m')
sarmin="2012-08-01"
sarmax="2012-08-31"
sarserver=$2
srvnamecut=$(echo $sarserver | awk -F"." '{print $1}')sar$(date +'%Y%m%d').tex

# Clean up

rm -f $outpath/*
rm -f $pdfpath/aux/*
rm -f $pdfpath/log/*

sartex() {
                sed -e 's/SERVERNAME/'"$sarserver"'/
                        s/DATEMIN/'"$sarmin"'/
            s/TID/'"$datetex"'/
                        s/DATEMAX/'"$sarmax"'/' $texpath/sarplot.tpl >                 $outpath/$srvnamecut
}


sargen() {
        # Concatentate sar data
        cat /path/to/csvdata/$customer/linux/$sarserver/cpu/*cpu* > /tmp/$sarserver-concatcpu.csv
        cat /path/to/csvdata/$customer/linux/$sarserver/mem/*mem* > /tmp/$sarserver-concatmem.csv
        cat /path/to/csvdata/$customer/linux/$sarserver/io/*io*  > /tmp/$sarserver-concatio.csv

        sartex

}

case $1 in

-sar)
        sargen
        ;;
*)
        echo "Please specify -sar"
        exit 1

esac

# Generate plot

cd $outpath ; latexmk -f -pdfdvi -e '$latex=q/latex --shell-escape %O %S/' -outdir=$outpath $srvnamecut > /dev/null 2>&1

cp $outpath/*.pdf $pdfpath

Which creates graphs that looks like this:

example