Bandwidth Monitoring Across Data Centers

by J. Edward Durrett

Recently, I needed a real-time report on bandwidth usage from multiple data centers. There are so many ways of
attaining and formatting this information that a book could be written on all of them. Basically, there are two
approaches: creating a solution from existing system tools or by obtaining additional software, either by
purchasing something proprietary or using open source.

The problem with the second approach is that it takes time to research and test products and requires additional
continual maintenance. In using a solution that is outside of the OS packages/ports, ie something proprietary,
that also means adding an additional step in the patch management process. Having been in the situation of
maintaining multiple versions of libc because a proprietary binary was slow to update, I can assure you this is
not something you should volunteer for.

Obtaining software, however, might be necessary if you or your staff doesn’t have the sufficient skill to
analyze the problem to determine exactly what it is needed. By throwing handfuls of dollars at the problem, you
can get a very complex tool that does a lot more than what the task requires.

In the case I am describing here, I wanted only to see in real time the bandwidth usage at various data centers.
Historical graphing, geo locating, packet analyses, etc, were not needed and were already handled by other tools
anyway. I just wanted a monitor I could see across the room that showed me the usage at various locations.

From the command line, I can just open a terminal, establish an ssh session, invoke tcp dump and pipe that to pv
to get the statistics I want:

xterm -title RDU -geometry 28x1 -fg green -fa 'Monospace' -fs 60 -e "ssh ■■■ sudo tcpdump -p -i em0  
-w - host ■■■.■■■.■■■.■■■ | pv -bert > /dev/null" &

This might seem like a lot is going on there, but it is just several small blocks assembled together for a
solution. And, most importantly, all the information to put together this solution already exists in the manual
pages and/or the configuration files. It is not secret! The main ingredients are xterm, ssh, sudo, tcpdump and pv.
If you have a graphical station, xterm should be there. For administering systems, ssh is must. For automation and
security, a properly configured sudo is required (I stress properly here). Not everyone might be familiar with
tcpdump in daily usage, but the manual pages are clearly written. In fact they are so well written, they are like
a textbook. Lastly, pv simply is a program than monitors data through a pipe and also has a nice manual page
explaining the options, although the flags -bert are really easy to remember if you grew up on Sesame Street.

In order to monotor all the data centers at once, I put these commands in a script to launch multiple xterms:

xterm -title RDU -geometry 28x1 -fg green -fa 'Monospace' -fs 60 -e "ssh ■■■ sudo tcpdump -p -i em0 -w -
host ■■■.■■■.■■■.■■■ | pv -bert > /dev/null" &
xterm -title EZE -geometry 28x1 -fg green -fa 'Monospace' -fs 60 -e "ssh ■■■ sudo tcpdump -p -i em0 -w -
host ■■■.■■■.■■■.■■■ | pv -bert > /dev/null" &
xterm -title FRA -geometry 28x1 -fg green -fa 'Monospace' -fs 60 -e "ssh ■■■ sudo tcpdump -p -i em0 -w -
host ■■■.■■■.■■■.■■■ | pv -bert > /dev/null" &
xterm -title SJC -geometry 28x1 -fg green -fa 'Monospace' -fs 60 -e "ssh ■■■ sudo tcpdump -p -i em0 -w -
host ■■■.■■■.■■■.■■■ | pv -bert > /dev/null" &
xterm -title DFW -geometry 28x1 -fg green -fa 'Monospace' -fs 60 -e "ssh ■■■ sudo tcpdump -p -i em0 -w -
host ■■■.■■■.■■■.■■■ | pv -bert > /dev/null" &

The result is live monitoring I can see across the room:

In conclusion, simple assembly of existing tools can often provide a quick and inexpensive solution. From a
management perspective, I am leery of requests to buy an expensive tool or invest the time in an over complicated
open source package to complete an easy task. Simple is always better and usually cheaper as well.

Copyright (c) 2019, Jason Edward Durrett - All content on this site, unless otherwise noted, is subject to this license.

Please contact me if any errors, such as erroneous / misleading content or missing / incomplete attribution, are found.