Router Monitoring Automation

by J. Edward Durrett

A router is a key piece of network infrastructure without which there would not be a network. Misconfiguration,
unauthorized configuration, ill planned and undocumented changes, and exploitation and penetration all can impact
an organization’s ability to complete its mission. In addition, low market routers that are often found in
small offices and organizations with tight budgets are often not updated. Solutions for monitoring include
buying a nicer router, buying a software monitoring package and hiring a person or devote more labor to monitor the
router to ensure continuous business operation. Of course, that is a non-starter for an organization without a
large budget and it is not advisable even if there are funds for such an undertaking. Here is a low cost, no
labor, scalable approach to monitoring router configurations.

Assuming there is shell access to the router and it is running some version or derivative of Linux/Unix, setting
up continuous monitoring with automated evaluation and intelligent notification takes only an hour or two to set
up when writing a script from scratch. It is even faster with a pre-written scripti. The examples below are based on a Linux router.

Before getting to the script, here is a little technical information on how the script works. The router runs an
SSH server which we will log into. Since there is not a lot of space to install programs, we have to log in as
root. Logging in as a less privileged user and using Sudo would of course be a better option and I would use it if
possible.

Much the same as collecting evidence while doing incident response, we want to check our router following the
order of volatility. We do this by checking the non-persistent states of the router configuration using various
commands (ie what is in memory). Instead of parsing the command output line by line, running a checksum on the
output and comparing that checksum with a known good checksum. To get the checksum of the running iptables
configuration we do:


ssh -i .ssh/router_key root@192.168.x.x iptables -L | sha256sum 
96693ba95561ec4a388f55de96e216aef798a378017637456d80f35d45ccd54c

To get the check sum of the routing table, we do:

ssh -i .ssh/router_key root@192.168.x.x netstat -nr | sha256sum 
88eb9db15846e31394729111b88b52186bf1a36f1ec28f7422e69749401f1ee6


There script will alert if the checksums don’t match and there is an option to include what does not match.

Some other things to check are check connected devices against our inventory of known authorized devices:


ssh -i .ssh/router_key root@192.168.x.x cat /proc/net/arp | awk ‘{print$4}’ | sed -e 's/type//' 
b4:00:00:00:00:87
40:00:00:00:00:b3
00:00:00:00:00:95
...
ac:00:00:00:00:91
00:00:00:00:00:03
00:00:00:00:00:0d


And then listening ports:
 
ssh -i .ssh/router_key root@192.168.x.x netstat -an | grep LISTEN | awk '{print$4}' | sed -e 's/::/any/g
0.0.0.0:80
0.0.0.0:53
192.168.x.x:22
any:80
any:53


Then there is the persistent configurations. These are the configurations that come into effect when the router
boots. They are also the ones that, in my experience , are prone to be changed, either intentionally or mistakenly
by an administrator. The changes don’t manifest themselves until a power failure or other such event six
months down the road and no one knows why the internet does not work. And, of course, no one remembers making the
change.

We can verify the checksums of individual files, although that adds significant amount of resources putting up and
tearing down the connection, such as:

 
ssh -i /home/jed/.ssh/router_key root@192.168.3.1 cat /etc/config/firewall | sha256sum
3aa60db2624f5837a3b1fa838ae38387e366b231d2ce83dfc8761f429a631121


The second is to take advantage of what seems like a limitation, the small amount of space to work in. We can grab
an image of the disk, and then run a file integrity monitor on the directory we expand the image into. We do this
by using dd and piping the output back to the local machine:

 
ssh -i .ssh/router_key root@192.168.x.x "dd if=/dev/configdev bs=65536 conv=noerror,sync" | cat > image.img


However, in smaller routers, that is not possible. There is a permanent read only partition with the firmware and
default configurations and another partition with the customized configurations. These two get merged on boot to
create the run time root file system [1]. Taking a device level snapshot gives you one of the two, but not what
the router is actually running. Unless the merged runtime root device is recreated on the machine running the
checksum, the checksum test will fail.

I put this all together in a script. Running it with output to the terminal results in:

 
./router_monitor.sh verbose

WARNING: Bad checksum for /etc/config/firewall

firewall: Good checksum for runtime configuration ...

WARNING: Bad checksum - runtime configuration changed for routes!!!

ports: Good checksum for runtime configuration ...

devices: Good checksum for runtime configuration ...


I purposefully used some bad checksums while testing to ensure I got it right. However, it is boring to test a
router every day and look at the output, so passing the monitor command to the script makes it run silently and
only alert on a checksum mismatch. It is possible to have it alert you any way you can script – by mail, SMS,
XMMP, https POST, etc.

Here is the full script:

 
#!/bin/sh
#
# (c) 2017 jed@jedwardurrett.com

router=""
identity_file="" # Full path of SSH public key
ssh_user=""
files="" # File to check
checksum_file="./checksums" # Location of local checksums
mode=$1
##State Monitoring configuration. States require more setup and filtering. Set the state commands appropriately.
##Examples might not work - commands dependent on system and goals
check_states="YES" # Set to YES to enable NO to disable
checksum_state="./statesum" # Location of checksums for states
check_firewall="YES"
firewall_command="iptables -nL" # Get the active iptables rules
check_routes="YES"
routes_command="netstat -nr" # Get the full active routing table
check_devices="YES"
devices_command="cat /proc/net/arp | awk '{if (NR!=1) {print$1}}'" # Just get the mac list
check_ports="YES"
ports_command="netstat -nl | grep 'tcp\|udp' | awk '{print$4}' | tr -s : | cut -d : -f2" # Only get listening/open ports

##Advanced Checks - provide details on what changed
advanced_checks="NO" # Set to YES to enable. Requires addtional setup.

alert() {
#This is an example using mutt to send the alerts
#There are a lot of different ways to do this
#Configure what you like best
#printf "\nWARNING: Bad checksum for "$files"|"$states"\n" > alert.tmp
#mutt -F /home/user/.muttrc -s ALERT user@example.com < alert.tmp
#rm alert.tmp
}

##End of configurations

ssh_check_files() {
ssh -i $identity_file $ssh_user@$router cat $files | sha256sum | awk '{print$1}'
}

ssh_check_states() {
ssh -i $identity_file $ssh_user@$router $state_command | sha256sum | awk '{print$1}'
}

compare_checksums() {
original_checksum=`cat $checksum_file | grep $files | awk '{print$1}'`
if [ $checksum = $original_checksum ] ; then
cc=1
else
cc=0
fi
}

compare_statesums() {
original_checksum=`cat $checksum_state | grep $states | awk '{print$1}'`
if [ $checksum = $original_checksum ] ; then
cc=1
else
cc=0
fi
}

state_set() {
if [ $check_firewall = "YES" ] ; then
states="firewall"
fi
if [ $check_routes = "YES" ] ; then
states="$states routes"
fi
if [ $check_ports = "YES" ] ; then
states="$states ports"
fi
if [ $check_devices = "YES" ] ; then
states="$states devices"
fi
}

case $mode in

init)
checksum_count=0
: > $checksum_file
for files in $files; do
ssh_check_files | awk -v selif=$files '{print$1" "selif}' >> $checksum_file
checksum_count=$((checksum_count+1))
printf "\nChecksums added: "$checksum_count
done
: > $checksum_state
if [ $check_states = "YES" ] ; then
state_set
for states in $states; do
state_command=$(eval "echo \$$(echo ${states}_command)")
ssh_check_states | awk -v setats=$states '{print$1" "setats}' >> $checksum_state
checksum_count=$((checksum_count+1))
printf "\nChecksums added: "$checksum_count
done
fi
printf "\nChecksum generation complete!\n"
exit 0
;;

monitor)
for files in $files; do
checksum=`ssh_check_files`
compare_checksums
if [ $cc -eq 0 ] ; then
alert ##Functioon for alert - Nothing is done in monitor mode when everything is correct
fi
done
files=""
if [ $check_states = "YES" ] ; then
state_set
for states in $states; do
state_command=$(eval "echo \$$(echo ${states}_command)")
checksum=`ssh_check_states`
compare_statesums
if [ $cc -eq 0 ] ; then
alert ##Function for alert
fi
done
fi
;;

verbose)
for files in $files; do
checksum=`ssh_check_files`
compare_checksums
if [ $cc -eq 1 ] ; then
printf "\n"$files": Good checksum ... \n"
else
printf "\nWARNING: Bad checksum for "$files"\n"
fi
done
if [ $check_states = "YES" ] ; then
state_set
for states in $states; do
state_command=$(eval "echo \$$(echo ${states}_command)")
checksum=`ssh_check_states`
compare_statesums
if [ $cc -eq 1 ] ; then
printf "\n"$states": Good checksum for runtime configuration ... \n"
else
printf "\nWARNING: Bad checksum - runtime configuration changed for "$states"!!!\n"
fi
done
fi
;;

*|help|usage)
printf "This script monitors a remote router for changes in configuration and states\n"
printf "Usage: router_monitor.sh [option]\n"
printf "\nOptions are:\n"
printf "\nhelp:\n"
printf " This message.\n"
printf "\ninit:\n"
printf " Create initial checksum file. This will empty and repopulate an existing checksum file.\n"
printf "\nmonitor:\n"
printf " Monitor router at intervals as directed by cron and mail output if a change occurs.\n"
printf "\nverbose:\n"
printf " Print output to screen. Meant for one time checks and initial testing.\n"
exit 1
;;

esac

exit 0


A text version of this script can be found here <<<<<.

In conclusion, setting up a monitor on routers with shell access it fairly straight forward. The simple script
above monitors a router and notifies when changes are made. Everything is automated, so once it is set up there is
no additional labor cost in doing the monitoring itself. Reacting to alerts, however, will require intervention. If
a change management process is in place, this really should not happen unless it is in fact a true emergency.

References:

[1] https://wiki.openwrt.org/doc/techref/flash.layout







Copyright (c) 2019, Jason Edward Durrett - All content on this site, unless otherwise noted, is subject to this license.

Please contact me if any errors, such as erroneous / misleading content or missing / incomplete attribution, are found.