Building a pull-based config change notification system with Python and Slack, part 1

Hello

Moving on one step forward from simple show commands, i’d like to be able to receive notifications if there’s been a change to a routers’s configuration. There are several ways of doing this but i’ll start simple: if i find that the config file has a new timestamp, i would like to do a backup of the new config and compare it line by line with the previous backup. I can also try to look into the log buffer of the router to see logged commands (example line: User:testuser logged command:no logging console).

Having all this info should be reason enough to send a notification, e.g. to slack. For this I can use the Notifiers library.
Finally, i can use this output as the definition of done when i start implementing changes using unicon, where config delta should be equal to the config command set of the implementation task. If it’s not equal, the implementation should be investigated.

let’s get down to work then…

Comparing file timestamps

import os

filename = 'testfile.txt'
currentstamp = os.stat(filename).st_mtime
print('This is the current timestamp of the config file ' + currentstamp)
with open('cachedstamps.txt', 'r') as reader:
  oldstamp = float(reader.read())
  print('This was the old timestamp from the archived timestamp file ' + oldstamp)
f = open('cachedstamps.txt', "w")
f.write(str(currentstamp))
if currentstamp != oldstamp:
  print('As you can see, the timestamps are different! It looks like this file has been modified')
else:
  print('this file has not been modified')

This was a nice idea until I realized that nvram: files don’t have a date attached to them.

Well, on to a new idea, copying the startup file, creating a hash out of the content and comparing the hashes.

I’ve created two scripts now: the first one copies the config and makes a hash out of it, then I modify the config on the router and run the second script, which gets the modified config, the new hash, and compares the old hash versus the new hash. If they are different, the config has been modifed.

Creating hashes from the old config

def makeoldhash():
  hasher = hashlib.md5()
  with open('oldconfig.bak', 'rb') as afile:
   buf = afile.read()
   hasher.update(buf)
  print(hasher.hexdigest())
  f = open('cashedhashes.txt', "w")
  f.write(str(hasher.hexdigest()))

def getoldconfig():
  realdevicelist = devicelist.split('%')
  print(realdevicelist)
  testbed = loader.load(testbedfile)
  x = 0
  for x in range(x,len(realdevicelist)):
    devicename = realdevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute(command)
    f = open("oldconfig.bak", "w")
    f.write(output)
    f.close()
    c.destroy()

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

getoldconfig()
makeoldhash()

Creating hashes from updated config

from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import hashlib


def getnewconfig():
  realdevicelist = devicelist.split('%')
  print(realdevicelist)
  testbed = loader.load(testbedfile)
  x = 0
  for x in range(x,len(realdevicelist)):
    devicename = realdevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute(command)
    f = open("modifiedconfig.bak", "w")
    f.write(output)
    f.close()
    c.destroy()

def makenewhash():
  hasher = hashlib.md5()
  with open('modifiedconfig.bak', 'rb') as afile:  
    buf = afile.read()
    hasher.update(buf)
  f = open('newhashes.txt', "w")
  f.write(str(hasher.hexdigest()))

def comparehash():
  with open('cashedhashes.txt', 'r') as reader:
    oldhash = str(reader.read())
    print(oldhash)
  with open('newhashes.txt', 'r') as reader:
    newhash = str(reader.read())
    print(newhash)
  if oldhash == newhash:
    print('identical! config has not been changed!')
  else:
    print('someone changed the config')

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

getnewconfig()
makenewhash()
comparehash()

Now i run the first script, modify the config on R1, then run the second script. I’m still using unicon so i have my old testbed.yml file with IPs, passwords. Unicon handles the connection for me so that i don’t need to worry about waiting for prompts etc.

python3 part1.py 'show run' testbed.yml R1

(here i modified the config)

python3 part2.py 'show run' testbed.yml R1

The result is:

9af95565fda2b8e4d40b04edc26d146c
a03c81972dd99787e7f31542f3a53dbe
someone changed the config

Now i need to build the part where the script notifies Slack that the first script has been run (always early in the morning, so that i get an updated hash), and then before midnight i need another notification that part2 has been run (where i get the new hash), the second script should then compare the hashes and tell me if the config has been changed. If it has been changed, another script should run, which should compare the config files and give me the delta. Ideally, it should also get some info from the router’s log buffer about the logging time and commands.

Notifying Slack

To notify Slack, i’ve reworked the comparehash method. Your Slack webhook url needs to be used so that the app can connect to your Slack channel.

def comparehash():
  with open('cashedhashes.txt', 'r') as reader:
    oldhash = str(reader.read())
    print('the old hash from this mornings config is ' + oldhash)
  with open('newhashes.txt', 'r') as reader:
    newhash = str(reader.read())
    print('the new hash from current config is ' + newhash)
  if oldhash == newhash:
    print('the hashes are identical! This means that config on this router has not been changed!')
    p = get_notifier('slack')
    p.notify(webhook_url='yourslackwebhookurl', message='config on this router has not been changed today!')
  else:
    print('hashes are different! This may mean that someone changed the config')
    p = get_notifier('slack')
    p.notify(webhook_url='yourslackwebhookurl', message='Hello network team. This is the report for R1, 12 oct 2020. Id like to report that the config on this router has been changed today! Investigating…')

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

getnewconfig()
makenewhash()
comparehash()

running the updated script again…

R1#
the old hash from this mornings config is 9af95565fda2b8e4d40b04edc26d146c
the new hash from current config is a03c81972dd99787e7f31542f3a53dbe
hashes are different! This may mean that someone changed the config

The next step: comparing the config change with Diffios and reporting the changes to slack + getting user logins from log buffer to see who has made the changes.
Finally, I would like to reverse the logic: it is the router which should notify a service on my script server (the moment the config has been changed), which in turn should trigger slack notification (=a push-based notification system) or at least the app should monitor the slack channel constantly (a continous pull-based system) for any inputs from the slack user, while the bot should answer with router outputs, something like:
Hello Tom, run ‚show ip int brief’ on R1
App should parse this input, execute the command on R1 and the bot should return the output in the channel.