Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 2

This is currently the output.
First, i’m parsing the file with the changes.

Then i’m checking if the file with changes contains routers that are not in my infrastructure. If it does, i remove the changes for these routers from the modification plan (‚an error has been found and removed’)

Finally, i compare the remaining changes with existing configs of routers. If a match is found, the change doesn’t make sense.

def makedictionaryfromchangefile():
    
    #looping through list of configs and parsing the config
    with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          stext = configchangetext.split('\n')
          mydict = dict(map(lambda s: s.split(','), stext))
          Dict = { }
          for k, v in mydict.items():
            v = dict(z.split(";") for z in v.split("?"))
            Dict[k] =  {}
            y = 0
            for y in range(y,len(v)):
              eachitem = list(v.items())[y]       
              Dict[k][eachitem[0]] = eachitem[1]
    
    return Dict
def comparechangewithconfig():
    
    Dict = makedictionaryfromchangefile()
    #i need to get the list of devices for which change should made. This may be different than the list of routers that are in the environment(typos etc.)
    
    
    for i in Dict.keys():
        #print(' I am now inside the change dictionary, and i am in the main key ' + str(i) + str(Dict[i]))
        #this will just print parents 
        #for j in Dict[i].keys():
            #print(' i am printing what will be changed for ' + str(i) + ' and the changes are as follows ' + str(j))
        for l, m in Dict[i].items():
        	print('i am showing now how I want router ' + str(i) + ' config to be modified: ' + l + '  ' + m)
    realdevicelist = []
    for i in Dict.keys():
      #i add the devices to the list from dict main keys
      realdevicelist.append(i)
      for j in Dict[i].keys():
      	#i don't do anything here
        pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
    while True:
      try:
        x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
        for x in range(x,len(realdevicelist)):
          with open(realdevicelist[x], 'r') as reader:
            
            config = reader.read()
            my_config_as_list = config.split("\n")
      except FileNotFoundError as e:
          print('*******************************ERROR**************************************************************************')
          print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do not have a config. This error appeared in ' + realdevicelist[x])
          print('*******************************ERROR**************************************************************************')
          
          del Dict[realdevicelist[x]]
          realdevicelist.remove(realdevicelist[x])
          print('An error has been found and removed. ')
          
          
          continue
      else:
        print('After removing all errors from config file, this will be implemented. ')
        pp = pprint.PrettyPrinter(indent=4)
        pp.pprint(Dict)
        break 
    #print('This is the list again ' + str(realdevicelist))   
    for i in Dict.keys():
    	#print('I am in ' + str(i))
    	with open(i, 'r') as reader:
        #i'm on R1 now   
            config = reader.read()
            my_config_as_list = config.split("\n")
            for j, k in Dict[i].items():
              #print(j + ' and its value ' + k)
              parse = CiscoConfParse(my_config_as_list)
              answer = parse.find_objects('^' + j)
              #print(answer)
              if len(answer) > 0:
                #print('a subkey from Dict matches a parent object from the config. This does not tell us anything yet. Maybe the value will be modified, maybe not') 
                pass
              second_answer = parse.find_objects_w_child(j, k)
              if len(second_answer) > 0:
                print('i am on ' + i + ' in subkey ' + j + ' in value ' + k + ' and I have found a match in the existing configuration of ' + i + ' so i am not going to implement this change')
              else:
              	print(' no match  in config of ' + i + ' found when on ' + i + ' in subkey ' + j + ' in value ' + k + ' so i will implement this change ')
                #connect to i and implement config

Unfortunately, I’ve found a bug that overrides the change value for a given interface if this interface appears twice in the change config. I need to be able to create a dictionary with change values as a list, not just with single values. This needs further research.

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 1 – getting inputs

Hello

Unfortunately, making changes in Unicon is not idempotent. Unicon doesn’t realize natively if a change in the config is needed or not, so it will try to add config irrespectively of whether the config you want to add already exists on the target device or not. Having this feature can also help avoid mistakes such as overriding valid config by mistake.

I was thinking along these lines: I should take the existing running config and the implementation change config set, and analyze this change to see if a particular config change will actually modify the config or not. I believe that in order to do this, the running config needs to be analyzed in terms of keys and values, e.g: interface name is a key, ip address is a value.

int loop1 KEY
ip addr 1.1.1.1 255.255.255.0 VALUE

So the first condition is that if i want to implement a change on fa0/0, i should look for the key interface fa0/0, if the key doesn’t exist, it’s a real change and it should be implemented.

The second condition is that if i want to implement a change on a key that already exists, then I need to compare the values: if the new value is different, it’s a real change again and it should be implemented.
Fortunately, a library exists that already understands this key-value relationship: CiscoConfParse

Additionally, I’ve been thinking that to make any progress with my Python skills, i should try to use some cool features i didn’t know before. This time it is the lambda feature; it is basically an unnamed def. Here it helps me split the string into two parts, and the map function takes the first resulting string and maps it to the key, the second string becomes the value.

from ciscoconfparse import CiscoConfParse

#this is my config
parse = CiscoConfParse([ '!', 'interface FastEthernet0/0', ' ip address 192.168.0.1 255.255.255.0', '!', 'interface FastEthernet0/1', ' ip address 172.16.0.1 255.255.255.0'])

#Looking for condition 1: does a key(parent) exist? i'm parsing all objects to find words beginning with 'interf'

all_intfs = parse.find_objects(r"^interf")

#Looking for condition 2: if a key (parent) exists, does the same value (child) exists for this parent?

answer = parse.find_objects_w_child(parentspec=r"^interface FastEthernet0/0", childspec=r"192.168.0.1")
#this returns a list, so i need to see if the length of this list is >0

#the logic should be like this: if change has a key and value, look for the same key in existing config, if len>0, look for value for this key, if len>0, change not needed.
#let's say we have an upcoming parent-child config change, already parsed to the following string

mychangeinconfig = 'int fa0/0:ip address 192.168.0.1 255.255.255.0'
mylist = []
mylist.append(mychangeinconfig)

#we can try to make this change into a key: value dictionary

def list_to_dict(somelist):
return dict(map(lambda s: s.split(':'), somelist))
mydict = list_to_dict(mylist)
for k, v in mydict.items():
  myparent = k
  mychild = v

#or we can present this change config in the form of two separate strings, this doesn't matter too much.

mychangeparent = 'interface FastEthernet0/0'
mychangechild = 'ip address 192.168.0.1 255.255.255.0'
#now the key is int fa0/0 and the value is ip address 192.168.0.1 255.255.255.0
#we could now put this in the search algorithm
#answer = parse.find_objects_w_child(parentspec=myparent, childspec=mychild)

#LOGIC: if answer found in the form of list, the change is not needed.
if len(answer)>0:
  #change not needed
else:
  #implement change

#DONE Challenge1: to prepare a text file with following config changes:
R1
int fa0/0
 ip addr 172.16.0.1 255.255.255.0
int fa0/1
 ip addr 169.254.0.1 255.255.255.0
R2
logging buffered informational
R3
int vlan 100
 shut
@ and parse it such that the final form is the following:
{R1: {int fa0/0:ip addr 172.16.0.1 255.255.255.0, int fa0/1:ip addr 169.254.0.1 255.255.255.0}, R2: {logging buffered informational}, R3: {int vlan 100: shut}}


#Challenge2 TODO: to find out if the following changes should be implemented or if they already exist
change = {R1: {int fa0/0:ip addr 172.16.0.1 255.255.255.0}, R2: {logging buffered notifications}, R3: {int vlan 100: shut}} 

This should be fun with my low-end programming skills 😀

Challenge 1 – preparing the input with the new change

The input with the change needs to be prepared in a way that will be possible to parse.

#i cheated in preparing the change config set, but only a little. The thing is, i need to have clearly separated input to be able to parse it. Here i use commas to separate router names from keys, semicolons to separate keys from values and question marks to separate keys:values from new keys:values. Finally /n begins another router change config. 

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0\nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'

print('this is raw text input for my change' + text)

def text_2_dic(atext):
  stext = atext.split('\n')
  mydict = dict(map(lambda s: s.split(','), stext))
  Dict = { }
  for k, v in mydict.items():
    v = dict(x.split(";") for x in v.split("?"))
    for a,b in v.items():
      Dict[k] = {}
      Dict[k][a] = b
  return Dict
print('and this is final dictionary with the parsed change input: \n ' + str(text_2_dic(text))
C:\Users\A666773\Desktop>python listtest.py
this is raw text input for my change: R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0
R2,int fa0/0;ip address 192.168.0.2 255.255.255.0
R3,int fa0/0;ip address 192.168.0.3 255.255.255.0
and this is final dictionary with the parsed change input:
{'R1': {'int fa0/1': 'ip address 192.168.1.1 255.255.255.0'}, 'R2': {'int fa0/0': 'ip address 192.168.0.2 255.255.255.0'}, 'R3': {'int fa0/0': 'ip address 192.168.0.3 255.255.255.0'}}

This was actually wrong, because the final dictionary didn’t have the int fa0/0 change. The problem was in initialising the dictionary inside the loop rather than outside:

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0\nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'
  def text_2_dic(atext):
    stext = atext.split('\n')
    mydict = dict(map(lambda s: s.split(','), stext))

  print('+++++++++++++++++++++++++++++++++++++++++++\n\n')
  print('this is the predictionary \n' + str(mydict))
  print('+++++++++++++++++++++++++++++++++++++++++++\n\n')
  Dict = { }
  for k, v in mydict.items():
    v = dict(x.split(";") for x in v.split("?"))
    Dict[k] = {}
    for a,b in v.items():
      Dict[k][a] = b 
  return Dict
print('and this is final dictionary: \n ' + str(text_2_dic(text)))

OUTPUT:

Another problem appeared: what about configs which are one-line only? Not all configs have a parent and a child, like logging buffered in this case below. For now, i’ll use a trick with a semicolon and a empty space value. We’ll see what happens later. Maybe this will have to be reworked.

INPUT

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0?logging buffered; \nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'

OUTPUT

and this is final dictionary:
{'R1': {'int fa0/0': 'ip address 192.168.0.1 255.255.255.0', 'int fa0/1': 'ip address 192.168.1.1 255.255.255.0', 'logging buffered': ' '}, 'R2': {'int fa0/0': 'ip address 192.168.0.2 255.255.255.0'}, 'R3': {'int fa0/0': 'ip address 192.168.0.3 255.255.255.0'}}

This was still wrong though, because if i inserted into the input multiple changes for a single subkey (e.g. interface), these changes were not found in the final dictionary.

I need to find if dictionary v contains multiple items. If it does, i loop through these items creating tuplets (eachtuplet), and then i update the dictionary from those tuplets.

def text_2_dic(atext): 
  stext = atext.split('\n') 
  mydict = dict(map(lambda s: s.split(','), stext)) 
  print('+++++++++++++++++++++++++++++++++++++++++++\n\n') 
  print('this is the predictionary \n' + str(mydict)) 
  print('+++++++++++++++++++++++++++++++++++++++++++\n\n') 
  Dict = { } 
  for k, v in mydict.items(): 
    v = dict(x.split(";") for x in v.split("?")) 
    Dict[k] = {} 
    x = 0 
    for x in range(x,len(v)): 
      eachtuplet = list(v.items())[x] 
      Dict[k][eachtuplet[0]] = eachtuplet[1] 
return Dict

INPUT:

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/0;desc LAN?int fa0/0;shut?int fa0/1;ip address 192.168.1.1 255.255.255.0?logging buffered; \nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'

OUTPUT

And this is the final dictionary:
{'R1': {'int fa0/0': 'shut', 'int fa0/1': 'ip address 192.168.1.1 255.255.255.0', 'logging buffered': ' '}, 'R2': {'int fa0/0': 'ip address 192.168.0.2 255.255.255.0'}, 'R3': {'int fa0/0': 'ip address 192.168.0.3 255.255.255.0'}}

Adding pretty print…

import pprint
(...)
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(text_2_dic(text))

Obviously there are more scenarios to be accounted for (deleting with ‚no’ or nested commands like qos policies) but for now that’s enough. I can improve that later.

Challenge 2 – loading and parsing the configs

Now that we have this nice dictionary with the parsed config changes, we need to load the existing configs.

For now, i will be using mock (abbreviated device configs) and a mock (custom) change text file.

This is the beginning of the larger def that will later compare the inputs and return True or False.

def comparechangewithconfigof(routerfilelist):
  realdevicelist = routerfilelist.split('%')
  x = 0
#looping through list of configs and parsing the config
  for x in range(x,len(realdevicelist)):
    with open(realdevicelist[x], 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parsedconfig = CiscoConfParse(my_config_as_list)
#opening the change file and transforming into Dict
    with open('change.txt', 'r') as reader:
      configchangetext = reader.read()
      stext = configchangetext.split('\n')
      mydict = dict(map(lambda s: s.split(','), stext))
      Dict = { }
      for k, v in mydict.items():
        v = dict(z.split(";") for z in v.split("?"))
        Dict[k] = {}
        y = 0
        for y in range(y,len(v)):
          eachitem = list(v.items())[y]
          Dict[k][eachitem[0]] = eachitem[1]
print(Dict)


comparechangewithconfigof('R1%R2%R3')

Inputs are now ready. Now it’s time to compare the change input with the existing configs and ask if the change makes sense (worth implementing?). To be continued next week in part 2.

Building a pull-based config change notification system with Python and Slack, part 2 (Diffios)

In this part I would like to get a full report on Slack regarding what changed and who changed it.
Diffios will help me with the first part, while using the log buffer should help me with the who bit.

A short introduction to diffios:

tode@ubuntu:~$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import diffios
baseline = "oldconfig.bak"
comparison = "modifiedconfig.bak"
ignore = "ignore.txt"
diff = diffios.Compare(baseline, comparison, ignore)
print(diff.delta())
--- baseline
+++ comparison
+ logging buffered informational

This is pretty self-explanatory, but as you can see these two config files are only different in that i’ve added the logging buffered informational in between the writes and Diffio shows the delta. The ignore.txt files includes only 2 words: „Current configuration” because it doesn’t make sense to put size of config (which appears in the running config file) as delta.
Let’s update our report script now and run part2.py again:

Ok now we know what’s been changed, now on to who’s changed it. I can use the command show archive log config all

After the script is updated again, here’s the Slack output.

to be continued in Building a pull-based config change notification system with Python and Slack part 3 : Interacting with devices in Slack

Building a pull-based config change notification system with Python and Slack, part 1

Hello

Moving on one step forward from simple show commands, i’d like to be able to receive notifications if there’s been a change to a routers’s configuration. There are several ways of doing this but i’ll start simple: if i find that the config file has a new timestamp, i would like to do a backup of the new config and compare it line by line with the previous backup. I can also try to look into the log buffer of the router to see logged commands (example line: User:testuser logged command:no logging console).

Having all this info should be reason enough to send a notification, e.g. to slack. For this I can use the Notifiers library.
Finally, i can use this output as the definition of done when i start implementing changes using unicon, where config delta should be equal to the config command set of the implementation task. If it’s not equal, the implementation should be investigated.

let’s get down to work then…

Comparing file timestamps

import os

filename = 'testfile.txt'
currentstamp = os.stat(filename).st_mtime
print('This is the current timestamp of the config file ' + currentstamp)
with open('cachedstamps.txt', 'r') as reader:
  oldstamp = float(reader.read())
  print('This was the old timestamp from the archived timestamp file ' + oldstamp)
f = open('cachedstamps.txt', "w")
f.write(str(currentstamp))
if currentstamp != oldstamp:
  print('As you can see, the timestamps are different! It looks like this file has been modified')
else:
  print('this file has not been modified')

This was a nice idea until I realized that nvram: files don’t have a date attached to them.

Well, on to a new idea, copying the startup file, creating a hash out of the content and comparing the hashes.

I’ve created two scripts now: the first one copies the config and makes a hash out of it, then I modify the config on the router and run the second script, which gets the modified config, the new hash, and compares the old hash versus the new hash. If they are different, the config has been modifed.

Creating hashes from the old config

def makeoldhash():
  hasher = hashlib.md5()
  with open('oldconfig.bak', 'rb') as afile:
   buf = afile.read()
   hasher.update(buf)
  print(hasher.hexdigest())
  f = open('cashedhashes.txt', "w")
  f.write(str(hasher.hexdigest()))

def getoldconfig():
  realdevicelist = devicelist.split('%')
  print(realdevicelist)
  testbed = loader.load(testbedfile)
  x = 0
  for x in range(x,len(realdevicelist)):
    devicename = realdevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute(command)
    f = open("oldconfig.bak", "w")
    f.write(output)
    f.close()
    c.destroy()

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

getoldconfig()
makeoldhash()

Creating hashes from updated config

from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import hashlib


def getnewconfig():
  realdevicelist = devicelist.split('%')
  print(realdevicelist)
  testbed = loader.load(testbedfile)
  x = 0
  for x in range(x,len(realdevicelist)):
    devicename = realdevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute(command)
    f = open("modifiedconfig.bak", "w")
    f.write(output)
    f.close()
    c.destroy()

def makenewhash():
  hasher = hashlib.md5()
  with open('modifiedconfig.bak', 'rb') as afile:  
    buf = afile.read()
    hasher.update(buf)
  f = open('newhashes.txt', "w")
  f.write(str(hasher.hexdigest()))

def comparehash():
  with open('cashedhashes.txt', 'r') as reader:
    oldhash = str(reader.read())
    print(oldhash)
  with open('newhashes.txt', 'r') as reader:
    newhash = str(reader.read())
    print(newhash)
  if oldhash == newhash:
    print('identical! config has not been changed!')
  else:
    print('someone changed the config')

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

getnewconfig()
makenewhash()
comparehash()

Now i run the first script, modify the config on R1, then run the second script. I’m still using unicon so i have my old testbed.yml file with IPs, passwords. Unicon handles the connection for me so that i don’t need to worry about waiting for prompts etc.


python3 part1.py 'show run' testbed.yml R1

(here i modified the config)

python3 part2.py 'show run' testbed.yml R1

The result is:

9af95565fda2b8e4d40b04edc26d146c
a03c81972dd99787e7f31542f3a53dbe
someone changed the config

Now i need to build the part where the script notifies Slack that the first script has been run (always early in the morning, so that i get an updated hash), and then before midnight i need another notification that part2 has been run (where i get the new hash), the second script should then compare the hashes and tell me if the config has been changed. If it has been changed, another script should run, which should compare the config files and give me the delta. Ideally, it should also get some info from the router’s log buffer about the logging time and commands.

Notifying Slack

To notify Slack, i’ve reworked the comparehash method. Your Slack webhook url needs to be used so that the app can connect to your Slack channel.

def comparehash():
  with open('cashedhashes.txt', 'r') as reader:
    oldhash = str(reader.read())
    print('the old hash from this mornings config is ' + oldhash)
  with open('newhashes.txt', 'r') as reader:
    newhash = str(reader.read())
    print('the new hash from current config is ' + newhash)
  if oldhash == newhash:
    print('the hashes are identical! This means that config on this router has not been changed!')
    p = get_notifier('slack')
    p.notify(webhook_url='yourslackwebhookurl', message='config on this router has not been changed today!')
  else:
    print('hashes are different! This may mean that someone changed the config')
    p = get_notifier('slack')
    p.notify(webhook_url='yourslackwebhookurl', message='Hello network team. This is the report for R1, 12 oct 2020. Id like to report that the config on this router has been changed today! Investigating…')

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

getnewconfig()
makenewhash()
comparehash()

running the updated script again…

R1#
the old hash from this mornings config is 9af95565fda2b8e4d40b04edc26d146c
the new hash from current config is a03c81972dd99787e7f31542f3a53dbe
hashes are different! This may mean that someone changed the config

The next step: comparing the config change with Diffios and reporting the changes to slack + getting user logins from log buffer to see who has made the changes.
Finally, I would like to reverse the logic: it is the router which should notify a service on my script server (the moment the config has been changed), which in turn should trigger slack notification (=a push-based notification system) or at least the app should monitor the slack channel constantly (a continous pull-based system) for any inputs from the slack user, while the bot should answer with router outputs, something like:
Hello Tom, run ‚show ip int brief’ on R1
App should parse this input, execute the command on R1 and the bot should return the output in the channel.

Show commands on groups of devices using Unicon and .yml hostfiles

Hello

After I created a script in Netmiko that connected to a number of devices to create an uptime report, I thought it would be nice to do a similar thing using Unicon. It was a much easier job.

here’s the python code that does the magic:

from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys

def executeshowongroup(command,testbedfile,devicelist):
  realdevicelist = devicelist.split('%')
  print(realdevicelist)
  testbed = loader.load(testbedfile)
  x = 0
  
  for x in range(x,len(realdevicelist)):
    devicename = realdevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    c.execute(command)

if name == "main":
  command = sys.argv[1]
  testbedfile = sys.argv[2]
  devicelist = sys.argv[3]

executeshowongroup(command,testbedfile,devicelist)

and here’s the testbed.yml:

devices:
  myproxy:
   os: linux
   type: linux
   credentials:
    default:
     username: cisco
     password: cisco
    connections:
     cli:
      protocol: ssh
      ip: 1.1.1.1
      port: 722
   router1:
    os: ios
    type: router
    connections:
     defaults:
     class: unicon.Unicon
     cli:
      command: connect router1
      proxy: myproxy
   router2:
    os: ios
    type: router
    connections:
     defaults:
     class: unicon.Unicon
     cli:
      command: connect router2
      proxy: myproxy

Now I run the script with:

python3 unicongroupscript.py ‚show ip int brief’ newtestbed.yml router1%router2


It’s important to notice what is NOT in this script: gone are the time.sleep() commands, so i don’t need to worry if my proxy is particularly slow on any given day. I get the output much, much faster than in the case of netmiko. And the whole logic is done in python, unlike in Ansible where I’m artificially limited to the yaml syntax.

This is really, really cool stuff. One thing that is missing is the built-in group functionality in the testbed file, so scripts need to be run on individual hosts (or like in my case: with groups of hosts where delimiter between each host is %). I’ll try to talk to Cisco about this.

Creating a nice html-like uptime report with Pandas

This time it’s a dry-run, so instead of polling routers, i use ready strings ‚processor is …’. If you haven’t read the previous posts: in the full version of the script, i poll the router using Netmiko, issue the command ”show version”, take the bit with the uptime and process this paragraph to create a report, for example i want to know if the router has reloaded recently.

I’m using pandas with a nested dictionary to create a HTML table with values. If the script finds the word ”week”, and if the number > 4, it’s a success, otherwise it’s a partial failure (=<4) or a failure (word ”week” not found).

import re
import pandas as pd

test_results = {}
endresultstring = "processor is 3 days"
if bool(re.findall(r'(week)', endresultstring)):
   if int(str(re.findall(r'[0-9]+', endresultstring)[0])) > 4:
      test_results.update( {'R1' : {'uptime test': 'success'}})
   else:
      test_results.update( {'R1' : {'uptime test': 'partial failure'}})
else:
   test_results.update( {'R1' : {'uptime test': 'failure'}})

endresultstring = "processor is 3 weeks"
if bool(re.findall(r'(week)', endresultstring)):
   if int(str(re.findall(r'[0-9]+', endresultstring)[0])) > 4:
      test_results.update( {'R2' : {'uptime test': 'success'}})
   else:
      test_results.update( {'R2' : {'uptime test': 'partial failure'}})
else:
   test_results.update( {'R2' : {'uptime test': 'failure'}})

endresultstring = "processor is 5 weeks"
if bool(re.findall(r'(week)', endresultstring)):
   if int(str(re.findall(r'[0-9]+', endresultstring)[0])) > 4:
      test_results.update( {'R3' : {'uptime test': 'success'}})   
   else:
      test_results.update( {'R3' : {'uptime test': 'partial failure'}})
else:
   test_results.update( {'R3' : {'uptime test': 'failure'}})
df = pd.DataFrame(data=test_results)
df = df.fillna(' ').T
print(df)

Output is very nice:

Getting clear uptime data from Cisco router

Hello

Every larger IT company has some kind of infrastructure to monitor if routers are up or down. But in some companies, this data is relatively hard to get because the monitoring server can be hidden behind some ssl vpn gateways, and it takes up to 10 minutes to see if a router is/has been down, so nobody ever checks it. Due to how event engines are configured, if a router reloads, an incident will not be created and the support team will never know that a router has reloaded.

I don’t like being in the dark so I got down to work.

Of course, it is very easy to write up a script that gets output of show version from a cisco router to see the uptime. But the problem becomes more acute if you have hundreds of routers. Then you’d not be interested in plowing through hundreds of uptime lines, you’d want to get alarms only, like ”watch out, this router has been down recently because uptime is short”.
Because I will be running my script every now and again only, i’m only interested to see if the router has been down in the last 4 weeks so i cut off the details (days, minutes).

I rewrote the last script a bit to do just that: I take output of show version, find the uptime, turn uptime into a score and spit out the status: ok or not ok.

myrouters = {"R1": {"RouterOne": "172.16.0.1"}, "R2": {"RouterTwo": "172.16.0.2"}, "R3": {"RouterThree": "172.16.0.3"}, "R4": {"RouterFour": "172.16.0.4"}}

#this loop will go through the dictionary of dictionaries, connect to each router using its IP (subval), issue show version command
 
for mainkey in myrouters.keys():
    mainval = myrouters[mainkey]
    for subkey in mainval.keys():
      subval = mainval[subkey]
      print('i am now on router ' + subkey + ' , its IP address is ' + subval)
      net_connect.write_channel("connect " + subval + " \n")
      time.sleep(3)
      redispatch(net_connect, device_type="cisco_ios")
      net_connect.enable
      result = net_connect.send_command_timing("show ver")

#here i initiate an empty list, because re.findall returns a list

      endresult = []
      endresult = re.findall(r'processor .*?minutes', result)
      endresult.insert(0,subkey)
      endresultstring = endresult[1]

    #This string contains just the phrase 'processor is up 4 weeks'
    #next i query if the phrase contain the word 'week' and if it does, i take this string and make a list of digits from it, join the digits, thus making a number of weeks. There must be a better way to do this... 

      foundweeks = bool(re.findall(r'(week)', endresultstring))
      if foundweeks:
        numberweeks = re.findall(r'.*?weeks', endresultstring)
        listweeks = str(numberweeks)
        stringweeks = str(listweeks[2:23])
        processweeks = re.findall(r'\d', stringweeks)
        numberweeks = ''.join(map(str,processweeks))

    #here i calculate a score (weeks times 25)  
      
        finalscore = int(numberweeks)*25
        if finalscore>100:
            print("This router is just fine! It has been up over " + str(int(finalscore)/25) + " weeks")
        else:
            print("oh no! it's been down in the last 4 weeks! you'd better check this router")

And the output is as follows:

i am now on router R1 whose name is RouterOne and IP is 172.16.0.1
This router is just fine! It has been up over 11.0 weeks

i am now on router R2 whose name is RouterTwo and IP is 172.16.0.2
This router is just fine! It has been up over 41.0 weeks

i am now on router R3 whose name is RouterThree and IP is 172.16.0.3
This router is just fine! It has been up over 29.0 weeks

i am now on router R4 whose name is RouterFour and IP is 172.16.0.4
This router is just fine! It has been up over 28.0 weeks

Eventually I found this code to be so obnoxiously wordy that I rewrote the last part (taking the number of lines from 12 to 7) changing the logic slightly: if the word ‚week’ is found in the part of uptime output, the output will be empty; otherwise the router has been down lately and something is printed out, depending on whether the number of weeks is between 1 and 4, or below 1.

if bool(re.findall(r'(week)', endresultstring)):
   if int(str(re.findall(r'[0-9]+', endresultstring)[0])) > 4:
      pass
   else:
      print("oh no! it's been down in the last 4 weeks! you'd better check this router")
else:
   print("This router has been down in the last week!!!")

Iterating through dictionary of dictionaries to get data from multiple routers behind a custom proxy using Netmiko

I’ve been wanting to improve my netmiko script which connects to a custom (non-ssh) proxy, connects to a few routers there and outputs some info.

I’m doing a few things here: first is a dictionary of dictionaries myrouters. Then I create an embedded loop which first goes through the main key (R1, R2, R3) and then through the subkeys of the embedded dictionary (Berlinrouter etc.).

Then I use a simple regex re.findall(r’Uptime .?minutes’, result) to only get uptime and another regex to get the serial number from output of the show version command.

Finally, i want to know how long eigrp tunnels have been up (because it’s not necessarily the same as router uptime) but the string that I get from show ip eigrp neighour is a bit complex so I split it into a list and take some elements from the list. By the way the way I do this below is really not optimal so reworking could involve textfsm templates (mentioned a few posts back) . Unfortunately I can’t spontaneously come up with a nice regex to only get what i want from the output even after grepping it.

from netmiko import ConnectHandler
import time
from netmiko import redispatch
import re
jumpserver= {'device_type':'terminal_server','ip':'1.1.1.1','username':'cisco','password':'cisco','port':22,'global_delay_factor':2,'session_log': 'output.txt'}
net_connect=ConnectHandler(**jumpserver)
net_connect.find_prompt()
myrouters = {"R1": {"FrankfurtRouter": "192.168.0.1"}, "R2": {"BerlinRouter": "192.168.0.2"}, "R3": {"Hamburgrouter": "192.168.0.3"}}
for mainkey in myrouters.keys():
mainval = myrouters[mainkey]
for subkey in mainval.keys():
subval = mainval[subkey]
print('i am now on router ' + mainkey + ' whose name is ' + subkey + ' and IP is ' + subval)
net_connect.write_channel("connect " + subval + " \n")
time.sleep(3)
redispatch(net_connect, device_type="cisco_ios")
net_connect.enable
result = net_connect.send_command_timing("show ver")
endresult = re.findall(r'Uptime .?minutes', result) endresult.insert(0,subkey)
print(endresult)
time.sleep(1)
newresult = net_connect.send_command_timing("show version")
endresultser = re.findall(r'Processor .', newresult)
endresultser.insert(0,subkey)
print(endresultser)
time.sleep(1)
eigrpresult = net_connect.send_command_timing("show ip eigrp vrf 1001 nei 172.16.0.1 | i Tu1001")
eigrpresult.split()
print("For " + subkey + " eigrp datatunnel uptime is " + eigrpresult[55:62])
net_connect.write_channel("exit\n")
time.sleep(1)

and the output…

i am now in mainkey R1 whose name is FrankfurtRouter and IP is 192.168.0.1
[‚FrankfurtRouter’, ‚Uptime for this control processor is 11 weeks, 3 hours, 53 minutes’]
[‚FrankfurtRouter’, ‚Processor board ID FCZ3557C1AL’]
For FrankfurtRouter eigrp datatunnel uptime is 4w4d
i am now in mainkey R2 whose name is BerlinRouter and IP is 192.168.0.2
[‚BerlinRouter’, ‚Uptime for this control processor is 41 weeks, 2 days, 17 hours, 18 minutes’]
[‚BerlinRouter’, ‚Processor board ID FCZ8445C1AH’]
For BerlinRouter eigrp datatunnel uptime is 4w4d
i am now in mainkey R3 whose name is Hamburgrouter and IP is 192.168.0.3
[‚Hamburgrouter’, ‚Uptime for this control processor is 29 weeks, 21 hours, 38 minutes’]
[‚Hamburgrouter’, ‚Processor board ID FCZ368990W7’]
For Hamburgrouter eigrp datatunnel uptime is 2d06h

One important thing to mention here is that I use some time.sleep stuff, which I personally don’t like. It would be a better idea to use pexpect to wait for router prompts, but pexpect library won’t work on python in Windows. You’d need to run it from a Linux machine.

Vagaries of ikev2

A bit of a break from programming because i have oodles of work.

One of the main tasks where I work is creating and changing VPN tunnels with other companies so I normally feel pretty confident that I can set up a tunnel in a few minutes. However, recently I came across an interesting case where ikev1 and ikev2 behaved differently (presenting different show outputs) if the same error was present in the vpn config. Unfortunately, i started with ikev2 which showed a bizarre output so I wrongly assumed a bug and spent hours and hours trying to find the problem.
Let’s say we have multiple crypto map tunnels from the hub R1 to R2,R3, R4 with the crypto map entries in the following order:

crypto map MYMAP 12 ipsec isakmp

crypto acl R1>R2

permit ip 192.168.0.0 0.0.0.255 10.0.0.0 0.0.255.255

ip route 10.0.1.0 255.255.255.0 <R2 peer IP>

crypto map MYMAP 13 ipsec isakmp

crypto acl R1>R3

permit ip 192.168.0.0 0.0.0.255 172.16.0.0 0.0.0.255

crypto map MYMAP 14 ipsec isakmp

crypto acl R1>R4

permit ip 192.168.0.0 0.0.0.255 10.0.100.0 0.0.0.255

permit ip 192.168.0.0 0.0.0.255 192.168.1.0 0.0.0.255

ip route 10.0.100.0 255.255.255.0 <R4 peer IP>

If you have a lot of experience with vpn’s, you might have already spotted the problem. R4’s subnet is a subset of R2’s subnet so there will be a problem and it’s easy to see if you have 4 tunnels, but a bit more difficult if you have hundreds of tunnels.
Now a series of interesting things will happen: R1>R2 will be fine, same as R1>R3. In case of R1>R4, let’s additionally assume that in case of the first SA, the application on the side of R4 will try initiate a connection to the side of R1, but in case of the second SA, the application on R1 side will initiate a connection to R4. What will happen now?
In case of ikev1: second SA will form just fine, but the first SA will fail with error ‚Peer not found’ on R1, and Phase 2 mismatch on R4 because R4 will receive 10.0.0.0 proxy from R1 instead of 10.0.100.0. This is despite the fact that routing in both cases is fine (only routes to /24 networks are present through peers). This is because routing is secondary to ISAKMP and IPSEC negotiations.

In case of ikev2: both SAs will form (!) and you will see (crypto session remote <ip of r4> details) that packets from 10.0.100.0 are decrypted, but then these packets will disappear in the bitbucket.

I can’t tell you how much time I spent looking up the significance of error ”peer not found”. What it means basically is that when a peer router sends us phase 2 policies (proxies), the match is found for a different peer (higher in crypto map hierarchy).

I will try to lab this up this week.

ChatOps – Ansible gets status from router, notifies Slack channel

Hello

Just a teaser of the idea that i’m toying with while i’m still working hard on the automation post. The idea of chatops is that your automation scripts should send status notifications (or any other notifications) to a common workspace, such as Slack. I would love to see daily statuses of my routers in my slack channel:
R1: ok
R2: ok
R3: nok

The other direction is also possible (oh the chaos an attacker could wreak upon the network with this one) : it is possible to actually run scripts from Slack.

For now i’ve just registered my slack account, created an app, and ran a test curl command from my VM to the slack channel. I’ll try to built a slack notification into my ansible scripts now.

chatops

Obviously I couldn’t leave this at this unfinished stage 😀 I took my ansible playbook and modified it:

- name: add_entry_to_acl 
  hosts: testrouter
  tasks:
   - name: add_new_entry
     ios_config:
     lines:
     - "{{ acl20 }}" 
     parents: ip access-list extended permit_www
     before: ip access-list extended permit_www 
     save_when: modified
   - name: send notification to Slack
     local_action:
     module: slack
     token: <here enter your slack webhook token>
     channel: "#things"
     msg: "Name of the host is {{ ansible_net_hostname }} and the software version is {{ ansible_net_version }} while the platform is {{ ansible_net_model }}"

 

tode@ubuntu:~/ansiblefolder$ ansible-playbook aclplaybook.yml

PLAY [add_entry_to_acl] ********************************************************

TASK [Gathering Facts] *********************************************************
[WARNING]: Ignoring timeout(10) for ios_facts
[WARNING]: default value for `gather_subset` will be changed to `min` from
`!config` v2.11 onwards
ok: [testrouter]

TASK [add_new_entry] ***********************************************************
changed: [testrouter]

TASK [show clock] **************************************************************
ok: [testrouter]

TASK [send notification to Slack] **********************************************
ok: [testrouter -> localhost]

PLAY RECAP *********************************************************************
testrouter : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

 

and voila:

final_chat