Wyróżnione

Networks can be fun (for life)

john_whitelbow

Hi!

My name is Tomek De Wille. This is my network automation blog.

Jordan Peterson says: ”you can be so much more than you are”. I would like my blog to reflect this sentence. Let’s learn as much as possible in the time we have.

At the moment I would like to focus on automation frameworks for networks with occasional use of Python. I will be trying to present some python features using real-life examples, which I will be then using to create some automation scripts for my network.

Automatic topology generation in GNS3,

Hello

This is just a blog entry stub, a placeholder for an idea that i’ve been toying with. Whenever I need to lab sth up in GNS3, it takes up to an hour to do a simple topology.
I would like to have a script with an xml file, where I set variables like the number of routers, the role of the router (branch, hq, other), the features that should be enabled (ospf, eigrp) etc etc. This shouldn’t be something very complex, but it should spare me the lab setup effort.
These should be 2 scripts:

  • one should create the routers in GNS3 (i hear that now you can do this using rest calls) and telnet to the default ports to set up basic SSH.
  • the other should derive the config from the role and feature set and ip address ranges.

This will be a poor man’s SDN for GNS3 😀

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 6: Nested configs and mass changes.

Hello

Today I wanted to see how multiple nested commands (e.g. archive > log config > notify syslog, archive > log config > logging enable) will be configured using my script.

I’ve discovered an interesting thing. With a config like this:

R1, archive, log config, notify syslog,
R1, archive,log config, logging enable,
R1, archive, log config , hidekeys,

Only one of the commands (the last one) would be implemented on the router. It seems that while the defaultdict can have a list of values under a key, its keys still need to be unique. There are a few ways to work around this:

  • adding randomized empty spaces in the subkeys (_log config, log config_, log config, ). This is actually quite nice because spaces have no impact on the ios,
  • cutting the last letter of the key in each command set (archive, archiv, archi)
  • trying to change the logic: having just Archive word as the key, and then [log config, notify syslog] as the values. But this would require a heavy rework of the code. Alas, it’s probably the best solution. So this would be just a defaultdict+defaultdict, rather than a defauldict+defaultdict+defaultdict. On a second thought even a single defaultdict would do: {R1: [archive, log config, notify syslog]}. Why I never thought about this before is beyond me. I made my code (and life) so much more complicated with this triple nested defaultdict.

Anyways, let’s do some mass implementations using the ’empty spaces’ workaround

This is the input:

tode@ubuntu:~/perfectscripts$ cat change.txt
 R1, interface FastEthernet 1/0, ip address 100.100.0.2 255.255.255.0, 
 R1, interface FastEthernet 1/0, no shut, 
 R1, crypto isakmp policy 10, encr aes, 
 R1, crypto isakmp policy 10, hash sha, 
 R1, crypto isakmp policy 10, lifetime 3600, 
 R1, crypto isakmp policy 10, group 5, 
 R1, crypto isakmp policy 10, auth pre-share, 
 R1, crypto isakmp key cisco address 0.0.0.0 0.0.0.0, , 
 R1, crypto ipsec transform-set MYSET esp-des esp-sha-hmac, , 
 R1, crypto map MYMAP 10 ipsec-isakmp, set peer 100.100.1.2,
 R1, crypto map MYMAP 10 ipsec-isakmp, set transform-set MYSET, 
 R1, crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R1, ip access-list extended 101, permit ip host 1.1.1.1 host 2.2.2.2, 
 R1, interface FastEthernet 1/0, crypto map MYMAP, 
 R1, ip route 2.2.2.2 255.255.255.255 100.100.0.1, , 
 R1, int loopback 1, ip address 1.1.1.1 255.255.255.255, 
 R1, ip route 100.100.1.2 255.255.255.255 100.100.0.1, , 
 R1, archive,  log config,   notify syslog,
 R1, archiv,  log config,   logging enable,
 R1, archi,  log config,   hidekeys,
 R2, interface FastEthernet 1/0, ip address 100.100.1.2 255.255.255.0, 
 R2, interface FastEthernet 1/0, no shut, 
 R2, crypto isakmp policy 10, encr aes, 
 R2, crypto isakmp policy 10, hash sha, 
 R2, crypto isakmp policy 10, lifetime 3600, 
 R2, crypto isakmp policy 10, group 5, 
 R2, crypto isakmp policy 10, auth pre-share, 
 R2, crypto isakmp key cisco address 0.0.0.0 0.0.0.0, , 
 R2, crypto ipsec transform-set MYSET esp-des esp-sha-hmac, , 
 R2, crypto map MYMAP 10 ipsec-isakmp, set peer 100.100.0.2,
 R2, crypto map MYMAP 10 ipsec-isakmp, set transform-set MYSET, 
 R2, crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R2, ip access-list extended 101, permit ip host 2.2.2.2 host 1.1.1.1, 
 R2, interface FastEthernet 1/0, crypto map MYMAP, 
 R2, ip route 1.1.1.1 255.255.255.255 100.100.0.2, , 
 R2, int loopback 1, ip address 2.2.2.2 255.255.255.255, 
 R2, ip route 100.100.0.2 255.255.255.255 100.100.1.1, , 
 R2, archive, log config, notify syslog,
 R2, archiv,  log config,   logging enable,
 R2, archi,  log config,   hidekeys,tode@ubuntu:~/perfectscripts$

And the output is…

tode@ubuntu:~/perfectscripts$ python3 MassChangeWithAmanda.py critical R1%R2
  interface FastEthernet 1/0
  ip address 100.100.0.2 255.255.255.0
 no shut
 crypto map MYMAP
 crypto isakmp policy 10
  encr aes
 hash sha
 lifetime 3600
 group 5
 auth pre-share
 crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 crypto map MYMAP 10 ipsec-isakmp
  set peer 100.100.1.2
 set transform-set MYSET
 match address 101
 ip access-list extended 101
  permit ip host 1.1.1.1 host 2.2.2.2
 ip route 2.2.2.2 255.255.255.255 100.100.0.1
 int loopback 1
  ip address 1.1.1.1 255.255.255.255
 ip route 100.100.1.2 255.255.255.255 100.100.0.1
 archive
  log config
  notify syslog
 log config
  logging enable
  log config 
  hidekeys
 I am trying to implement this change on router R1 Are you sure? yes
  interface FastEthernet 1/0
  ip address 100.100.1.2 255.255.255.0
 no shut
 crypto map MYMAP
 crypto isakmp policy 10
  encr aes
 hash sha
 lifetime 3600
 group 5
 auth pre-share
 crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 crypto map MYMAP 10 ipsec-isakmp
  set peer 100.100.0.2
 set transform-set MYSET
 match address 101
 ip access-list extended 101
  permit ip host 2.2.2.2 host 1.1.1.1
 ip route 1.1.1.1 255.255.255.255 100.100.0.2
 int loopback 1
  ip address 2.2.2.2 255.255.255.255
 ip route 100.100.0.2 255.255.255.255 100.100.1.1
 archive
  log config
  notify syslog
  archiv
  log config
  logging enable
  archi
  log config
  hidekeys
 I am trying to implement this change on router R2 Are you sure? yes

I can see one big problem here: because commands are grouped, the script is trying to insert the map on interface fa1/0 when the map doesn’t exist yet. This will not be implemented. It will be implemented if the same script runs again. Oh bother. Diffios change log confirms this:

tode@ubuntu:~/perfectscripts$ cat R1_change_delta.bak 
 --- baseline
 +++ comparison
 1: Current configuration : 2239 bytes
 2: interface FastEthernet1/0
 no ip address
 shutdown
 1: Current configuration : 2767 bytes
 2: access-list 101 permit ip host 1.1.1.1 host 2.2.2.2
 3: archive
 logging enable
 4: crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 5: crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 6: crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 5
 lifetime 3600
 7: crypto map MYMAP 10 ipsec-isakmp
 set peer 100.100.1.2
 set transform-set MYSET
 match address 101
 8: interface FastEthernet1/0
 ip address 100.100.0.2 255.255.255.0
 9: interface Loopback1
 ip address 1.1.1.1 255.255.255.255
 10: ip route 100.100.1.2 255.255.255.255 100.100.0.1
 11: ip route 2.2.2.2 255.255.255.255 100.100.0.1
 tode@ubuntu:~/perfectscripts$ cat R2_change_delta.bak 
 --- baseline
 +++ comparison
 1: Current configuration : 1225 bytes
 2: interface FastEthernet1/0
 no ip address
 shutdown
 1: Current configuration : 1791 bytes
 2: access-list 101 permit ip host 2.2.2.2 host 1.1.1.1
 3: archive
 logging enable
 notify syslog contenttype plaintext
 4: crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 5: crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 6: crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 5
 lifetime 3600
 7: crypto map MYMAP 10 ipsec-isakmp
 set peer 100.100.0.2
 set transform-set MYSET
 match address 101
 8: interface FastEthernet1/0
 ip address 100.100.1.2 255.255.255.0
 9: interface Loopback1
 ip address 2.2.2.2 255.255.255.255
 10: ip route 1.1.1.1 255.255.255.255 100.100.0.2
 11: ip route 100.100.0.2 255.255.255.255 100.100.1.1
 tode@ubuntu:~/perfectscripts$  

After small edits, let’s run the script twice and read the change log from the 2nd run.

tode@ubuntu:~/perfectscripts$ cat R1_change_delta.bak 
 --- baseline
 +++ comparison
 tode@ubuntu:~/perfectscripts$ cat R2_change_delta.bak 
 --- baseline
 +++ comparison

This is nice, nothing changes during the 2nd run. Let’s now check if the script is actually trying to implement config during the 2nd (and every subsequent) run.
It is. Oh boy. The reason is that the input file needs to be sanitized. If it’s not (if change commands don’t reflect the existing config 100%) the script will not realize these are the same commands and will try to implement them again.
This is the sanitized input file:

tode@ubuntu:~/perfectscripts$ cat change.txt 
 R1,crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R1,access-list 101 permit ip host 1.1.1.1 host 2.2.2.2, , 
 R1,interface FastEthernet1/0, ip address 100.100.0.2 255.255.255.0, 
 R1,interface FastEthernet1/0, no shut, 
 R1,interface FastEthernet1/0, crypto map MYMAP, 
 R1,ip route 2.2.2.2 255.255.255.255 100.100.0.1, , 
 R1,interface Loopback1, ip address 1.1.1.1 255.255.255.255, 
 R1,ip route 100.100.1.2 255.255.255.255 100.100.0.1, ,
 R1,archive, log config,  notify syslog,
 R1,archiv,  log config,   logging enable,
 R1,archi,  log config,   hidekeys,
 R2,crypto isakmp policy 10, encr aes, 
 R2,crypto isakmp policy 10, hash sha, 
 R2,crypto isakmp policy 10, lifetime 3600, 
 R2,crypto isakmp policy 10, group 5, 
 R2,crypto isakmp policy 10, authentication pre-share, 
 R2,crypto isakmp key cisco address 0.0.0.0 0.0.0.0, , 
 R2,crypto ipsec transform-set MYSET esp-des esp-sha-hmac, , 
 R2,crypto map MYMAP 10 ipsec-isakmp, set peer 100.100.0.2,
 R2,crypto map MYMAP 10 ipsec-isakmp, set transform-set MYSET, 
 R2,crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R2,access-list 101 permit ip host 2.2.2.2 host 1.1.1.1, ,  
 R2,interface FastEthernet1/0, ip address 100.100.1.2 255.255.255.0, 
 R2,interface FastEthernet1/0, no shut,  
 R2,interface FastEthernet1/0, crypto map MYMAP, 
 R2,ip route 1.1.1.1 255.255.255.255 100.100.0.2, , 
 R2,interface Loopback1, ip address 2.2.2.2 255.255.255.255, 
 R2,ip route 100.100.0.2 255.255.255.255 100.100.1.1, , 
 R2,archive, log config,  notify syslog,
 R2,archiv,  log config,   logging enable,
 R2,archi,  log config,   hidekeys,tode@ubuntu:~/perfectscripts$ 

and the live output for R2:

crypto isakmp policy 10
  hash sha
 interface FastEthernet1/0
  no shut
 archiv
   log config
    logging enable
 archi
   log config
    hidekeys
 I am trying to implement this change on router R2 Are you sure?

Something is still left. Hash sha is the automatic choice, so doesn’t appear in the config file. This is not something that we need to handle, because sha should no longer be used for VPNs (obsolete).
No shut needs a separate code fix, because if a port is open, there is no command ‚no shut’ on the port.
For the nested commands i don’t have a good solution yet. I believe the whole code needs to be reworked (a single defaultdict with a list of values) for idempotency to work properly.

RAW CODE

Here’s the code with the ‚no shut’ fix:

#!/usr/bin/python3

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging
from gtts import gTTS
import re

LEVELS = {'debug': logging.DEBUG,
          'info': logging.INFO,
          'warning': logging.WARNING,
          'error': logging.ERROR,
          'critical': logging.CRITICAL}

level_name = sys.argv[1]
level = LEVELS.get(level_name, logging.NOTSET)
logging.basicConfig(level=level)
unicondevicelist = sys.argv[2].split("%") 

#loading the testbed file with router management IPs
testbed = loader.load('testbed.yml')

#downloading router configuration files and storing them away as .bak files
#x = 0
for x in range(0,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()
with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            dataline = line.split(',')
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
          
logging.info('\n')
for k, v in my_dict.items():
  logging.info('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  my_new_dict = my_dict[k].items()
  logging.info('\n')
  for item in my_new_dict:
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        logging.info(newstringordict)
      else:
        for k in newstringordict.keys():
          logging.info(k)
realdevicelist = []
for i in my_dict.keys():
  realdevicelist.append(i)
  for j in my_dict[i].keys():
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
m = 1
while True:  
  try:
    x = 0       
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    if m==1:
      text = "I found " + str(m) + " change for a router that is not in the testbed."
    if m > 1:
      text = "I found another error. "
    logging.critical('\n')
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    m = m + 1
    continue
  else:
    for k, v in my_dict.items():
      my_new_dict = my_dict[k].items()
      for item in my_new_dict:
      #item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            logging.info(newstringordict)
          else:
            for z in newstringordict.keys():
              logging.info(z)       
    break        
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      my_new_dict = my_dict[router].items()
      for tpl in my_new_dict:
        interface = tpl[0]
        cfgs = tpl[1]
        for key in cfgs.keys():
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #logging.info('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #logging.warning('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    pass
                  elif key == ' no shut':
                     findshut = ' shut'
                     shutanswer = parse.find_objects_w_child(interface, findshut)
                     if len(shutanswer) > 0:
                        pass
                     else:
                     	pass
                  else:
                    logging.warning(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                    my_final_dict[router][interface][key] = cfgs[key]
                    #connect to router and implement interface, key
          else:
                logging.warning(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement
finalstatement = ''
for k, v in my_final_dict.items():
  my_newfinal_dict = my_final_dict[k].items()
  globlist = []
  for item in my_newfinal_dict:
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        logging.info(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          n = newstringordict.get(j)
          globlist.append(n)
  c = testbed.devices[k]
  c.connect(log_stdout=False)
  for line in globlist:
  	print(line)
  decision = input("I am trying to implement this change on router " + k + " Are you sure? " )
  if "yes" in decision:
    output = c.configure(globlist)
    if 'Invalid' in output:
      logging.critical(output)
  else:
    pass
  c.destroy()

unicondevicelist = realdevicelist
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'
    c = testbed.devices[afterchanges]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    #time.sleep(1)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    diff = diffios.Compare(beforechanges, afterchanges)
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'
    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    os.remove(afterchanges)
    os.remove(beforechanges)




Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 5: interactive changes and more.

This adventure is coming to an end 🙂 It’s been definitely nice as an intellectual exercise, but one can clearly see the downsides: this is not actually network as code, but rather ”network as one messy change text file”. Of course, this text file could be made nicer in excel and underneath it would be a comma-delimited processable csv, but at its best it would be a ”network as a colourful excel file”. I’ve created a monster 😀

But hey, we’re not quite finished making the monster stronger and… weirder. I’ve decided to be bold and added a female voiceover that comments on the various stages of the script processing. Also, the script is now interactive.

#!/usr/bin/python3

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging
from gtts import gTTS

LEVELS = {'debug': logging.DEBUG,
          'info': logging.INFO,
          'warning': logging.WARNING,
          'error': logging.ERROR,
          'critical': logging.CRITICAL}


level_name = sys.argv[1]
level = LEVELS.get(level_name, logging.NOTSET)
logging.basicConfig(level=level)

unicondevicelist = sys.argv[2].split("%")
#print(unicondevicelist) 
text = "This change will be executed on 3 routers: router 1, router 2, router 3. I am going to look at the change text file now "
language =  'en'
speech = gTTS(text = text, lang = language, slow = False)
speech.save("text.mp3")
os.system("play text.mp3")

#loading the testbed file with router management IPs
testbed = loader.load('testbed.yml')

#downloading router configuration files and storing them away as .bak files
x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    f = open(devicename, "w")
    text = "I downloaded a config file from a router."
    language =  'en'
    speech = gTTS(text = text, lang = language, slow = False)
    speech.save("text.mp3")
    os.system("play text.mp3")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()
text = "I now downloaded configs of all routers. I am going to parse this config now."

language =  'en'
speech = gTTS(text = text, lang = language, slow = False)
speech.save("text.mp3")
os.system("play text.mp3")

#parsing the file with the changes
with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            dataline = line.split(',')  
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            
time.sleep(1)
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
logging.info('***************************************************************************************************************************')
for k, v in my_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  logging.info('\n')
  for item in my_new_dict:
    #item is a tuple consisting of string and default dict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        logging.info(newstringordict)
      else:
        for k in newstringordict.keys():
          logging.info(k)
  time.sleep(2)
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('\n\n\n')        	      
logging.info('***************************************************************************************************************************')
logging.info('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')

realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
m = 1
while True:  
  try:
    x = 0
         
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    if m==1:
      text = "I found " + str(m) + " change for a router that is not in the testbed."
    if m > 1:
      text = "I found another error. "
    
    language =  'en'
    speech = gTTS(text = text, lang = language, slow = False)
    speech.save("text.mp3")
    os.system("play text.mp3")
    logging.critical('\n')
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    logging.info('*******************************INFO****************************************************************************')
    logging.info('An error has been found and removed. ')
    logging.info('*******************************INFO****************************************************************************')
    logging.info('\n')
    m = m + 1
    continue
  else:
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('****************************SANITISE FINISHED **************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************SANITISE RESULTS PHASE**********************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    for k, v in my_dict.items():
      logging.info('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      for item in my_new_dict:
      #item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            logging.info(newstringordict)
          else:
            for z in newstringordict.keys():
              logging.info(z)
    logging.info('\n')          
    logging.info('************************************************************************************************************************')
    logging.info('*******************************END OF PLAN*********************************************************************************')
    logging.info('************************************************************************************************************************')
    break 
logging.info('\n')          
logging.info('***************************************************************************************************************************')
logging.info('*******************************COMPARE PHASE*******************************************************************************')
logging.info('***************************************************************************************************************************')
    
logging.info('***************************************************************************************************************************')
logging.info('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
logging.info('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      my_new_dict = my_dict[router].items()
      for tpl in my_new_dict:
        interface = tpl[0]
        cfgs = tpl[1]
        for key in cfgs.keys():
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #logging.info('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #logging.warning('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #logging.warning('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    logging.warning(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                    my_final_dict[router][interface][key] = cfgs[key]
                    #connect to router and implement interface, key
          else:
                logging.warning(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement
finalstatement = ''

for k, v in my_final_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  logging.info('\n')
  globlist = []
  for item in my_newfinal_dict:
    #item is a tuple consisting of a string and defaultdict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        logging.info(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #logging.info(j)
  c = testbed.devices[k]
  c.connect(log_stdout=False)
  finalstatement += ' On ' 
  finalstatement += k
  finalstatement += ' i have tried to configure '
  finalstatement += str(globlist)
  finalstatement += ' \n\n '
  #here implement interactivity
  
  text = "I need you to confirm a configuration change on router " + k + " in the terminal window. Type yes to confirm"
  language =  'en'
  speech = gTTS(text = text, lang = language, slow = False)
  speech.save("text.mp3")
  os.system("play text.mp3")
  decision = input("I am trying to implement the following on router " + k + " " + str(globlist) + " Are you sure you want this? " )
  if "yes" in decision:
    output = c.configure(globlist)
    if 'Invalid' in output:
      logging.critical(output)
      text = "I have found some invalid commands. God damn it, incompetent dumbasses"
      language =  'en'
      speech = gTTS(text = text, lang = language, slow = False)
      speech.save("text.mp3")
      os.system("play text.mp3")
  else:
    pass
  #here implement check if weird return from router, e.g. invalid something
  c.destroy()

logging.info(finalstatement + '\n\n')
logging.warning('***************COMPARE IN PROGRESS****FOR A DETAILED CHANGE LOG FILE SEE CHANGE_DELTA.BAK FILES*******************')


unicondevicelist = realdevicelist
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'
    c = testbed.devices[afterchanges]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    time.sleep(2)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    #time.sleep(10)
    baseline = beforechanges
    time.sleep(3)
    comparison = afterchanges
    time.sleep(3)
    #ignore = 'ignore.txt'
    diff = diffios.Compare(baseline, comparison)
    time.sleep(1)
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'
    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    os.remove(afterchanges)
    os.remove(beforechanges)
logging.info('*******************************THIS IS THE END OF ALL OUTPUT**********************************************')
text = "I have finished executing the changes and prepared a detailed change log. Amanda out"
language =  'en'
speech = gTTS(text = text, lang = language, slow = False)
speech.save("text.mp3")
os.system("play text.mp3")

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 4: implementation of logic fixes

This time the script connects to routers in GNS3, downloads the config, parses the change file, gets the delta (makes a list of commands from it for each router) and implements it using Unicon configure.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys


unicondevicelist = ['R1', 'R2', 'R3']
print(unicondevicelist[0])
testbed = loader.load('testbed.yml')
x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    c.destroy()

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
            print(dataline)
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('\n\n\n')        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for z in newstringordict.keys():
              print(z)
    print('\n')          
    print('***************************************************************************************************************************')
    print('*******************************END OF PLAN*********************************************************************************')
    print('***************************************************************************************************************************')
    
    break 

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
print('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key)
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change')
                  pass
                else:
                  #print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                  my_final_dict[router][interface][key] = ' '
                  #print(router + '\n ' + interface + '\n ' + key + '\n')
                  #connect to router and implement interface, key


for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  globlist = []
  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        print(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #print(j)
  print(globlist)
  print('i am in ' + str(k) + '\n')
  c = testbed.devices[k]
  c.connect()
  output = c.configure(globlist)
  c.destroy()

There is some interesting stuff that’s happening here, and still the script keeps implementing one-line commands, such as logging buffered informational, which needs to be fixed. A slight change in the code, then:

for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key + ' and this is value ' + cfgs[key])
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #print('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                    my_final_dict[router][interface][key] = cfgs[key]
                   
          else:
                print(interface + ' not found on router ' + router)
                my_final_dict[router][interface][key] = cfgs[key]
                
for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  globlist = []
  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        print(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #print(j)
  print(globlist)
  print('i am in ' + str(k) + '\n')
  c = testbed.devices[k]
  c.connect()
  output = c.configure(globlist)
  c.destroy()

I’ve changed two things here: first i added a condition where the parent (like logging buffered informational command) was found in the config but then the value of the tuple is empty, because there can be no subcommand here. Secondly, i’ve added a general Else statement, because sometimes the parent is not found at all (like a missing interface, or a missing general command), and it will have to be implemented.

The dictionary my_final_dict ”gathers” all the data for which implementation conditions are met.

The next step was to log all the changes resulting from the change.txt and remove the interim files. To this end, i copy the files to .bak files at the beginning of the script, then I download the config at the end of the script once again from all routers and do a final diffios compare.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time



unicondevicelist = ['R1', 'R2', 'R3']
testbed = loader.load('testbed.yml')
x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
            print(dataline)
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('\n\n\n')        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for z in newstringordict.keys():
              print(z)
    print('\n')          
    print('***************************************************************************************************************************')
    print('*******************************END OF PLAN*********************************************************************************')
    print('***************************************************************************************************************************')
    
    break 

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
print('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key + ' and this is value ' + cfgs[key])
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #print('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                    my_final_dict[router][interface][key] = cfgs[key]
                    #print(router + '\n ' + interface + '\n ' + key + '\n')
                    #connect to router and implement interface, key
          else:
                print(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement

finalstatement = ''
for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  globlist = []

  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        print(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #print(j)
 
  c = testbed.devices[k]
  c.connect()
  finalstatement += ' On ' 
  finalstatement += k
  finalstatement += ' i have tried to configure '
  finalstatement += str(globlist)
  finalstatement += ' \n\n '
  

  output = c.configure(globlist)
  c.destroy()

print(finalstatement + '\n\n')
#now download new config again and diffios against old config to see what has really been configured.

print('***************FINAL COMPARE PHASE, FOR A DETAILED CHANGE LOG FILE SEE CHANGE_DELTA.BAK FILES*******************')


unicondevicelist = ['R1', 'R2']
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'

    c = testbed.devices[afterchanges]
    c.connect()
    output = c.execute('show run')
    time.sleep(2)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    #time.sleep(10)


    baseline = beforechanges
    
    time.sleep(3)
    comparison = afterchanges
    
    time.sleep(3)
    #ignore = 'ignore.txt'
    diff = diffios.Compare(baseline, comparison)
    time.sleep(1)
    #print('****************THIS IS DIFFIOS FOR DELTA ' + afterchanges + '***********************************************')
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'

    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    #print('****************THIS IS END OF DIFFIOS FOR ' + afterchanges + '***********************************************')
    os.remove(afterchanges)
    os.remove(beforechanges)


print('*******************************THIS IS THE END OF ALL OUTPUT**********************************************')

So if my change input is the following command batch (which includes fake routers, fake interfaces, bad commands (like service-policy input QOS which will not be implemented because policy QOS doesn’t exist):

tode@ubuntu:~$ cat change.txt
R1,interface FastEthernet2/0, shut,
R1,interface FastEthernet0/0, description LAN,
R1,interface FastEthernet1/0, shut,
R1,interface FastEthernet1/0, service-policy input QOS,
R2,interface FastEthernet2/0, ip address 192.168.0.1 255.255.255.0,
R2,interface FastEthernet2/0, description LAN,
R2,interface FastEthernet0/2, description WAN,
R2,interface FastEthernet0/0, description DMZ,
R8,interface FastEthernet4/0, description LAN,
R9,interface Loopback1, description LOOP,
R9,interface Loopback1, ip address 1.1.1.1 255.255.255.0,
R2,interface Loopback1, description LOOP,
R1,logging buffered errors, , 

Then my real change output is this:

tode@ubuntu:~$ cat R1_change_delta.bak
--- baseline
+++ comparison


--- Current configuration : 2239 bytes
--- logging buffered informational

+++ Current configuration : 2249 bytes
+++ interface FastEthernet0/0
+++  description LAN
+++ logging buffered errors

and this:

tode@ubuntu:~$ cat R2_change_delta.bak
--- baseline
+++ comparison


--- Current configuration : 1225 bytes
    interface FastEthernet2/0
--- no ip address

+++ Current configuration : 1337 bytes
    interface FastEthernet0/0
+++ description DMZ
    interface FastEthernet2/0
+++ description LAN
+++ ip address 192.168.0.1 255.255.255.0
    interface Loopback1
+++ description LOOP
+++ no ip address

Next steps

  • I should probably create an error.log file as well to collect all the stuff that is wrong with the input (non-existing routers, failed attempts to create interfaces, configs which were already there on the router so were not implemented etc.).
  • I should also think about deletion commands (no ip address etc.), and multiple nested commands (like QoS).
  • Maybe the script should also ask the user if he agrees to implement config whenever there is a need to do it (interactive change).
  • I should get rid of all hardcoded values


By the way the script breaks if there is an end of line character at the end of change.txt
This can be fixed with the following commands in vi before saving.

set noeol
set nofixendofline

This is now a version with the logging implemented and some hardcoded values removed.

#!/usr/bin/python3

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging

LEVELS = {'debug': logging.DEBUG,
          'info': logging.INFO,
          'warning': logging.WARNING,
          'error': logging.ERROR,
          'critical': logging.CRITICAL}


level_name = sys.argv[1]
level = LEVELS.get(level_name, logging.NOTSET)
logging.basicConfig(level=level)

unicondevicelist = sys.argv[2].split("%")
print(unicondevicelist) 


testbed = loader.load('testbed.yml')


x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            dataline = line.split(',')  
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            
time.sleep(1)
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
logging.info('***************************************************************************************************************************')
for k, v in my_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  logging.info('\n')
  for item in my_new_dict:
    #item is a tuple consisting of string and default dict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        logging.info(newstringordict)
      else:
        for k in newstringordict.keys():
          logging.info(k)
  time.sleep(2)
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('\n\n\n')        	      
logging.info('***************************************************************************************************************************')
logging.info('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')

realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0        
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    logging.critical('\n')
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    logging.info('*******************************INFO****************************************************************************')
    logging.info('An error has been found and removed. ')
    logging.info('*******************************INFO****************************************************************************')
    logging.info('\n')
    continue
  else:
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('****************************SANITISE FINISHED **************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************SANITISE RESULTS PHASE**********************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    for k, v in my_dict.items():
      logging.info('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      for item in my_new_dict:
      #item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            logging.info(newstringordict)
          else:
            for z in newstringordict.keys():
              logging.info(z)
    logging.info('\n')          
    logging.info('************************************************************************************************************************')
    logging.info('*******************************END OF PLAN*********************************************************************************')
    logging.info('************************************************************************************************************************')
    break 
logging.info('\n')          
logging.info('***************************************************************************************************************************')
logging.info('*******************************COMPARE PHASE*******************************************************************************')
logging.info('***************************************************************************************************************************')
    
logging.info('***************************************************************************************************************************')
logging.info('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
logging.info('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      my_new_dict = my_dict[router].items()
      for tpl in my_new_dict:
        interface = tpl[0]
        cfgs = tpl[1]
        for key in cfgs.keys():
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #logging.info('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #logging.warning('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #logging.warning('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    logging.warning(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                    my_final_dict[router][interface][key] = cfgs[key]
                    #connect to router and implement interface, key
          else:
                logging.warning(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement
finalstatement = ''

for k, v in my_final_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  logging.info('\n')
  globlist = []
  for item in my_newfinal_dict:
    #item is a tuple consisting of string and default dict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        logging.info(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #logging.info(j)
  c = testbed.devices[k]
  c.connect(log_stdout=False)
  finalstatement += ' On ' 
  finalstatement += k
  finalstatement += ' i have tried to configure '
  finalstatement += str(globlist)
  finalstatement += ' \n\n '
  #here implement interactivity
  output = c.configure(globlist)
  logging.critical(output)
  #here implement check if weird return from router, e.g. invalid something
  c.destroy()

logging.info(finalstatement + '\n\n')
logging.warning('***************COMPARE IN PROGRESS****FOR A DETAILED CHANGE LOG FILE SEE CHANGE_DELTA.BAK FILES*******************')


unicondevicelist = realdevicelist
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'
    c = testbed.devices[afterchanges]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    time.sleep(2)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    #time.sleep(10)
    baseline = beforechanges
    time.sleep(3)
    comparison = afterchanges
    time.sleep(3)
    #ignore = 'ignore.txt'
    diff = diffios.Compare(baseline, comparison)
    time.sleep(1)
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'
    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    os.remove(afterchanges)
    os.remove(beforechanges)
logging.info('*******************************THIS IS THE END OF ALL OUTPUT**********************************************')


Now let’s execute this script. After the modifications i’m using a logging arg and a list of routers that I want to execute the change on. The output is much less verbose by default

tode@ubuntu:~$ python3 testtest.py critical R1%R2%R3
['R1', 'R2', 'R3']
CRITICAL:root:
CRITICAL:root:ERROR**
CRITICAL:root:[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R8
CRITICAL:root:ERROR**
CRITICAL:root:
CRITICAL:root:
CRITICAL:root:ERROR**
CRITICAL:root:[Errno 2] No such file or directory: 'R9'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R9
CRITICAL:root:ERROR**
CRITICAL:root:
CRITICAL:root:interface FastEthernet0/0
description LAN
interface FastEthernet1/0
service-policy input QOS
% policy map QOS not configured
logging buffered errors
CRITICAL:root:interface FastEthernet2/0
ip address 192.168.0.1 255.255.255.0
description LAN
interface FastEthernet0/2
interface FastEthernet0/2
^
% Invalid input detected at '^' marker.
description WAN
description WAN
^
% Invalid input detected at '^' marker.
interface FastEthernet0/0
description DMZ
interface Loopback1
description LOOP

----------------------------------------

tode@ubuntu:~$ cat R1_change_delta.bak
--- baseline
+++ comparison
1: Current configuration : 2239 bytes
2: logging buffered informational
1: Current configuration : 2249 bytes
2: interface FastEthernet0/0
description LAN
3: logging buffered errors
tode@ubuntu:~$ cat R2_change_delta.bak
--- baseline
+++ comparison
1: Current configuration : 1225 bytes
2: interface FastEthernet2/0
no ip address
1: Current configuration : 1337 bytes
2: interface FastEthernet0/0
description DMZ
3: interface FastEthernet2/0
description LAN
ip address 192.168.0.1 255.255.255.0
4: interface Loopback1
description LOOP

I will continue this in Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 5: negation commands and interactive changes

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 3: Rewriting the dictionary and getting the delta

Continuing from Part 2, where i found the bug in my ”input dictionary), which eliminated commands if they applied to the same interface. This is because a normal dictionary only allows unique keys, whereas with the defaultdict one can add a list of values to one key.
After a considerable amount of cursing, I rewrote the dictionary method:

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter



with open('change.txt', 'r') as reader
 configchangetext = reader.read()
 splittext = configchangetext.split('\n')
 my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
 for line in splittext:
        dataline = line.split(',')
        my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]

where the input file looks like this:

R1,interface fa0/1, shut,
R1,interface fa0/0, shut,
R1,interface fa0/0, shut,
R1,interface fa0/0, shut,
R1,interface fa0/0, hsrp 10,
R1,interface fa0/0, service-policy QOS,
R2,interface fa0/0, ip address 192.168.0.1 255.255.255.0,
R2,interface fa0/4, desc LAN,
R2,interface fa0/2, desc WAN,
R2,interface fa0/0, desc DMZ,

Now the question is: how to iterate through this dictionary so that we get a config block?

for k, v in my_dict.items():
 print (str(k) + '\n')

 items = my_dict[k].items()
 
 for item in items:
   #print(item)#each item is a tuple consisting of a string and a defaultdict which in turn consists of the command and an empty space...
   x = 0
   for x in range (x, len(item)):
     new_string_or_dict= item[x]
     if isinstance(new_string_or_dict, str):
       print(new_string_or_dict)
     else:
       for k in new_string_or_dict.keys():
         print(k)

So now I have both the dictionary and i know how to loop through this dictionary to generate a config block. Job almost done.

Now some small rewrite of the ”sanitize” method from part 2 blog entry, plus I insert a fake R8 config into the input text file.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)

        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do not have a config.\n This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n However, I still need to check if it makes sense to implement these changes because these may already be found in existing configs. ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************FINAL IMPLEMENTATION PLAN*******************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for k in newstringordict.keys():
              print(k)
    print('*******************************END OF PLAN***********************************************************************')
    break 
    #print('This is the list again ' + str(realdevicelist))  

And the output is quite magnificent, you can see how the changes for the non-existent R8 router are eliminated from the actual change config.

IMPLEMENTATION DRAFT PLAN
***
i am showing now how I want router config to be modified for router R1

interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
i am showing now how I want router config to be modified for router R8

interface fa0/8
desc LAN

END OF IMPLEMENTATION DRAFT PLAN***




CONFIG SANITIZATION IN PROGRESS



ERROR**
[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do not have a config.
This error appeared in R8
ERROR**
INFO* An error has been found and removed. INFO*


***SANITISE FINISHED **




SANITISE RESULTS PHASE***

After removing all errors from the implementation plan, what follows is the sanitized implementation plan.
However, I still need to check if it makes sense to implement these changes because these may already be found in existing configs.


FINAL IMPLEMENTATION PLAN


i am showing now how I want router config to be modified for each router:
R1
interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for each router:
R2
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
END OF PLAN

So now that we have the ‚sanitized’ input config block, it is time to compare this with the actual existing config and only select those commands for implementation that are not in the existing config.

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL PHASE*******************************************************************************')
print('***************************************************************************************************************************')
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key)
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change')
                  pass
                else:
                  #print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                  print(router + ' ' + interface + ' ' + key + '\n')
                #connect to i and implement config
              
              #print(answer)

And the final output of the whole script:

IMPLEMENTATION DRAFT PLAN
***
i am showing now how I want router config to be modified for router R1

interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
i am showing now how I want router config to be modified for router R8

interface fa0/8
desc LAN

END OF IMPLEMENTATION DRAFT PLAN***




CONFIG SANITIZATION IN PROGRESS



ERROR**
[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R8
ERROR**
INFO* An error has been found and removed. INFO*


***SANITISE FINISHED **




SANITISE RESULTS PHASE***

After removing all errors from the implementation plan, what follows is the sanitized implementation plan.
This is not the final implementation plan yet, because i still need to compare the plan with actual configs.


IMPLEMENTATION PLAN BEFORE COMPARE PHASE*


i am showing now how I want router config to be modified for each router:
R1
interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for each router:
R2
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN

END OF PLAN


COMPARE PHASE


FINAL CONFIG DELTA FOR IMPLEMENTATION

R1 interface fa0/0 hsrp 10
R1 interface fa0/0 service-policy QOS
R2 interface fa0/0 ip address 192.168.0.1 255.255.255.0
R2 interface fa0/0 desc DMZ

I can also create another (final) dictionary that will only gather valid, implementable items.
Here’s the change.txt file

R1,interface fa0/1, shut,
R1,interface fa0/0, desc LAN,
R1,interface fa0/0, shut,
R1,interface fa0/0, hsrp 10,
R1,interface fa0/0, service-policy QOS,
R2,interface fa0/0, ip address 192.168.0.1 255.255.255.0,
R2,interface fa0/4, desc LAN,
R2,interface fa0/2, desc WAN,
R2,interface fa0/0, desc DMZ,
R8,interface fa0/8, desc LAN,
R9,interface loop1, desc LOOP,
R2,interface loop1, desc LOOP,

here’s the R1 file

hostname R1
!
interface loop1
ip address 1.1.1.1 255.255.255.255
shut
!
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc LAN
shut
!
logging buffered

Here’s the R2 file

hostname R2
!
interface loop1
ip addr 2.2.2.2 255.255.255.255
interface fa0/0
ip addr 192.168.1.1 255.255.255.0

And here’s the (perhaps) latest form of the script.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('\n\n\n')        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for z in newstringordict.keys():
              print(z)
    print('\n')          
    print('***************************************************************************************************************************')
    print('*******************************END OF PLAN*********************************************************************************')
    print('***************************************************************************************************************************')
    
    break 

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
print('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key)
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change')
                  pass
                else:
                  #print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                  my_final_dict[router][interface][key] = ' '
                  #print(router + '\n ' + interface + '\n ' + key + '\n')
                  #connect to router and implement interface, key


for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)

This now produces the following output:

IMPLEMENTATION DRAFT PLAN
***
i am showing now how I want router config to be modified for router R1

interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
interface loop1
desc LOOP
i am showing now how I want router config to be modified for router R8

interface fa0/8
desc LAN
i am showing now how I want router config to be modified for router R9

interface loop1
desc LOOP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

END OF IMPLEMENTATION DRAFT PLAN***




CONFIG SANITIZATION IN PROGRESS



ERROR**
[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R8
ERROR**
INFO* An error has been found and removed. INFO*
ERROR**
[Errno 2] No such file or directory: 'R9'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R9
ERROR**
INFO* An error has been found and removed. INFO*


***SANITISE FINISHED **




SANITISE RESULTS PHASE***

After removing all errors from the implementation plan, what follows is the sanitized implementation plan.
This is not the final implementation plan yet, because i still need to compare the plan with actual configs.


IMPLEMENTATION PLAN BEFORE COMPARE PHASE*


i am showing now how I want router config to be modified for each router:
R1
interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for each router:
R2
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
interface loop1
desc LOOP

END OF PLAN


COMPARE PHASE


FINAL CONFIG DELTA FOR IMPLEMENTATION**
***
i am showing now how I FINALLY want router config to be modified for router R1

interface fa0/0
hsrp 10
service-policy QOS
i am showing now how I FINALLY want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface loop1
desc LOOP

The next step is to test this in GNS3: I will change the code so that the script downloads actual configs of GNS routers, then I will add the part where the script should push out the compare delta to execute the changes.

Using a nested default dictionary to create a router configuration block

Normal python dictionaries allow a single value for a given key. However, a default dictionary allows a list as values, which solves the problem of non-unique keys.
An example which is not allowed in python.

vodka: gray, vodka: blue, vodka: red

Let’s look how we can deal with this using defaultdict.

from collections import defaultdict


myconfigstring = ('interface fa0/0, ip address 192.168.0.2 255.255.255.0, shutdown, desc LAN\ninterface fa0/1, ip address 192.168.1.2 255.255.255.0, shutdown, desc WAN')
data_list = [lines.split(",") for lines in myconfigstring.split("\n")]
full_dict = defaultdict(list)

for line in data_list:
    print(line)
    interface = line[0]
    ipaddress = line[1]
    state = line[2]
    desc = line[3]
    
    interfacedetails = (ipaddress, state, desc)
    full_dict[interface].append(interfacedetails)
print(full_dict)

This produces:

defaultdict(, {'interface fa0/0': [(' ip address 192.168.0.2 255.255.255.0', ' shutdown', ' desc LAN')], 'interface fa0/1': [(' ip address 192.168.1.2 255.255.255.0', ' shutdown', ' desc WAN')]})

However, this is not a nested one yet, because we might want to have a dict like this:
{alcohols: {vodkas: {red, blue, green}, beers: {strong, weak}}}

Let’s now look at how we can use this to populate a router configuration block.

my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
my_dict['R1']['interface fa0/0']['ip address'] = '192.168.0.1 255.255.255.0'
my_dict['R1']['interface fa0/0']['desc'] = 'LAN'
my_dict['R2']['interface loop0/0']['ip address'] = '2.2.2.2 255.255.255.0'

This produces the desired result:

{"R1": {"interface fa0/0": {"ip address": "192.168.0.1 255.255.255.0", "desc": "LAN"}}, "R2": {"interface loop0/0": {"ip address": "2.2.2.2 255.255.255.0"}}}

Where do we go from here?

Now it’s time to rewrite our configuration change file and rewrite the method to create and populate the dictionary from that input file.

This is the original form of my method.

def makedictionaryfromchangefile():
     
    #looping through list of configs and parsing the config
    with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          stext = configchangetext.split('\n')
          mydict = dict(map(lambda s: s.split(','), stext))
          Dict = { }
          for k, v in mydict.items():
            v = dict(z.split(";") for z in v.split("?"))
            Dict[k] =  {}
            y = 0
            for y in range(y,len(v)):
              eachitem = list(v.items())[y]       
              Dict[k][eachitem[0]] = eachitem[1]
     
    return Dict

to be continued in Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 3, Rewriting the dictionary

Sanity checks of whisky list in my cellar with while loop with Continue keyword in Python

Let’s say that 3 years ago i looked into my cellar and made a detailed inventory of what I found there. This list is 100% reliable. This is a nested dictionary.

{ 'whisky': { '1983': 'full',
              '1981': 'full',
              '1976': 'label damaged '},
  'vodka': {'2005': 'funny smell'},
  'champagne': {'2018': 'pricey'}}

However, when I went to the cellar I realized that I had drunk the champagne a long time ago already. I prepared two simple text files with the names of what I still have. The first file is called whisky, the second file is called vodka. The files are empty.

Now I want to get rid of this item in my nested dictionary because i don’t have it anymore. The continue phrase in Except clause will help me sanitize the list and go back to the main Try clause.

import pprint
  Dict = { 'whisky': { '1983': 'full',
              '1981': 'full',
              '1976': 'label damaged '},
  'vodka': {'2005': 'funny smell'},
  'champagne': {'2018': 'pricey'}}
#i'm gathering here list of alcohol types
dictbottlelist = []
for i in Dict.keys():
  dictbottlelist.append(i)
  for j in Dict[i].keys():
    pass

while True:
  #trying to open file with a specific alcohol type
  try:
    x = 0
    for x in range(x,len(dictbottlelist)):
      with open(dictbottlelist[x], 'r') as reader:
        pass
  #if the file with alcohol type doesn't exist, delete it from inventory and continue with the loop      
  except FileNotFoundError as e:
      
      print(str(e) + '.This bottle is GONE ' + dictbottlelist[x])
      del Dict[dictbottlelist[x]]
      dictbottlelist.remove(dictbottlelist[x])
      print('An error in the list has been found and removed. ')
      continue
  else:
    print('After removing all errors from inventory, this is what you have in your cellar. ')
    pp = pprint.PrettyPrinter(indent=4)
    pp.pprint(Dict)
    break     
this is what you thought you had in your cellar:
{ 'champagne': {'2018': 'pricey'},
'vodka': {'2005': 'funny smell'},
'whisky': {'1976': 'label damaged ', '1981': 'full', '1983': 'full'}}
[Errno 2] No such file or directory: 'champagne'.This bottle is GONE champagne
An error in the list has been found and removed.
After removing all errors from inventory, this is what you really have in your cellar.
{ 'vodka': {'2005': 'funny smell'},
'whisky': {'1976': 'label damaged ', '1981': 'full', '1983': 'full'}}

I will use this to sanitize my config change file in the other blog post…

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 2

This is currently the output.
First, i’m parsing the file with the changes.

Then i’m checking if the file with changes contains routers that are not in my infrastructure. If it does, i remove the changes for these routers from the modification plan (‚an error has been found and removed’)

Finally, i compare the remaining changes with existing configs of routers. If a match is found, the change doesn’t make sense.

def makedictionaryfromchangefile():
    
    #looping through list of configs and parsing the config
    with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          stext = configchangetext.split('\n')
          mydict = dict(map(lambda s: s.split(','), stext))
          Dict = { }
          for k, v in mydict.items():
            v = dict(z.split(";") for z in v.split("?"))
            Dict[k] =  {}
            y = 0
            for y in range(y,len(v)):
              eachitem = list(v.items())[y]       
              Dict[k][eachitem[0]] = eachitem[1]
    
    return Dict
def comparechangewithconfig():
    
    Dict = makedictionaryfromchangefile()
    #i need to get the list of devices for which change should made. This may be different than the list of routers that are in the environment(typos etc.)
    
    
    for i in Dict.keys():
        #print(' I am now inside the change dictionary, and i am in the main key ' + str(i) + str(Dict[i]))
        #this will just print parents 
        #for j in Dict[i].keys():
            #print(' i am printing what will be changed for ' + str(i) + ' and the changes are as follows ' + str(j))
        for l, m in Dict[i].items():
        	print('i am showing now how I want router ' + str(i) + ' config to be modified: ' + l + '  ' + m)
    realdevicelist = []
    for i in Dict.keys():
      #i add the devices to the list from dict main keys
      realdevicelist.append(i)
      for j in Dict[i].keys():
      	#i don't do anything here
        pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
    while True:
      try:
        x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
        for x in range(x,len(realdevicelist)):
          with open(realdevicelist[x], 'r') as reader:
            
            config = reader.read()
            my_config_as_list = config.split("\n")
      except FileNotFoundError as e:
          print('*******************************ERROR**************************************************************************')
          print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do not have a config. This error appeared in ' + realdevicelist[x])
          print('*******************************ERROR**************************************************************************')
          
          del Dict[realdevicelist[x]]
          realdevicelist.remove(realdevicelist[x])
          print('An error has been found and removed. ')
          
          
          continue
      else:
        print('After removing all errors from config file, this will be implemented. ')
        pp = pprint.PrettyPrinter(indent=4)
        pp.pprint(Dict)
        break 
    #print('This is the list again ' + str(realdevicelist))   
    for i in Dict.keys():
    	#print('I am in ' + str(i))
    	with open(i, 'r') as reader:
        #i'm on R1 now   
            config = reader.read()
            my_config_as_list = config.split("\n")
            for j, k in Dict[i].items():
              #print(j + ' and its value ' + k)
              parse = CiscoConfParse(my_config_as_list)
              answer = parse.find_objects('^' + j)
              #print(answer)
              if len(answer) > 0:
                #print('a subkey from Dict matches a parent object from the config. This does not tell us anything yet. Maybe the value will be modified, maybe not') 
                pass
              second_answer = parse.find_objects_w_child(j, k)
              if len(second_answer) > 0:
                print('i am on ' + i + ' in subkey ' + j + ' in value ' + k + ' and I have found a match in the existing configuration of ' + i + ' so i am not going to implement this change')
              else:
              	print(' no match  in config of ' + i + ' found when on ' + i + ' in subkey ' + j + ' in value ' + k + ' so i will implement this change ')
                #connect to i and implement config

Unfortunately, I’ve found a bug that overrides the change value for a given interface if this interface appears twice in the change config. I need to be able to create a dictionary with change values as a list, not just with single values. This needs further research.

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 1 – getting inputs

Hello

Unfortunately, making changes in Unicon is not idempotent. Unicon doesn’t realize natively if a change in the config is needed or not, so it will try to add config irrespectively of whether the config you want to add already exists on the target device or not. Having this feature can also help avoid mistakes such as overriding valid config by mistake.

I was thinking along these lines: I should take the existing running config and the implementation change config set, and analyze this change to see if a particular config change will actually modify the config or not. I believe that in order to do this, the running config needs to be analyzed in terms of keys and values, e.g: interface name is a key, ip address is a value.

int loop1 KEY
ip addr 1.1.1.1 255.255.255.0 VALUE

So the first condition is that if i want to implement a change on fa0/0, i should look for the key interface fa0/0, if the key doesn’t exist, it’s a real change and it should be implemented.

The second condition is that if i want to implement a change on a key that already exists, then I need to compare the values: if the new value is different, it’s a real change again and it should be implemented.
Fortunately, a library exists that already understands this key-value relationship: CiscoConfParse

Additionally, I’ve been thinking that to make any progress with my Python skills, i should try to use some cool features i didn’t know before. This time it is the lambda feature; it is basically an unnamed def. Here it helps me split the string into two parts, and the map function takes the first resulting string and maps it to the key, the second string becomes the value.

from ciscoconfparse import CiscoConfParse

#this is my config
parse = CiscoConfParse([ '!', 'interface FastEthernet0/0', ' ip address 192.168.0.1 255.255.255.0', '!', 'interface FastEthernet0/1', ' ip address 172.16.0.1 255.255.255.0'])

#Looking for condition 1: does a key(parent) exist? i'm parsing all objects to find words beginning with 'interf'

all_intfs = parse.find_objects(r"^interf")

#Looking for condition 2: if a key (parent) exists, does the same value (child) exists for this parent?

answer = parse.find_objects_w_child(parentspec=r"^interface FastEthernet0/0", childspec=r"192.168.0.1")
#this returns a list, so i need to see if the length of this list is >0

#the logic should be like this: if change has a key and value, look for the same key in existing config, if len>0, look for value for this key, if len>0, change not needed.
#let's say we have an upcoming parent-child config change, already parsed to the following string

mychangeinconfig = 'int fa0/0:ip address 192.168.0.1 255.255.255.0'
mylist = []
mylist.append(mychangeinconfig)

#we can try to make this change into a key: value dictionary

def list_to_dict(somelist):
return dict(map(lambda s: s.split(':'), somelist))
mydict = list_to_dict(mylist)
for k, v in mydict.items():
  myparent = k
  mychild = v

#or we can present this change config in the form of two separate strings, this doesn't matter too much.

mychangeparent = 'interface FastEthernet0/0'
mychangechild = 'ip address 192.168.0.1 255.255.255.0'
#now the key is int fa0/0 and the value is ip address 192.168.0.1 255.255.255.0
#we could now put this in the search algorithm
#answer = parse.find_objects_w_child(parentspec=myparent, childspec=mychild)

#LOGIC: if answer found in the form of list, the change is not needed.
if len(answer)>0:
  #change not needed
else:
  #implement change

#DONE Challenge1: to prepare a text file with following config changes:
R1
int fa0/0
 ip addr 172.16.0.1 255.255.255.0
int fa0/1
 ip addr 169.254.0.1 255.255.255.0
R2
logging buffered informational
R3
int vlan 100
 shut
@ and parse it such that the final form is the following:
{R1: {int fa0/0:ip addr 172.16.0.1 255.255.255.0, int fa0/1:ip addr 169.254.0.1 255.255.255.0}, R2: {logging buffered informational}, R3: {int vlan 100: shut}}


#Challenge2 TODO: to find out if the following changes should be implemented or if they already exist
change = {R1: {int fa0/0:ip addr 172.16.0.1 255.255.255.0}, R2: {logging buffered notifications}, R3: {int vlan 100: shut}} 

This should be fun with my low-end programming skills 😀

Challenge 1 – preparing the input with the new change

The input with the change needs to be prepared in a way that will be possible to parse.

#i cheated in preparing the change config set, but only a little. The thing is, i need to have clearly separated input to be able to parse it. Here i use commas to separate router names from keys, semicolons to separate keys from values and question marks to separate keys:values from new keys:values. Finally /n begins another router change config. 

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0\nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'

print('this is raw text input for my change' + text)

def text_2_dic(atext):
  stext = atext.split('\n')
  mydict = dict(map(lambda s: s.split(','), stext))
  Dict = { }
  for k, v in mydict.items():
    v = dict(x.split(";") for x in v.split("?"))
    for a,b in v.items():
      Dict[k] = {}
      Dict[k][a] = b
  return Dict
print('and this is final dictionary with the parsed change input: \n ' + str(text_2_dic(text))
C:\Users\A666773\Desktop>python listtest.py
this is raw text input for my change: R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0
R2,int fa0/0;ip address 192.168.0.2 255.255.255.0
R3,int fa0/0;ip address 192.168.0.3 255.255.255.0
and this is final dictionary with the parsed change input:
{'R1': {'int fa0/1': 'ip address 192.168.1.1 255.255.255.0'}, 'R2': {'int fa0/0': 'ip address 192.168.0.2 255.255.255.0'}, 'R3': {'int fa0/0': 'ip address 192.168.0.3 255.255.255.0'}}

This was actually wrong, because the final dictionary didn’t have the int fa0/0 change. The problem was in initialising the dictionary inside the loop rather than outside:

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0\nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'
  def text_2_dic(atext):
    stext = atext.split('\n')
    mydict = dict(map(lambda s: s.split(','), stext))

  print('+++++++++++++++++++++++++++++++++++++++++++\n\n')
  print('this is the predictionary \n' + str(mydict))
  print('+++++++++++++++++++++++++++++++++++++++++++\n\n')
  Dict = { }
  for k, v in mydict.items():
    v = dict(x.split(";") for x in v.split("?"))
    Dict[k] = {}
    for a,b in v.items():
      Dict[k][a] = b 
  return Dict
print('and this is final dictionary: \n ' + str(text_2_dic(text)))

OUTPUT:

Another problem appeared: what about configs which are one-line only? Not all configs have a parent and a child, like logging buffered in this case below. For now, i’ll use a trick with a semicolon and a empty space value. We’ll see what happens later. Maybe this will have to be reworked.

INPUT

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/1;ip address 192.168.1.1 255.255.255.0?logging buffered; \nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'

OUTPUT

and this is final dictionary:
{'R1': {'int fa0/0': 'ip address 192.168.0.1 255.255.255.0', 'int fa0/1': 'ip address 192.168.1.1 255.255.255.0', 'logging buffered': ' '}, 'R2': {'int fa0/0': 'ip address 192.168.0.2 255.255.255.0'}, 'R3': {'int fa0/0': 'ip address 192.168.0.3 255.255.255.0'}}

This was still wrong though, because if i inserted into the input multiple changes for a single subkey (e.g. interface), these changes were not found in the final dictionary.

I need to find if dictionary v contains multiple items. If it does, i loop through these items creating tuplets (eachtuplet), and then i update the dictionary from those tuplets.

def text_2_dic(atext): 
  stext = atext.split('\n') 
  mydict = dict(map(lambda s: s.split(','), stext)) 
  print('+++++++++++++++++++++++++++++++++++++++++++\n\n') 
  print('this is the predictionary \n' + str(mydict)) 
  print('+++++++++++++++++++++++++++++++++++++++++++\n\n') 
  Dict = { } 
  for k, v in mydict.items(): 
    v = dict(x.split(";") for x in v.split("?")) 
    Dict[k] = {} 
    x = 0 
    for x in range(x,len(v)): 
      eachtuplet = list(v.items())[x] 
      Dict[k][eachtuplet[0]] = eachtuplet[1] 
return Dict

INPUT:

text = 'R1,int fa0/0;ip address 192.168.0.1 255.255.255.0?int fa0/0;desc LAN?int fa0/0;shut?int fa0/1;ip address 192.168.1.1 255.255.255.0?logging buffered; \nR2,int fa0/0;ip address 192.168.0.2 255.255.255.0\nR3,int fa0/0;ip address 192.168.0.3 255.255.255.0'

OUTPUT

And this is the final dictionary:
{'R1': {'int fa0/0': 'shut', 'int fa0/1': 'ip address 192.168.1.1 255.255.255.0', 'logging buffered': ' '}, 'R2': {'int fa0/0': 'ip address 192.168.0.2 255.255.255.0'}, 'R3': {'int fa0/0': 'ip address 192.168.0.3 255.255.255.0'}}

Adding pretty print…

import pprint
(...)
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(text_2_dic(text))

Obviously there are more scenarios to be accounted for (deleting with ‚no’ or nested commands like qos policies) but for now that’s enough. I can improve that later.

Challenge 2 – loading and parsing the configs

Now that we have this nice dictionary with the parsed config changes, we need to load the existing configs.

For now, i will be using mock (abbreviated device configs) and a mock (custom) change text file.

This is the beginning of the larger def that will later compare the inputs and return True or False.

def comparechangewithconfigof(routerfilelist):
  realdevicelist = routerfilelist.split('%')
  x = 0
#looping through list of configs and parsing the config
  for x in range(x,len(realdevicelist)):
    with open(realdevicelist[x], 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parsedconfig = CiscoConfParse(my_config_as_list)
#opening the change file and transforming into Dict
    with open('change.txt', 'r') as reader:
      configchangetext = reader.read()
      stext = configchangetext.split('\n')
      mydict = dict(map(lambda s: s.split(','), stext))
      Dict = { }
      for k, v in mydict.items():
        v = dict(z.split(";") for z in v.split("?"))
        Dict[k] = {}
        y = 0
        for y in range(y,len(v)):
          eachitem = list(v.items())[y]
          Dict[k][eachitem[0]] = eachitem[1]
print(Dict)


comparechangewithconfigof('R1%R2%R3')

Inputs are now ready. Now it’s time to compare the change input with the existing configs and ask if the change makes sense (worth implementing?). To be continued next week in part 2.

Building a pull-based config change notification system with Python and Slack, part 2 (Diffios)

In this part I would like to get a full report on Slack regarding what changed and who changed it.
Diffios will help me with the first part, while using the log buffer should help me with the who bit.

A short introduction to diffios:

tode@ubuntu:~$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import diffios
baseline = "oldconfig.bak"
comparison = "modifiedconfig.bak"
ignore = "ignore.txt"
diff = diffios.Compare(baseline, comparison, ignore)
print(diff.delta())
--- baseline
+++ comparison
+ logging buffered informational

This is pretty self-explanatory, but as you can see these two config files are only different in that i’ve added the logging buffered informational in between the writes and Diffio shows the delta. The ignore.txt files includes only 2 words: „Current configuration” because it doesn’t make sense to put size of config (which appears in the running config file) as delta.
Let’s update our report script now and run part2.py again:

Ok now we know what’s been changed, now on to who’s changed it. I can use the command show archive log config all

After the script is updated again, here’s the Slack output.

to be continued in Building a pull-based config change notification system with Python and Slack part 3 : Interacting with devices in Slack