Genie parsers

Hello

One of many cool things about Genie is that it provides parsers, which take command output from Cisco and translate it into a dictionary that has easily accessible (programatically) data.
To do that, Genie uses really complex regex expressions to scrape data from the output and then puts the values it gets into a dictionary schema.Let’s have a look at an example of a Genie parser output that processes data from ‚show version’

software version is: 15.4
router has been up for: 4 minutes

I’ve achieved this with just a few lines:

shver = r1.parse("show version")
print('software version is: ' + shver['version']['version_short'])
print('router has been up for: ' + shver['version']['uptime'])

Another parser gets data from show ip eigrp neighbors

139.21.173.1 neighbor has been up for: 00:02:22

and the code is:

print(list(sheigrpnei['eigrp_instance']['1251']['vrf']['1251']['address_family']['ipv4']['eigrp_interface']['Tunnel1251']['eigrp_nbr'].keys())[0] + ' neighbor has been up for: ' + sheigrpnei['eigrp_instance']['1251']['vrf']['1251']['address_family']['ipv4']['eigrp_interface']['Tunnel1251']['eigrp_nbr']['139.21.173.1']['uptime'])

This may look a bit more complex but the first part of print gets a list (first element) of keys from the dictionary, and the second part of print gets the value of uptime from the dictionary. It’s just that in case of output of the command show ip eigrp vrf <vrf> neighbor, the schema prepared by the author is a bit longer because there are more values.

Unfortunately, not all parsers that I need are available out of the (Genie) box. Why? Because there is no box. This is not a commercial product so the project is community-driven. There are regular updates of the Genie package and more parsers are added but if you need a certain parser immediately, you have to write it yourself. Genie documentation is pretty comprehensive but I have found that the best way of doing something new is to copy existing working stuff and creatively mold it to obtain a new result.

For my daily work i need to be able to get data about crypto tunnels and there is no parser at this point for show crypto isakmp sa. Let’s create it then by looking at the output:

R1#show crypto isakmp sa
 IPv4 Crypto ISAKMP SA
 dst             src             state          conn-id status
 139.23.204.37   80.0.0.1        QM_IDLE           1001 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE (deleted)

This is the ”design phase” where you think about the end result to create a dictionary schema. What do you need in that final dictionary so that it meets your requirements? Because this was the first parser i’d ever created, I decided that I only needed two values: the destination and the state.

# Python
import re

# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Or, Optional

# parser utils
from genie.libs.parser.utils.common import Common

__all__ = ['ShowCryptoIsakmpSASchema']
class ShowCryptoIsakmpSASchema(MetaParser):
    '''
    Schema for:
    <show crypto isakmp sa>
    '''

    #* my first schema is very simple: we might rework it later.
    schema = {
        'dst': {
            Any(): {
                Optional('STATE'): str,
            }
        }
    }

I envisioned (in my mind’s eye 😀 ) my dictionary to look like this:

{'dst': {'139.23.204.37': {'STATE': 'QM_IDLE'}}}

Now that the schema is ready, i can proceed with creating the class that processes the output.

class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
    '''
    Parser for:
    <show crypto isakmp sa>
    '''

    cli_command = 'show crypto isakmp sa' 

    #* could add extra parameters if needed
    def cli(self,vrf = '', output = None):
        #if output is None:
           #out = self.device.execute(self.cli_command)
        #else:
           #out = self.cli_command
    #for testing i will prepare mock input
        out = '139.23.204.37   80.0.10.1       QM_IDLE           1001 ACTIVE'
        # initial variables
parsed_dict = {}

Now it’s time to change the brain gear because it is necessary to prepare a regex expression that will match the command output (mock input from the code above).

One hour later (and some extensive learning at testing at regex101.com)

p1 = re.compile(r'^(?P([0-9]{1,3}.){3}[0-9]{1,3})\s+(?P([0-9]{1,3}.){3}[0-9]{1,3})\s+(?P[\w\d-.]+)\s+(?P[0-9]+)\s+(?P[\w\d-.]+)$')

Now that your brain is at the right temperature, the rest is laughably easy. Let’s check if our regex matches any lines from the output (well, the only one we have):

for line in out.splitlines():
            line = line.strip()
        m = p1.match(line)


Then the logic is as follows, if the regex has found matches in the lines, let’s put the values from m into a special group dictionary. Finally, let’s assign values (or subdictionaries) from that groupdictionary into our final dictionary:

if m:
                groups = m.groupdict()
                #print('this is groups: ' + str(groups))

                dst = groups['dst']
                dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})

                state = groups['STATE']
                dst_dict.update({'STATE': state})
               
        return parsed_dict

Now the parser code is ready! Here’s the full code:

ode@ubuntu:~/showcryptoisakmp/tode/parser/ios$ cat showcryptoisakmp.py 
#******************************************************************************
#*                              Parser Template
#* ----------------------------------------------------------------------------
#* ABOUT THIS TEMPLATE - Please read
#*
#* - Any comments with "#*" in front of them (like this entire comment box) are
#*   for template clarifications only and should be removed from the final
#*   product.
#*
#* - Anything enclosed in <> must be replaced by the appropriate text for your
#*   application
#*
#* Support:
#*    pyats-support@cisco.com
#*
#* Description:
#*   This template file describes how to write a standard parser.
#*
#* Read More:
#*   For the complete and up-to-date user guide on parsers, visit:
#*   https://pubhub.devnetcloud.com/media/pyats-packages/docs/genie/cookbooks/index.html#genie-libs-parsers
#*   https://wiki.cisco.com/display/GENIE/Genie+Parsers+Deepdive
#*
#*   List of available Genie parsers:
#*   http://wwwin-pyats.cisco.com/cisco-shared/genie/latest/genie_libs/#/parsers
#*
#*******************************************************************************

#*******************************************************************************
#* DOCSTRINGS
#*
#*   All parsers should use the built-in Python docstrings functionality
#*   to define parser/class/method headers.
#*
#* Format:
#*   Docstring format should follow:
#*   URL= https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html
#*
#* Read More:
#*   Python Docstrings, PEP 257:
#*   URL= http://legacy.python.org/dev/peps/pep-0257/
#*******************************************************************************

'''
showcryptoisakmp.py

ios parsers for the following show commands:
    * <your commands here>

'''

# Python
import re

# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Or, Optional

# parser utils
from genie.libs.parser.utils.common import Common

# ==============================
# Schema for:
#    * '<show crypto isakmp sa>'
# ==============================

__all__ = ['ShowCryptoIsakmpSASchema']
class ShowCryptoIsakmpSASchema(MetaParser):
    '''
    Schema for:
    <show crypto isakmp sa>
    '''

    #* define your schema here
    schema = {
        'dst': {
            Any(): {
                Optional('STATE'): str,
            }
        }
    }

# ==============================
# Parser for:
#    * '<show crypto isakmp sa>'
# ==============================
class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
    '''
    Parser for:
    <show crypto isakmp sa>
    '''

    cli_command = 'show crypto isakmp sa' 

    #* could add extra parameters if needed
    def cli(self,vrf = '', output = None):
        #if output is None:
           #out = self.device.execute(self.cli_command)
        #else:
           #out = self.cli_command
        out = '139.23.204.37   80.0.10.1       QM_IDLE           1001 ACTIVE'
        # initial variables
        parsed_dict = {}

        p1 = re.compile(r'^(?P<dst>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<src>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<STATE>[\w\d\-\.]+)\s+(?P<CONNID>[0-9]+)\s+(?P<status>[\w\d\-\.]+)$')

        for line in out.splitlines():
            line = line.strip()
        m = p1.match(line)
        if m:
                groups = m.groupdict()
                #print('this is groups: ' + str(groups))

                dst = groups['dst']
                dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})

                state = groups['STATE']
                dst_dict.update({'STATE': state})
        #print(parsed_dict)        
        return parsed_dict    

However, our job is not fully done yet. Now we need to stop using mocked input and use real command output from the router.

class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
    '''
    Parser for:
    <show crypto isakmp sa>
    '''

    cli_command = 'show crypto isakmp sa' 

    #* could add extra parameters if needed
    def cli(self,vrf = '', output = None):
        if output is None:
           out = self.device.execute(self.cli_command)
        else:
           out = self.cli_command

        print('this is output i got from dictionary: ' + str(out))
        #out = '139.23.204.37   80.0.10.1       QM_IDLE           1001 ACTIVE'
        # initial variables
        parsed_dict = {}

        p1 = re.compile(r'^(?P<dst>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<src>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<STATE>[\w\d\-\.]+)\s+(?P<CONNID>[0-9]+)\s+(?P<status>[\w\d\-\.]+)$')

        for line in out.splitlines():
            line = line.strip()
            print('this is next line ' + line)
            m = p1.match(line)
            if m:
                groups = m.groupdict()
                print('this is groups: ' + str(groups))

                dst = groups['dst']
                dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})

                state = groups['STATE']
                dst_dict.update({'STATE': state})
        print(parsed_dict)        
        return parsed_dict    

Now the parser returns a dictionary, so the question is how to get specific values from that dictionary? let’s create a short script for that.

import json
from genie.conf import Genie
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
from genie.conf import Genie
from genie.conf.base import Device
from genie.conf.base import Testbed
from genie.conf.base.base import DeviceFeature
from genie.libs.conf.vrf import Vrf
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Optional
import tode

testbed = Genie.init('testbed.yml')
r1 = testbed.devices['R1']
r1.connect(log_stdout=False)

shcrypto = r1.parse("show crypto isakmp sa")
ipstring = str(list(shcrypto['dst'].keys())[0])
state = str(shcrypto['dst'][ipstring]['STATE'])
print('tunnel destination ' + ipstring + ' is in state ' + state )

ipstring = str(list(shcrypto['dst'].keys())[1])
state = str(shcrypto['dst'][ipstring]['STATE'])
print('tunnel destination ' + ipstring + ' is in state ' + state )

On my spoke routers i will never have more than two tunnels, so this should be enough, but of course it would be better to have a for loop to go through the whole length of the list rather than getting specific list positions. For now this will be enough, though.

Now let’s test this. For now the original parser includes a lot of ”debug” code that we don’t need but it’s included here for the sake of clarity:

this is output i got from dictionary: IPv4 Crypto ISAKMP SA
 dst             src             state          conn-id status
 139.23.204.37   80.0.0.1        QM_IDLE           1001 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE (deleted)
 IPv6 Crypto ISAKMP SA
 this is next line IPv4 Crypto ISAKMP SA
 this is next line dst             src             state          conn-id status
 this is next line 139.23.204.37   80.0.0.1        QM_IDLE           1001 ACTIVE
 this is groups: {'dst': '139.23.204.37', 'src': '80.0.0.1', 'STATE': 'QM_IDLE', 'CONNID': '1001', 'status': 'ACTIVE'}
 this is next line 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE
 this is groups: {'dst': '139.23.204.38', 'src': '80.0.0.1', 'STATE': 'MM_NO_STATE', 'CONNID': '0', 'status': 'ACTIVE'}
 this is next line 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE (deleted)
 this is next line 
 this is next line IPv6 Crypto ISAKMP SA
 {'dst': {'139.23.204.37': {'STATE': 'QM_IDLE'}, '139.23.204.38': {'STATE': 'MM_NO_STATE'}}}
 tunnel destination 139.23.204.37 is in state QM_IDLE
 tunnel destination 139.23.204.38 is in state MM_NO_STATE

What is really interesting is the last two lines. I have been successful in transforming string output to a dictionary and getting specific values from the dictionary.

But to actually test the script, i had to do some more magic to ”register” the new parser in the genie libs.
to be continued…

Using Genielibs to create config

Hello

After reading the documentation, I realized that it is not necessary to create own methods (or j2 templates if you prefer the other way) to create the config. Instead, it is possible to use community-created classes, like vrf objects below.

For the first method I’m using a yaml config file with parameters, for the second method i’m injecting the parameters in the method ‚live’.

from genie.conf import Genie
from genie.conf.base import Device
from genie.conf.base import Testbed
from genie.conf.base.base import DeviceFeature
from genie.libs.conf.vrf import Vrf
from genie.libs.conf.static_routing.static_routing import StaticRouting
from pyats import topology
from genie.testbed import load
from genie.libs.conf.address_family import AddressFamily
from genie.libs.conf.bgp import RouteTarget



def Change17nov2020():
 
 with open('cisco_genie_cfg_loader_config_file.yml', 'rb') as f:
    conf = yaml.load(f.read())    
 
 r1 = load('testbed.yml').devices['R1']
 
 vrf1 = Vrf(name=conf['name'], description=conf['description'], rd=conf['rd'])
 
 r1.connect()
 r1.configure(str(vrf1.build_config(devices=[r1], apply=False)['R1']).split('\n'))

def Change18nov2020():
 r1 = load('testbed.yml').devices['R1']
 static_routing = StaticRouting()
 static_routing.device_attr[r1].vrf_attr['VRF1'].address_family_attr['ipv4'].route_attr['10.2.1.0/24'].next_hop_attr['192.168.1.2'].preference = 3

 
 r1.connect()
 r1.configure(str(static_routing.build_config(devices=[r1],apply=False['R1']).split('\n'))

Change17nov2020()
Change18nov2020()

Reverting the change is very easy, with the ‚unbuild_config’ method instead. Yes, reverting your change is as simple as adding two letters to the code.

Note that sometimes your out-of-the-box solution is not fully baked. You may occasionally need to ‚improve’ the genie libs a bit. In the example above, the vrf.py under ios-xe genie libraries didn’t have the address-family ipv4 command, which didn’t activate the routing table for the vrf. I ended up editing manually (under site packages) the vrf.py file to add the following line:

configurations.append_line(‚address-family ipv4 unicast’)

Alternatively (i think this is clearer), you can do this:

 def Change17nov2020():

   with open('cisco_genie_cfg_loader_config_file.yml', 'rb') as f:
     conf = yaml.load(f.read())    # load the config file
   testbed = load('testbed.yml')
   r1 = testbed.devices['R1']
   r1.connect()
   vrf1 = Vrf(name=conf['name'], description=conf['description'], rd=conf['rd'])
   r1.add_feature(vrf1)
   output = vrf1.build_config()

Putting it all together: OOB network programming, nested jinjas, decorators, slack, Genie

After a few months of configuring my test network using one-off scripts i have now realized that perhaps this was not entirely proper way of doing things.
In one of my previous blog posts i mentioned that configuring the router using that long change.txt wasn’t ”network as code” but rather network as a long messy text file. This was true, although it was a small step forward.
From now on I will try to combine all the things i’ve done before, that is Unicon, jinja templates but using the OOB approach and modifying the network using code, so that I have the object-oriented app logic in one file, change execution in another (importing the logic), jinja templates in another file (.j2) and jinja attributes in another file (.yml).


The endgoal will be to have the following workflow:
1. prepare the new config,

2. get current network configuration and state,

3. implement the config,

4. get new network configuration and state

5. get change notification with config delta in Slack.

6. Optional (start the workflow from Slack, the ”chatops” way)

First let’s create some basic logic.

class Router:

   def __init__(self, sship, sshuser, sshpass):
    self.config = []
    self.sship = sship
    self.sshuser = sshuser
    self.sshpass = sshpass

   def setIPaddress(self, intname, IP, mask):
    self.intname = intname
    self.ip = IP
    self.mask = mask
    commandlist = []
    commandlist.append(self.intname)
    commandlist.append('ip address ' + str(self.ip) + ' ' + str(self.mask))
    self.config.append(commandlist)
   
  

    def configure(self):
    	#todo

router = Router()
router.setIPaddress("int fa0/0","192.168.0.1","255.255.255.0")
print(router.config)

Now we have the OOB basic, I will take advantage of all the cool bits and pieces from the previous blog posts to create this full script including Unicon, jinja etc.

from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging
from gtts import gTTS
import re

class Router:

   def __init__(self):
    self.config = '' 

   def setIPaddress(self, intname, IP, mask):
    self.intname = intname
    self.ip = IP
    self.mask = mask
    commandlist = []
    commandlist.append("interface " + str(self.intname))
    commandlist.append('ip address ' + str(self.ip) + ' ' + str(self.mask))
    self.config += commandlist

   def prepareACLconfig(self, datafile, template):
       self.datafile = datafile
       self.template = template
       accesslists = yaml.load(open(self.datafile), Loader=yaml.SafeLoader)
       env = Environment(loader = FileSystemLoader('.'), trim_blocks=True, autoescape=True)
       template = env.get_template(self.template)
       acl_config = template.render(data=accesslists)
       self.config += acl_config

   def configure(self):
    testbed = loader.load('testbed.yml')
    c = testbed.devices['R1']
    c.connect()
    output = c.configure(router.config)
    c.destroy()
    #todo
#and now let's test an actual config change with acl data in the .yml object and template in the .j2 object, with the GNS3 routers described in the testbed.yml object.
      
router = Router()
router.prepareACLconfig('vpnacl.yml', 'acl.j2')
router.configure()


while all the data files look like this:

tode@ubuntu:~/perfectscripts/generator$ cat vpnacl.yml
 accesslists:
 aclname: permit_www
  aclcommand: permit
  protocol: ip
  source: 192.168.122.100
  sourcewildcard: 0.0.0.0
  destination: 8.8.8.8
  destwildcard: 0.0.0.0  


 tode@ubuntu:~/perfectscripts/generator$ cat acl.j2
 {% for accesslist in data.accesslists %}
 ip access-list extended {{accesslist.aclname}}
 {{accesslist.aclcommand}} {{accesslist.protocol}} {{accesslist.source}} {{accesslist.sourcewildcard}} {{accesslist.destination}} {{accesslist.destwildcard}}
 {% endfor %}


 tode@ubuntu:~/perfectscripts/generator$ cat testbed.yml
 devices:
   R1:
     connections:
       cli:
         ip: 192.168.122.100
         port: 23 
         protocol: telnet
     credentials:
       default:
         password: cisco
         username: cisco
       enable:
         password: cisco
     os: ios
     type: ios

Decorators

Let’s add a simple function here that will do ‚show run’ before and after the change. To do that, i’ve implemented a simple decorator.

def diffdecorator(func):
    def wrapper(self):
        testbed = loader.load('testbed.yml')
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.bak', "w")
        f.write(output)
        f.close()
        c.destroy()
        func()
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.txt', "w")
        f.write(output)
        f.close()
        c.destroy()
        diff = diffios.Compare('R1.bak', 'R1.txt')
        deltaoutput = diff.delta()
        filename = 'R1_change_delta.bak'
        f = open(filename, "w")
        f.write(deltaoutput)
        f.close()
    return wrapper

   @diffdecorator
   def configure():
    testbed = loader.load('testbed.yml')
    c = testbed.devices['R1']
    c.connect()
    output = c.configure(router.config)
    c.destroy()
    

Nested Jinja template

Going further, i created an ospf.j2 file and a corresponding ospf.yml file. Now because the ospf can have multiple network statements, a nested jinja template was needed.

tode@ubuntu:~/perfectscripts/generator$ cat ospf.yml
ospf:
   - processnumber: 100
     routerid: 1.1.1.1
     network: 
       - address: 192.168.122.0 0.0.0.255 area 0
       - address: 1.1.1.1 0.0.0.0 area 0
tode@ubuntu:~/perfectscripts/generator$ cat ospf.j2
{% for element in data.ospf%}
  router ospf {{element.processnumber}}
  router-id {{element.routerid}}
  {% for entry in element.network%}
    network {{entry.address}}
  {% endfor %}
{% endfor %}


This has produced the following config:

tode@ubuntu:~/perfectscripts/generator$ python3 ng_oob.py 
   router ospf 100
   router-id 1.1.1.1
       network 192.168.122.0 0.0.0.255 area 0
       network 1.1.1.1 0.0.0.0 area 0

Next steps:

  • using pyats to check network state before change (another decorator)
  • decision about folder structure with .ymls, templates and script change history
  • how to implement idempotency (file-based, or code-based)
  • using Slack to trigger changes (reverse bot)

Adding Slack notification

Let’s rewrite the decorator again:

def diffdecorator(func):
    def wrapper(self):
        testbed = loader.load('testbed.yml')
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.bak', "w")
        f.write(output)
        f.close()
        c.destroy()
        func()
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.txt', "w")
        f.write(output)
        f.close()
        c.destroy()
        diff = diffios.Compare('R1.bak', 'R1.txt')

        deltaoutput = diff.delta()
        p = get_notifier('slack')
        p.notify(webhook_url='https://hooks.slack.com/services/xxxxx/yyyyy/zzzzzz', message='config on this router has been changed!')
        p.notify(webhook_url='https://hooks.slack.com/services/xxxxx/yyyyy/zzzzzz', message=diff.delta())
        filename = 'R1_change_delta.bak'
        f = open(filename, "w")
        f.write(deltaoutput)
        f.close()
    return wrapper


Adding Genie magic

A while ago I was tasked with creating a report based on the uptime taken from output of ‚show version’ command plus software version. Preparing the regex was funny, and i was well proud of the way i did the ”screenscrubbing”, until I saw how you can do this easier.

The regex way for uptime doesn’t look that bad but what if Cisco changed the output of the show version command?

endresult = re.findall(r'control processor .*', result)

And here comes Genie. Initialising the testbed looks very similar to the unicon way ( Genie.init instead of loader.load )

     testbed = Genie.init('testbed.yml')
     r1 = testbed.devices['R1']
     r1.connect()
     shver = r1.parse("show version")
     
     print(shver['version']['version_short'])
     print(shver['version']['uptime'])

The output of this is:

12.4
 5 hours, 8 minutes

Now another thing we can do with Genie is compare the state of the network. This is different than a configuration diff. A Genie diff can for example show you which OSPF LSAs you received in between running two ”Genie learn commands”. Let me show you how it’s done:

genie learn ospf --testbed testbed.yml --device R1 --output learn
earning '['ospf']' on devices '['R1']'
 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.04s/it]
 +==============================================================================+
 | Genie Learn Summary for device R1                                            |
 +==============================================================================+
 |  Connected to R1                                                             |
 -   Log: learn/connection_R1.txt 
 Learnt feature 'ospf' 
 -  Ops structure:  learn/ospf_ios_R1_ops.txt 
 -  Device Console: learn/ospf_ios_R1_console.txt 
 ============================================================================== 

Then I run my python script which generates and implements the ospf config:

tode@ubuntu:~/perfectscripts/generator$ python3 ng_oob.py 
   router ospf 100
   router-id 1.1.1.1
       network 192.168.122.0 0.0.0.255 area 0
       network 1.1.1.1 0.0.0.0 area 0

(...) here i didn't include the unicon's screen output

Now i run the Genie learn again, this time i redirect the output to another folder:

tode@ubuntu:~/perfectscripts/generator$ genie diff learn learn2
 1it [00:00, 127.13it/s]
 +==============================================================================+
 | Genie Diff Summary between directories learn/ and learn2/                    |
 +==============================================================================+
 |  File: ospf_ios_R1_ops.txt                                                   |
 - Diff can be found at ./diff_ospf_ios_R1_ops.txt 
 

Inside the diff file is the following:

tode@ubuntu:~/perfectscripts/generator$ cat diff_ospf_ios_R1_ops.txt 
 --- learn/ospf_ios_R1_ops.txt
 +++ learn2/ospf_ios_R1_ops.txt
 +info: 
 + vrf: 
 + default: 
 + address_family: 
 ipv4: 
 instance: 
 100: 
 areas: 
 0.0.0.0: 
 area_id: 0.0.0.0
 area_type: normal
 database: 
 lsa_types: 
 1: 
 lsa_type: 1
 lsas: 
 1.1.1.1 1.1.1.1: 
 adv_router: 1.1.1.1
 lsa_id: 1.1.1.1
 ospfv2: 
 body: 
 router: 
 links: 
 1.1.1.1: 
 link_data: 255.255.255.255
 link_id: 1.1.1.1
 topologies: 
 0: 
 metric: 1
 mt_id: 0
 type: stub network
 192.168.122.0: 
 link_data: 255.255.255.0
 link_id: 192.168.122.0
 topologies: 
 0: 
 metric: 1
 mt_id: 0
 type: stub network
 num_of_links: 2
 header: 
 adv_router: 1.1.1.1
 age: 42
 checksum: 0xE54B
 length: 48
 lsa_id: 1.1.1.1
 option: None
 seq_num: 80000001
 type: 1
 interfaces: 
 FastEthernet0/0: 
 bfd: 
 enable: False
 cost: 1
 dead_interval: 40
 demand_circuit: False
 dr_ip_addr: 192.168.122.100
 dr_router_id: 1.1.1.1
 enable: True
 hello_interval: 10
 hello_timer: 00:00:03
 interface_type: broadcast
 lls: True
 name: FastEthernet0/0
 passive: False
 priority: 1
 retransmit_interval: 5
 state: dr
 transmit_delay: 1
 Loopback1: 
 bfd: 
 enable: False
 cost: 1
 demand_circuit: False
 enable: True
 interface_type: loopback
 name: Loopback1
 mpls: 
 te: 
 enable: False
 statistics: 
 area_scope_lsa_cksum_sum: 0x00E54B
 area_scope_lsa_count: 1
 spf_runs_count: 1
 bfd: 
 enable: False
 graceful_restart: 
 cisco: 
 enable: False
 type: cisco
 ietf: 
 enable: False
 type: ietf
 mpls: 
 ldp: 
 autoconfig: False
 autoconfig_area_id: 0.0.0.0
 nsr: 
 enable: False
 preference: 
 single_value: 
 all: 110
 router_id: 1.1.1.1
 spf_control: 
 paths: 4
 throttle: 
 spf: 
 hold: 10000
 maximum: 10000
 start: 5000
 stub_router: 
 always: 
 always: False
 external_lsa: False
 include_stub: False
 summary_lsa: False 

What i’m going to do is include this learn process before and after each change, on top of the diffios change, so that we can have more proof that our change has been significant.

However, we can use this Genie diff also if no change has taken place, or if the change has been external to our administrative domain. Something breaks down even if we didn’t change anything, so how about running a Genie diff to check if we now have fewer ospf routes than before? or fewer bgp routes? It’s never been easier to troubleshoot a network problem.

…to be continued…

Implementing idempotency in Unicon part 7, Special case: Crypto ACL.

Hello

One special case I’m dealing with at the moment is that when expanding the existing VPNs based on crypto ACLs, I get an excel sheet and I need to decide if the new connection between company A and company B necessitates a new crypto SA, in other words: do I need to add an entry in the crypto ACL, or perhaps this new connection is part of the existing old SAs.

This also forms part of the idempotency check that I’m trying to implement, because it doesn’t make sense to add a new ACL entry, if the new entries is a subset of any of the existing entries. So the new entry would need to be compared to each and every one of the existing entries.

An example: I have the following crypto ACL:

10 permit ip 1.1.1.0 0.0.0.255 2.2.2.0 0.0.0.255

This creates a single crypto SA.

The client now sends me an excel file, with the following change:
Please add a connection between 1.1.1.64 to 3.3.3.1

I would now need to create a new SA:

20 permit ip 1.1.1.0 0.0.0.255 3.3.3.0 0.0.0.255

Of course, i could add an SA for single hosts but then if the client sends a similar request for 1.1.1.72 to 3.3.3.55, the crypto ACL needs to be modified on both sides. I don’t want (or don’t need) redundant communication so i prefer to add /24 subnets (or even /16 if i can).

Here comes my pain point. Looking at the excel sheet and deciding whether the new entry is needed or not is manual work, so if I have a non-technical coordinator, they’re not able to say whether the workflow should be: coordinator>VPN specialist>inform other VPN specialist (other GW)> FW specialist>back to coordinator>process people or simply Coordinator>FW specialist>Coordinator.

The logic should be as follows: For the following change when entry 20 is added, when are we allowed to skip entry 20?
10 permit ip A1 B1

20 permit ip A2 B2

Entry 20 is not needed only if:

A2==A1 && B2==B1
OR
A2 is part of A1 && B2 is part of B1.

Let’s look at the netaddr python library documentation:

>>> s1 = IPSet(['192.0.2.0', '192.0.2.1', '192.0.2.2'])
>>> s2 = IPSet(['192.0.2.2', '192.0.2.3', '192.0.2.4'])
>>> s1 & s2
IPSet(['192.0.2.2/32'])
>>> s1.isdisjoint(s2)
False
>>> s1 = IPSet(['192.0.2.0', '192.0.2.1'])
>>> s2 = IPSet(['192.0.2.3', '192.0.2.4'])
>>> s1 & s2
IPSet([])
>>> s1.isdisjoint(s2)
True 

So if we use the disjoint method, our ACL idempotency check should now be (relatively) easy to write up.

In the end I implemented it in a different way:

from cisco_acl.regexes import ace_match
from netaddr import IPSet
from netaddr import IPNetwork
import iplib

source1 = input('what is the source ip ?')
wildcard = input('what is the wildcard ?')
wildcard_converted = iplib.convert_nm(wildcard, 'bit')
source1 = source1 + '/' + wildcard_converted

dest1 = input('what is the dest ip ?')
wildcard = input('what is the wildcard ?')
wildcard_converted = iplib.convert_nm(wildcard, 'bit')
dest1 = dest1 + '/' + wildcard_converted

source2 = input('what is the source ip ?')
wildcard = input('what is the wildcard ?')
wildcard_converted = iplib.convert_nm(wildcard, 'bit')
source2 = source2 + '/' + wildcard_converted

dest2 = input('what is the dest ip ?')
wildcard = input('what is the wildcard ?')
wildcard_converted = iplib.convert_nm(wildcard, 'bit')
dest2 = dest2 + '/' + wildcard_converted


result1 = IPNetwork(source2) in IPSet([source1])
result2 = IPNetwork(dest2) in IPSet([dest1])

if result1 and result2:
  print('we should not add this')
else:
  print('we should add this')
tode@DESKTOP:~$ python3 aclscript.py
 what is the source ip ?192.168.0.0
 what is the wildcard ?0.0.0.255
 
 what is the dest ip ?172.16.1.0
 what is the wildcard ?0.0.0.127
 
 what is the source ip ?192.168.0.0
 what is the wildcard ?0.0.0.31
 
 what is the dest ip ?172.16.1.0
 what is the wildcard ?0.0.0.3
 
 we should not add this

Here i needed to think how I would get the input and compare it with the new input. I came up with this. I will gather existing ACL entries as one dictionary and compare it with the new dictionary of new entries.

from cisco_acl.regexes import ace_match
from netaddr import IPSet
from netaddr import IPNetwork
import iplib

dictionary1 = {"192.168.0.0/24": "10.0.0.0/24", "192.168.1.0/24": "10.0.0.0/30"}

dictionary3 = {"192.168.3.0/24": "10.0.0.0/29", "192.168.4.0/24": "10.0.1.0/27"}

for newkey in dictionary3:
    for oldkey in dictionary1:
        if IPNetwork(newkey) in IPSet([oldkey]) and IPNetwork(dictionary3[newkey]) in IPSet([dictionary1[oldkey]]):
           print('nothing to do')
        else:
           print('something to do')

Now i need to populate the dictionaries with fields from an excel sheet.


If the value of row 1, column C = OLD, i should populate olddictionary,
if the value of row 1, column C = NEW, i should populate newdictionary,

from openpyxl import load_workbook
from netaddr import IPSet
from netaddr import IPNetwork


# load excel with its path
wrkbk = load_workbook("bpadb.xlsx")
sh = wrkbk["Sheet1"]
# to iterate through cells in Column A and print value in console
src = ''
dst = ''
olddict = {}
newdict = {}
changedict = {}
removaldict = {}
for i in range(1,10):
    src = "A" + str(i)
    dst = "B" + str(i)
    comment = "C" + str(i)
    src_cell = sh[src]
    dst_cell = sh[dst]
    comment_cell = sh[comment]
    if comment_cell.value == "OLD":
      olddict[src_cell.value] = dst_cell.value
    if comment_cell.value == "NEW":
      newdict[src_cell.value] = dst_cell.value
print('this is old dictionary ' + str(olddict))
print('this is new dictionary ' + str(newdict))

for newkey in newdict:

    for oldkey in olddict:
        if IPNetwork(newkey) in IPSet([oldkey]) and IPNetwork(newdict[newkey]) in IPSet([olddict[oldkey]]):
           #print('for this new entry ' + str(newkey) + ' ' + str(newdict[newkey]) + ' when compared with entry ' + str(oldkey) + ' ' + str(olddict[oldkey]) + ' there is nothing to do')
           removaldict[newkey] = newdict[newkey]
        else:
           #print(' you need to implement this new acl entry ' + str(newkey) + ' ' + str(newdict[newkey]) + ' because it was not inside ' + str(oldkey) + ' ' + str(olddict[oldkey]))
           changedict[newkey] = newdict[newkey]

print('this is change dictionary ' + str(changedict))
print('this is removal dictionary ' + str(removaldict))


intersect = []
for item in changedict.keys():
    if item in removaldict:
        intersect.append(item)
        #intersect.append(removaldict[item])

print('Two dictionaries intersect here so these entries should be removed: ' + str(intersect))
#print(intersect[0])
#print(intersect[1])
for i in range(0,len(intersect)):
  changedict.pop(intersect[i])
print('final form of changedict is ' + str(changedict))
C:\Users\tode\Desktop>python bpa_script.py
 this is old dictionary {'192.168.0.0/24': '172.16.0.1/32', '192.168.1.0/24': '10.0.0.0/29', '192.168.3.0/26': '169.254.0.0/31'}
 this is new dictionary {'192.168.1.0/27': '10.0.0.0/30', '192.168.3.0/27': '169.254.1.0/31', '192.168.3.0/28': '169.254.0.0/31', '192.168.3.0/29': '169.254.3.0/31'}
 this is change dictionary {'192.168.1.0/27': '10.0.0.0/30', '192.168.3.0/27': '169.254.1.0/31', '192.168.3.0/28': '169.254.0.0/31', '192.168.3.0/29': '169.254.3.0/31'}
 this is removal dictionary {'192.168.1.0/27': '10.0.0.0/30', '192.168.3.0/28': '169.254.0.0/31'}
 Two dictionaries intersect here so these entries should be removed: ['192.168.1.0/27', '192.168.3.0/28']
 final form of changedict is {'192.168.3.0/27': '169.254.1.0/31', '192.168.3.0/29': '169.254.3.0/31'}

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 6: Nested configs and mass changes.

Hello

Today I wanted to see how multiple nested commands (e.g. archive > log config > notify syslog, archive > log config > logging enable) will be configured using my script.

I’ve discovered an interesting thing. With a config like this:

R1, archive, log config, notify syslog,
R1, archive,log config, logging enable,
R1, archive, log config , hidekeys,

Only one of the commands (the last one) would be implemented on the router. It seems that while the defaultdict can have a list of values under a key, its keys still need to be unique. There are a few ways to work around this:

  • adding randomized empty spaces in the subkeys (_log config, log config_, log config, ). This is actually quite nice because spaces have no impact on the ios,
  • cutting the last letter of the key in each command set (archive, archiv, archi)
  • trying to change the logic: having just Archive word as the key, and then [log config, notify syslog] as the values. But this would require a heavy rework of the code. Alas, it’s probably the best solution. So this would be just a defaultdict+defaultdict, rather than a defauldict+defaultdict+defaultdict. On a second thought even a single defaultdict would do: {R1: [archive, log config, notify syslog]}. Why I never thought about this before is beyond me. I made my code (and life) so much more complicated with this triple nested defaultdict.

Anyways, let’s do some mass implementations using the ’empty spaces’ workaround

This is the input:

tode@ubuntu:~/perfectscripts$ cat change.txt
 R1, interface FastEthernet 1/0, ip address 100.100.0.2 255.255.255.0, 
 R1, interface FastEthernet 1/0, no shut, 
 R1, crypto isakmp policy 10, encr aes, 
 R1, crypto isakmp policy 10, hash sha, 
 R1, crypto isakmp policy 10, lifetime 3600, 
 R1, crypto isakmp policy 10, group 5, 
 R1, crypto isakmp policy 10, auth pre-share, 
 R1, crypto isakmp key cisco address 0.0.0.0 0.0.0.0, , 
 R1, crypto ipsec transform-set MYSET esp-des esp-sha-hmac, , 
 R1, crypto map MYMAP 10 ipsec-isakmp, set peer 100.100.1.2,
 R1, crypto map MYMAP 10 ipsec-isakmp, set transform-set MYSET, 
 R1, crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R1, ip access-list extended 101, permit ip host 1.1.1.1 host 2.2.2.2, 
 R1, interface FastEthernet 1/0, crypto map MYMAP, 
 R1, ip route 2.2.2.2 255.255.255.255 100.100.0.1, , 
 R1, int loopback 1, ip address 1.1.1.1 255.255.255.255, 
 R1, ip route 100.100.1.2 255.255.255.255 100.100.0.1, , 
 R1, archive,  log config,   notify syslog,
 R1, archiv,  log config,   logging enable,
 R1, archi,  log config,   hidekeys,
 R2, interface FastEthernet 1/0, ip address 100.100.1.2 255.255.255.0, 
 R2, interface FastEthernet 1/0, no shut, 
 R2, crypto isakmp policy 10, encr aes, 
 R2, crypto isakmp policy 10, hash sha, 
 R2, crypto isakmp policy 10, lifetime 3600, 
 R2, crypto isakmp policy 10, group 5, 
 R2, crypto isakmp policy 10, auth pre-share, 
 R2, crypto isakmp key cisco address 0.0.0.0 0.0.0.0, , 
 R2, crypto ipsec transform-set MYSET esp-des esp-sha-hmac, , 
 R2, crypto map MYMAP 10 ipsec-isakmp, set peer 100.100.0.2,
 R2, crypto map MYMAP 10 ipsec-isakmp, set transform-set MYSET, 
 R2, crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R2, ip access-list extended 101, permit ip host 2.2.2.2 host 1.1.1.1, 
 R2, interface FastEthernet 1/0, crypto map MYMAP, 
 R2, ip route 1.1.1.1 255.255.255.255 100.100.0.2, , 
 R2, int loopback 1, ip address 2.2.2.2 255.255.255.255, 
 R2, ip route 100.100.0.2 255.255.255.255 100.100.1.1, , 
 R2, archive, log config, notify syslog,
 R2, archiv,  log config,   logging enable,
 R2, archi,  log config,   hidekeys,tode@ubuntu:~/perfectscripts$

And the output is…

tode@ubuntu:~/perfectscripts$ python3 MassChangeWithAmanda.py critical R1%R2
  interface FastEthernet 1/0
  ip address 100.100.0.2 255.255.255.0
 no shut
 crypto map MYMAP
 crypto isakmp policy 10
  encr aes
 hash sha
 lifetime 3600
 group 5
 auth pre-share
 crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 crypto map MYMAP 10 ipsec-isakmp
  set peer 100.100.1.2
 set transform-set MYSET
 match address 101
 ip access-list extended 101
  permit ip host 1.1.1.1 host 2.2.2.2
 ip route 2.2.2.2 255.255.255.255 100.100.0.1
 int loopback 1
  ip address 1.1.1.1 255.255.255.255
 ip route 100.100.1.2 255.255.255.255 100.100.0.1
 archive
  log config
  notify syslog
 log config
  logging enable
  log config 
  hidekeys
 I am trying to implement this change on router R1 Are you sure? yes
  interface FastEthernet 1/0
  ip address 100.100.1.2 255.255.255.0
 no shut
 crypto map MYMAP
 crypto isakmp policy 10
  encr aes
 hash sha
 lifetime 3600
 group 5
 auth pre-share
 crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 crypto map MYMAP 10 ipsec-isakmp
  set peer 100.100.0.2
 set transform-set MYSET
 match address 101
 ip access-list extended 101
  permit ip host 2.2.2.2 host 1.1.1.1
 ip route 1.1.1.1 255.255.255.255 100.100.0.2
 int loopback 1
  ip address 2.2.2.2 255.255.255.255
 ip route 100.100.0.2 255.255.255.255 100.100.1.1
 archive
  log config
  notify syslog
  archiv
  log config
  logging enable
  archi
  log config
  hidekeys
 I am trying to implement this change on router R2 Are you sure? yes

I can see one big problem here: because commands are grouped, the script is trying to insert the map on interface fa1/0 when the map doesn’t exist yet. This will not be implemented. It will be implemented if the same script runs again. Oh bother. Diffios change log confirms this:

tode@ubuntu:~/perfectscripts$ cat R1_change_delta.bak 
 --- baseline
 +++ comparison
 1: Current configuration : 2239 bytes
 2: interface FastEthernet1/0
 no ip address
 shutdown
 1: Current configuration : 2767 bytes
 2: access-list 101 permit ip host 1.1.1.1 host 2.2.2.2
 3: archive
 logging enable
 4: crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 5: crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 6: crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 5
 lifetime 3600
 7: crypto map MYMAP 10 ipsec-isakmp
 set peer 100.100.1.2
 set transform-set MYSET
 match address 101
 8: interface FastEthernet1/0
 ip address 100.100.0.2 255.255.255.0
 9: interface Loopback1
 ip address 1.1.1.1 255.255.255.255
 10: ip route 100.100.1.2 255.255.255.255 100.100.0.1
 11: ip route 2.2.2.2 255.255.255.255 100.100.0.1
 tode@ubuntu:~/perfectscripts$ cat R2_change_delta.bak 
 --- baseline
 +++ comparison
 1: Current configuration : 1225 bytes
 2: interface FastEthernet1/0
 no ip address
 shutdown
 1: Current configuration : 1791 bytes
 2: access-list 101 permit ip host 2.2.2.2 host 1.1.1.1
 3: archive
 logging enable
 notify syslog contenttype plaintext
 4: crypto ipsec transform-set MYSET esp-des esp-sha-hmac
 5: crypto isakmp key cisco address 0.0.0.0 0.0.0.0
 6: crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 5
 lifetime 3600
 7: crypto map MYMAP 10 ipsec-isakmp
 set peer 100.100.0.2
 set transform-set MYSET
 match address 101
 8: interface FastEthernet1/0
 ip address 100.100.1.2 255.255.255.0
 9: interface Loopback1
 ip address 2.2.2.2 255.255.255.255
 10: ip route 1.1.1.1 255.255.255.255 100.100.0.2
 11: ip route 100.100.0.2 255.255.255.255 100.100.1.1
 tode@ubuntu:~/perfectscripts$  

After small edits, let’s run the script twice and read the change log from the 2nd run.

tode@ubuntu:~/perfectscripts$ cat R1_change_delta.bak 
 --- baseline
 +++ comparison
 tode@ubuntu:~/perfectscripts$ cat R2_change_delta.bak 
 --- baseline
 +++ comparison

This is nice, nothing changes during the 2nd run. Let’s now check if the script is actually trying to implement config during the 2nd (and every subsequent) run.
It is. Oh boy. The reason is that the input file needs to be sanitized. If it’s not (if change commands don’t reflect the existing config 100%) the script will not realize these are the same commands and will try to implement them again.
This is the sanitized input file:

tode@ubuntu:~/perfectscripts$ cat change.txt 
 R1,crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R1,access-list 101 permit ip host 1.1.1.1 host 2.2.2.2, , 
 R1,interface FastEthernet1/0, ip address 100.100.0.2 255.255.255.0, 
 R1,interface FastEthernet1/0, no shut, 
 R1,interface FastEthernet1/0, crypto map MYMAP, 
 R1,ip route 2.2.2.2 255.255.255.255 100.100.0.1, , 
 R1,interface Loopback1, ip address 1.1.1.1 255.255.255.255, 
 R1,ip route 100.100.1.2 255.255.255.255 100.100.0.1, ,
 R1,archive, log config,  notify syslog,
 R1,archiv,  log config,   logging enable,
 R1,archi,  log config,   hidekeys,
 R2,crypto isakmp policy 10, encr aes, 
 R2,crypto isakmp policy 10, hash sha, 
 R2,crypto isakmp policy 10, lifetime 3600, 
 R2,crypto isakmp policy 10, group 5, 
 R2,crypto isakmp policy 10, authentication pre-share, 
 R2,crypto isakmp key cisco address 0.0.0.0 0.0.0.0, , 
 R2,crypto ipsec transform-set MYSET esp-des esp-sha-hmac, , 
 R2,crypto map MYMAP 10 ipsec-isakmp, set peer 100.100.0.2,
 R2,crypto map MYMAP 10 ipsec-isakmp, set transform-set MYSET, 
 R2,crypto map MYMAP 10 ipsec-isakmp, match address 101, 
 R2,access-list 101 permit ip host 2.2.2.2 host 1.1.1.1, ,  
 R2,interface FastEthernet1/0, ip address 100.100.1.2 255.255.255.0, 
 R2,interface FastEthernet1/0, no shut,  
 R2,interface FastEthernet1/0, crypto map MYMAP, 
 R2,ip route 1.1.1.1 255.255.255.255 100.100.0.2, , 
 R2,interface Loopback1, ip address 2.2.2.2 255.255.255.255, 
 R2,ip route 100.100.0.2 255.255.255.255 100.100.1.1, , 
 R2,archive, log config,  notify syslog,
 R2,archiv,  log config,   logging enable,
 R2,archi,  log config,   hidekeys,tode@ubuntu:~/perfectscripts$ 

and the live output for R2:

crypto isakmp policy 10
  hash sha
 interface FastEthernet1/0
  no shut
 archiv
   log config
    logging enable
 archi
   log config
    hidekeys
 I am trying to implement this change on router R2 Are you sure?

Something is still left. Hash sha is the automatic choice, so doesn’t appear in the config file. This is not something that we need to handle, because sha should no longer be used for VPNs (obsolete).
No shut needs a separate code fix, because if a port is open, there is no command ‚no shut’ on the port.
For the nested commands i don’t have a good solution yet. I believe the whole code needs to be reworked (a single defaultdict with a list of values) for idempotency to work properly.

RAW CODE

Here’s the code with the ‚no shut’ fix:

#!/usr/bin/python3

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging
from gtts import gTTS
import re

LEVELS = {'debug': logging.DEBUG,
          'info': logging.INFO,
          'warning': logging.WARNING,
          'error': logging.ERROR,
          'critical': logging.CRITICAL}

level_name = sys.argv[1]
level = LEVELS.get(level_name, logging.NOTSET)
logging.basicConfig(level=level)
unicondevicelist = sys.argv[2].split("%") 

#loading the testbed file with router management IPs
testbed = loader.load('testbed.yml')

#downloading router configuration files and storing them away as .bak files
#x = 0
for x in range(0,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()
with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            dataline = line.split(',')
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
          
logging.info('\n')
for k, v in my_dict.items():
  logging.info('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  my_new_dict = my_dict[k].items()
  logging.info('\n')
  for item in my_new_dict:
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        logging.info(newstringordict)
      else:
        for k in newstringordict.keys():
          logging.info(k)
realdevicelist = []
for i in my_dict.keys():
  realdevicelist.append(i)
  for j in my_dict[i].keys():
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
m = 1
while True:  
  try:
    x = 0       
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    if m==1:
      text = "I found " + str(m) + " change for a router that is not in the testbed."
    if m > 1:
      text = "I found another error. "
    logging.critical('\n')
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    m = m + 1
    continue
  else:
    for k, v in my_dict.items():
      my_new_dict = my_dict[k].items()
      for item in my_new_dict:
      #item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            logging.info(newstringordict)
          else:
            for z in newstringordict.keys():
              logging.info(z)       
    break        
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      my_new_dict = my_dict[router].items()
      for tpl in my_new_dict:
        interface = tpl[0]
        cfgs = tpl[1]
        for key in cfgs.keys():
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #logging.info('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #logging.warning('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    pass
                  elif key == ' no shut':
                     findshut = ' shut'
                     shutanswer = parse.find_objects_w_child(interface, findshut)
                     if len(shutanswer) > 0:
                        pass
                     else:
                     	pass
                  else:
                    logging.warning(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                    my_final_dict[router][interface][key] = cfgs[key]
                    #connect to router and implement interface, key
          else:
                logging.warning(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement
finalstatement = ''
for k, v in my_final_dict.items():
  my_newfinal_dict = my_final_dict[k].items()
  globlist = []
  for item in my_newfinal_dict:
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        logging.info(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          n = newstringordict.get(j)
          globlist.append(n)
  c = testbed.devices[k]
  c.connect(log_stdout=False)
  for line in globlist:
  	print(line)
  decision = input("I am trying to implement this change on router " + k + " Are you sure? " )
  if "yes" in decision:
    output = c.configure(globlist)
    if 'Invalid' in output:
      logging.critical(output)
  else:
    pass
  c.destroy()

unicondevicelist = realdevicelist
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'
    c = testbed.devices[afterchanges]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    #time.sleep(1)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    diff = diffios.Compare(beforechanges, afterchanges)
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'
    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    os.remove(afterchanges)
    os.remove(beforechanges)




Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 5: interactive changes and more.

This adventure is coming to an end 🙂 It’s been definitely nice as an intellectual exercise, but one can clearly see the downsides: this is not actually network as code, but rather ”network as one messy change text file”. Of course, this text file could be made nicer in excel and underneath it would be a comma-delimited processable csv, but at its best it would be a ”network as a colourful excel file”. I’ve created a monster 😀

But hey, we’re not quite finished making the monster stronger and… weirder. I’ve decided to be bold and added a female voiceover that comments on the various stages of the script processing. Also, the script is now interactive.

#!/usr/bin/python3

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging
from gtts import gTTS

LEVELS = {'debug': logging.DEBUG,
          'info': logging.INFO,
          'warning': logging.WARNING,
          'error': logging.ERROR,
          'critical': logging.CRITICAL}


level_name = sys.argv[1]
level = LEVELS.get(level_name, logging.NOTSET)
logging.basicConfig(level=level)

unicondevicelist = sys.argv[2].split("%")
#print(unicondevicelist) 
text = "This change will be executed on 3 routers: router 1, router 2, router 3. I am going to look at the change text file now "
language =  'en'
speech = gTTS(text = text, lang = language, slow = False)
speech.save("text.mp3")
os.system("play text.mp3")

#loading the testbed file with router management IPs
testbed = loader.load('testbed.yml')

#downloading router configuration files and storing them away as .bak files
x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    f = open(devicename, "w")
    text = "I downloaded a config file from a router."
    language =  'en'
    speech = gTTS(text = text, lang = language, slow = False)
    speech.save("text.mp3")
    os.system("play text.mp3")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()
text = "I now downloaded configs of all routers. I am going to parse this config now."

language =  'en'
speech = gTTS(text = text, lang = language, slow = False)
speech.save("text.mp3")
os.system("play text.mp3")

#parsing the file with the changes
with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            dataline = line.split(',')  
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            
time.sleep(1)
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
logging.info('***************************************************************************************************************************')
for k, v in my_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  logging.info('\n')
  for item in my_new_dict:
    #item is a tuple consisting of string and default dict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        logging.info(newstringordict)
      else:
        for k in newstringordict.keys():
          logging.info(k)
  time.sleep(2)
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('\n\n\n')        	      
logging.info('***************************************************************************************************************************')
logging.info('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')

realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
m = 1
while True:  
  try:
    x = 0
         
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    if m==1:
      text = "I found " + str(m) + " change for a router that is not in the testbed."
    if m > 1:
      text = "I found another error. "
    
    language =  'en'
    speech = gTTS(text = text, lang = language, slow = False)
    speech.save("text.mp3")
    os.system("play text.mp3")
    logging.critical('\n')
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    logging.info('*******************************INFO****************************************************************************')
    logging.info('An error has been found and removed. ')
    logging.info('*******************************INFO****************************************************************************')
    logging.info('\n')
    m = m + 1
    continue
  else:
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('****************************SANITISE FINISHED **************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************SANITISE RESULTS PHASE**********************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    for k, v in my_dict.items():
      logging.info('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      for item in my_new_dict:
      #item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            logging.info(newstringordict)
          else:
            for z in newstringordict.keys():
              logging.info(z)
    logging.info('\n')          
    logging.info('************************************************************************************************************************')
    logging.info('*******************************END OF PLAN*********************************************************************************')
    logging.info('************************************************************************************************************************')
    break 
logging.info('\n')          
logging.info('***************************************************************************************************************************')
logging.info('*******************************COMPARE PHASE*******************************************************************************')
logging.info('***************************************************************************************************************************')
    
logging.info('***************************************************************************************************************************')
logging.info('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
logging.info('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      my_new_dict = my_dict[router].items()
      for tpl in my_new_dict:
        interface = tpl[0]
        cfgs = tpl[1]
        for key in cfgs.keys():
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #logging.info('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #logging.warning('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #logging.warning('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    logging.warning(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                    my_final_dict[router][interface][key] = cfgs[key]
                    #connect to router and implement interface, key
          else:
                logging.warning(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement
finalstatement = ''

for k, v in my_final_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  logging.info('\n')
  globlist = []
  for item in my_newfinal_dict:
    #item is a tuple consisting of a string and defaultdict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        logging.info(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #logging.info(j)
  c = testbed.devices[k]
  c.connect(log_stdout=False)
  finalstatement += ' On ' 
  finalstatement += k
  finalstatement += ' i have tried to configure '
  finalstatement += str(globlist)
  finalstatement += ' \n\n '
  #here implement interactivity
  
  text = "I need you to confirm a configuration change on router " + k + " in the terminal window. Type yes to confirm"
  language =  'en'
  speech = gTTS(text = text, lang = language, slow = False)
  speech.save("text.mp3")
  os.system("play text.mp3")
  decision = input("I am trying to implement the following on router " + k + " " + str(globlist) + " Are you sure you want this? " )
  if "yes" in decision:
    output = c.configure(globlist)
    if 'Invalid' in output:
      logging.critical(output)
      text = "I have found some invalid commands. God damn it, incompetent dumbasses"
      language =  'en'
      speech = gTTS(text = text, lang = language, slow = False)
      speech.save("text.mp3")
      os.system("play text.mp3")
  else:
    pass
  #here implement check if weird return from router, e.g. invalid something
  c.destroy()

logging.info(finalstatement + '\n\n')
logging.warning('***************COMPARE IN PROGRESS****FOR A DETAILED CHANGE LOG FILE SEE CHANGE_DELTA.BAK FILES*******************')


unicondevicelist = realdevicelist
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'
    c = testbed.devices[afterchanges]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    time.sleep(2)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    #time.sleep(10)
    baseline = beforechanges
    time.sleep(3)
    comparison = afterchanges
    time.sleep(3)
    #ignore = 'ignore.txt'
    diff = diffios.Compare(baseline, comparison)
    time.sleep(1)
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'
    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    os.remove(afterchanges)
    os.remove(beforechanges)
logging.info('*******************************THIS IS THE END OF ALL OUTPUT**********************************************')
text = "I have finished executing the changes and prepared a detailed change log. Amanda out"
language =  'en'
speech = gTTS(text = text, lang = language, slow = False)
speech.save("text.mp3")
os.system("play text.mp3")

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 4: implementation of logic fixes

This time the script connects to routers in GNS3, downloads the config, parses the change file, gets the delta (makes a list of commands from it for each router) and implements it using Unicon configure.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys


unicondevicelist = ['R1', 'R2', 'R3']
print(unicondevicelist[0])
testbed = loader.load('testbed.yml')
x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    c.destroy()

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
            print(dataline)
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('\n\n\n')        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for z in newstringordict.keys():
              print(z)
    print('\n')          
    print('***************************************************************************************************************************')
    print('*******************************END OF PLAN*********************************************************************************')
    print('***************************************************************************************************************************')
    
    break 

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
print('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key)
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change')
                  pass
                else:
                  #print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                  my_final_dict[router][interface][key] = ' '
                  #print(router + '\n ' + interface + '\n ' + key + '\n')
                  #connect to router and implement interface, key


for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  globlist = []
  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        print(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #print(j)
  print(globlist)
  print('i am in ' + str(k) + '\n')
  c = testbed.devices[k]
  c.connect()
  output = c.configure(globlist)
  c.destroy()

There is some interesting stuff that’s happening here, and still the script keeps implementing one-line commands, such as logging buffered informational, which needs to be fixed. A slight change in the code, then:

for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key + ' and this is value ' + cfgs[key])
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #print('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                    my_final_dict[router][interface][key] = cfgs[key]
                   
          else:
                print(interface + ' not found on router ' + router)
                my_final_dict[router][interface][key] = cfgs[key]
                
for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  globlist = []
  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        print(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #print(j)
  print(globlist)
  print('i am in ' + str(k) + '\n')
  c = testbed.devices[k]
  c.connect()
  output = c.configure(globlist)
  c.destroy()

I’ve changed two things here: first i added a condition where the parent (like logging buffered informational command) was found in the config but then the value of the tuple is empty, because there can be no subcommand here. Secondly, i’ve added a general Else statement, because sometimes the parent is not found at all (like a missing interface, or a missing general command), and it will have to be implemented.

The dictionary my_final_dict ”gathers” all the data for which implementation conditions are met.

The next step was to log all the changes resulting from the change.txt and remove the interim files. To this end, i copy the files to .bak files at the beginning of the script, then I download the config at the end of the script once again from all routers and do a final diffios compare.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time



unicondevicelist = ['R1', 'R2', 'R3']
testbed = loader.load('testbed.yml')
x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect()
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
            print(dataline)
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('\n\n\n')        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for z in newstringordict.keys():
              print(z)
    print('\n')          
    print('***************************************************************************************************************************')
    print('*******************************END OF PLAN*********************************************************************************')
    print('***************************************************************************************************************************')
    
    break 

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
print('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key + ' and this is value ' + cfgs[key])
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #print('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                    my_final_dict[router][interface][key] = cfgs[key]
                    #print(router + '\n ' + interface + '\n ' + key + '\n')
                    #connect to router and implement interface, key
          else:
                print(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement

finalstatement = ''
for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  globlist = []

  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        print(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #print(j)
 
  c = testbed.devices[k]
  c.connect()
  finalstatement += ' On ' 
  finalstatement += k
  finalstatement += ' i have tried to configure '
  finalstatement += str(globlist)
  finalstatement += ' \n\n '
  

  output = c.configure(globlist)
  c.destroy()

print(finalstatement + '\n\n')
#now download new config again and diffios against old config to see what has really been configured.

print('***************FINAL COMPARE PHASE, FOR A DETAILED CHANGE LOG FILE SEE CHANGE_DELTA.BAK FILES*******************')


unicondevicelist = ['R1', 'R2']
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'

    c = testbed.devices[afterchanges]
    c.connect()
    output = c.execute('show run')
    time.sleep(2)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    #time.sleep(10)


    baseline = beforechanges
    
    time.sleep(3)
    comparison = afterchanges
    
    time.sleep(3)
    #ignore = 'ignore.txt'
    diff = diffios.Compare(baseline, comparison)
    time.sleep(1)
    #print('****************THIS IS DIFFIOS FOR DELTA ' + afterchanges + '***********************************************')
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'

    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    #print('****************THIS IS END OF DIFFIOS FOR ' + afterchanges + '***********************************************')
    os.remove(afterchanges)
    os.remove(beforechanges)


print('*******************************THIS IS THE END OF ALL OUTPUT**********************************************')

So if my change input is the following command batch (which includes fake routers, fake interfaces, bad commands (like service-policy input QOS which will not be implemented because policy QOS doesn’t exist):

tode@ubuntu:~$ cat change.txt
R1,interface FastEthernet2/0, shut,
R1,interface FastEthernet0/0, description LAN,
R1,interface FastEthernet1/0, shut,
R1,interface FastEthernet1/0, service-policy input QOS,
R2,interface FastEthernet2/0, ip address 192.168.0.1 255.255.255.0,
R2,interface FastEthernet2/0, description LAN,
R2,interface FastEthernet0/2, description WAN,
R2,interface FastEthernet0/0, description DMZ,
R8,interface FastEthernet4/0, description LAN,
R9,interface Loopback1, description LOOP,
R9,interface Loopback1, ip address 1.1.1.1 255.255.255.0,
R2,interface Loopback1, description LOOP,
R1,logging buffered errors, , 

Then my real change output is this:

tode@ubuntu:~$ cat R1_change_delta.bak
--- baseline
+++ comparison


--- Current configuration : 2239 bytes
--- logging buffered informational

+++ Current configuration : 2249 bytes
+++ interface FastEthernet0/0
+++  description LAN
+++ logging buffered errors

and this:

tode@ubuntu:~$ cat R2_change_delta.bak
--- baseline
+++ comparison


--- Current configuration : 1225 bytes
    interface FastEthernet2/0
--- no ip address

+++ Current configuration : 1337 bytes
    interface FastEthernet0/0
+++ description DMZ
    interface FastEthernet2/0
+++ description LAN
+++ ip address 192.168.0.1 255.255.255.0
    interface Loopback1
+++ description LOOP
+++ no ip address

Next steps

  • I should probably create an error.log file as well to collect all the stuff that is wrong with the input (non-existing routers, failed attempts to create interfaces, configs which were already there on the router so were not implemented etc.).
  • I should also think about deletion commands (no ip address etc.), and multiple nested commands (like QoS).
  • Maybe the script should also ask the user if he agrees to implement config whenever there is a need to do it (interactive change).
  • I should get rid of all hardcoded values


By the way the script breaks if there is an end of line character at the end of change.txt
This can be fixed with the following commands in vi before saving.

set noeol
set nofixendofline

This is now a version with the logging implemented and some hardcoded values removed.

#!/usr/bin/python3

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging

LEVELS = {'debug': logging.DEBUG,
          'info': logging.INFO,
          'warning': logging.WARNING,
          'error': logging.ERROR,
          'critical': logging.CRITICAL}


level_name = sys.argv[1]
level = LEVELS.get(level_name, logging.NOTSET)
logging.basicConfig(level=level)

unicondevicelist = sys.argv[2].split("%")
print(unicondevicelist) 


testbed = loader.load('testbed.yml')


x = 0
for x in range(x,len(unicondevicelist)):
    devicename = unicondevicelist[x]
    c = testbed.devices[devicename]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    f = open(devicename, "w")
    f.write(output)
    f.close()
    newdevicename = devicename + '.bak'
    shutil.copyfile(devicename, newdevicename)
    c.destroy()

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            dataline = line.split(',')  
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            
time.sleep(1)
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
logging.info('***************************************************************************************************************************')
for k, v in my_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  logging.info('\n')
  for item in my_new_dict:
    #item is a tuple consisting of string and default dict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        logging.info(newstringordict)
      else:
        for k in newstringordict.keys():
          logging.info(k)
  time.sleep(2)
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
logging.info('\n\n\n')        	      
logging.info('***************************************************************************************************************************')
logging.info('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('\n')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')
logging.info('***************************************************************************************************************************')

realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0        
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    logging.critical('\n')
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    logging.critical('*******************************ERROR***************************************************************************')
    logging.critical('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    logging.info('*******************************INFO****************************************************************************')
    logging.info('An error has been found and removed. ')
    logging.info('*******************************INFO****************************************************************************')
    logging.info('\n')
    continue
  else:
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('****************************SANITISE FINISHED **************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************SANITISE RESULTS PHASE**********************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    logging.info('\n\n\n')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    logging.info('************************************************************************************************************************')
    logging.info('************************************************************************************************************************')
    for k, v in my_dict.items():
      logging.info('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      for item in my_new_dict:
      #item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            logging.info(newstringordict)
          else:
            for z in newstringordict.keys():
              logging.info(z)
    logging.info('\n')          
    logging.info('************************************************************************************************************************')
    logging.info('*******************************END OF PLAN*********************************************************************************')
    logging.info('************************************************************************************************************************')
    break 
logging.info('\n')          
logging.info('***************************************************************************************************************************')
logging.info('*******************************COMPARE PHASE*******************************************************************************')
logging.info('***************************************************************************************************************************')
    
logging.info('***************************************************************************************************************************')
logging.info('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
logging.info('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      my_new_dict = my_dict[router].items()
      for tpl in my_new_dict:
        interface = tpl[0]
        cfgs = tpl[1]
        for key in cfgs.keys():
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #logging.info('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #logging.warning('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change ' + interface + ' ' + key)
                  pass
                else:
                  if key == ' ':
                    #logging.warning('parent found but detailed match parent and key not found because this is a dummy value. not doing anything (cases like logging buffered informational')
                    pass
                  else:
                    logging.warning(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                    my_final_dict[router][interface][key] = cfgs[key]
                    #connect to router and implement interface, key
          else:
                logging.warning(interface + ' not found on router ' + router + ' but i will try to create this ' + interface + ' with the value ' + key)
                my_final_dict[router][interface][key] = cfgs[key]
                #connect to router and try to implement
finalstatement = ''

for k, v in my_final_dict.items():
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  logging.info('\nHere i am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  logging.info('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  logging.info('\n')
  globlist = []
  for item in my_newfinal_dict:
    #item is a tuple consisting of string and default dict
    x = 0
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        globlist.append(newstringordict)
        logging.info(newstringordict)
      else:
        for j in newstringordict.keys():
          globlist.append(j)
          #logging.info(j)
  c = testbed.devices[k]
  c.connect(log_stdout=False)
  finalstatement += ' On ' 
  finalstatement += k
  finalstatement += ' i have tried to configure '
  finalstatement += str(globlist)
  finalstatement += ' \n\n '
  #here implement interactivity
  output = c.configure(globlist)
  logging.critical(output)
  #here implement check if weird return from router, e.g. invalid something
  c.destroy()

logging.info(finalstatement + '\n\n')
logging.warning('***************COMPARE IN PROGRESS****FOR A DETAILED CHANGE LOG FILE SEE CHANGE_DELTA.BAK FILES*******************')


unicondevicelist = realdevicelist
x = 0
for x in range(x,len(unicondevicelist)):
    afterchanges = unicondevicelist[x]
    beforechanges = afterchanges + '.bak'
    c = testbed.devices[afterchanges]
    c.connect(log_stdout=False)
    output = c.execute('show run')
    time.sleep(2)
    f = open(afterchanges, "w")
    f.write(output)
    f.close()
    #time.sleep(10)
    baseline = beforechanges
    time.sleep(3)
    comparison = afterchanges
    time.sleep(3)
    #ignore = 'ignore.txt'
    diff = diffios.Compare(baseline, comparison)
    time.sleep(1)
    deltaoutput = diff.delta()
    filename = afterchanges + '_change_delta.bak'
    f = open(filename, "w")
    f.write(deltaoutput)
    f.close()
    os.remove(afterchanges)
    os.remove(beforechanges)
logging.info('*******************************THIS IS THE END OF ALL OUTPUT**********************************************')


Now let’s execute this script. After the modifications i’m using a logging arg and a list of routers that I want to execute the change on. The output is much less verbose by default

tode@ubuntu:~$ python3 testtest.py critical R1%R2%R3
['R1', 'R2', 'R3']
CRITICAL:root:
CRITICAL:root:ERROR**
CRITICAL:root:[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R8
CRITICAL:root:ERROR**
CRITICAL:root:
CRITICAL:root:
CRITICAL:root:ERROR**
CRITICAL:root:[Errno 2] No such file or directory: 'R9'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R9
CRITICAL:root:ERROR**
CRITICAL:root:
CRITICAL:root:interface FastEthernet0/0
description LAN
interface FastEthernet1/0
service-policy input QOS
% policy map QOS not configured
logging buffered errors
CRITICAL:root:interface FastEthernet2/0
ip address 192.168.0.1 255.255.255.0
description LAN
interface FastEthernet0/2
interface FastEthernet0/2
^
% Invalid input detected at '^' marker.
description WAN
description WAN
^
% Invalid input detected at '^' marker.
interface FastEthernet0/0
description DMZ
interface Loopback1
description LOOP

----------------------------------------

tode@ubuntu:~$ cat R1_change_delta.bak
--- baseline
+++ comparison
1: Current configuration : 2239 bytes
2: logging buffered informational
1: Current configuration : 2249 bytes
2: interface FastEthernet0/0
description LAN
3: logging buffered errors
tode@ubuntu:~$ cat R2_change_delta.bak
--- baseline
+++ comparison
1: Current configuration : 1225 bytes
2: interface FastEthernet2/0
no ip address
1: Current configuration : 1337 bytes
2: interface FastEthernet0/0
description DMZ
3: interface FastEthernet2/0
description LAN
ip address 192.168.0.1 255.255.255.0
4: interface Loopback1
description LOOP

I will continue this in Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 5: negation commands and interactive changes

Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 3: Rewriting the dictionary and getting the delta

Continuing from Part 2, where i found the bug in my ”input dictionary), which eliminated commands if they applied to the same interface. This is because a normal dictionary only allows unique keys, whereas with the defaultdict one can add a list of values to one key.
After a considerable amount of cursing, I rewrote the dictionary method:

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter



with open('change.txt', 'r') as reader
 configchangetext = reader.read()
 splittext = configchangetext.split('\n')
 my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
 for line in splittext:
        dataline = line.split(',')
        my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]

where the input file looks like this:

R1,interface fa0/1, shut,
R1,interface fa0/0, shut,
R1,interface fa0/0, shut,
R1,interface fa0/0, shut,
R1,interface fa0/0, hsrp 10,
R1,interface fa0/0, service-policy QOS,
R2,interface fa0/0, ip address 192.168.0.1 255.255.255.0,
R2,interface fa0/4, desc LAN,
R2,interface fa0/2, desc WAN,
R2,interface fa0/0, desc DMZ,

Now the question is: how to iterate through this dictionary so that we get a config block?

for k, v in my_dict.items():
 print (str(k) + '\n')

 items = my_dict[k].items()
 
 for item in items:
   #print(item)#each item is a tuple consisting of a string and a defaultdict which in turn consists of the command and an empty space...
   x = 0
   for x in range (x, len(item)):
     new_string_or_dict= item[x]
     if isinstance(new_string_or_dict, str):
       print(new_string_or_dict)
     else:
       for k in new_string_or_dict.keys():
         print(k)

So now I have both the dictionary and i know how to loop through this dictionary to generate a config block. Job almost done.

Now some small rewrite of the ”sanitize” method from part 2 blog entry, plus I insert a fake R8 config into the input text file.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)

        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do not have a config.\n This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n However, I still need to check if it makes sense to implement these changes because these may already be found in existing configs. ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************FINAL IMPLEMENTATION PLAN*******************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for k in newstringordict.keys():
              print(k)
    print('*******************************END OF PLAN***********************************************************************')
    break 
    #print('This is the list again ' + str(realdevicelist))  

And the output is quite magnificent, you can see how the changes for the non-existent R8 router are eliminated from the actual change config.

IMPLEMENTATION DRAFT PLAN
***
i am showing now how I want router config to be modified for router R1

interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
i am showing now how I want router config to be modified for router R8

interface fa0/8
desc LAN

END OF IMPLEMENTATION DRAFT PLAN***




CONFIG SANITIZATION IN PROGRESS



ERROR**
[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do not have a config.
This error appeared in R8
ERROR**
INFO* An error has been found and removed. INFO*


***SANITISE FINISHED **




SANITISE RESULTS PHASE***

After removing all errors from the implementation plan, what follows is the sanitized implementation plan.
However, I still need to check if it makes sense to implement these changes because these may already be found in existing configs.


FINAL IMPLEMENTATION PLAN


i am showing now how I want router config to be modified for each router:
R1
interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for each router:
R2
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
END OF PLAN

So now that we have the ‚sanitized’ input config block, it is time to compare this with the actual existing config and only select those commands for implementation that are not in the existing config.

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL PHASE*******************************************************************************')
print('***************************************************************************************************************************')
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key)
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change')
                  pass
                else:
                  #print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                  print(router + ' ' + interface + ' ' + key + '\n')
                #connect to i and implement config
              
              #print(answer)

And the final output of the whole script:

IMPLEMENTATION DRAFT PLAN
***
i am showing now how I want router config to be modified for router R1

interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
i am showing now how I want router config to be modified for router R8

interface fa0/8
desc LAN

END OF IMPLEMENTATION DRAFT PLAN***




CONFIG SANITIZATION IN PROGRESS



ERROR**
[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R8
ERROR**
INFO* An error has been found and removed. INFO*


***SANITISE FINISHED **




SANITISE RESULTS PHASE***

After removing all errors from the implementation plan, what follows is the sanitized implementation plan.
This is not the final implementation plan yet, because i still need to compare the plan with actual configs.


IMPLEMENTATION PLAN BEFORE COMPARE PHASE*


i am showing now how I want router config to be modified for each router:
R1
interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for each router:
R2
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN

END OF PLAN


COMPARE PHASE


FINAL CONFIG DELTA FOR IMPLEMENTATION

R1 interface fa0/0 hsrp 10
R1 interface fa0/0 service-policy QOS
R2 interface fa0/0 ip address 192.168.0.1 255.255.255.0
R2 interface fa0/0 desc DMZ

I can also create another (final) dictionary that will only gather valid, implementable items.
Here’s the change.txt file

R1,interface fa0/1, shut,
R1,interface fa0/0, desc LAN,
R1,interface fa0/0, shut,
R1,interface fa0/0, hsrp 10,
R1,interface fa0/0, service-policy QOS,
R2,interface fa0/0, ip address 192.168.0.1 255.255.255.0,
R2,interface fa0/4, desc LAN,
R2,interface fa0/2, desc WAN,
R2,interface fa0/0, desc DMZ,
R8,interface fa0/8, desc LAN,
R9,interface loop1, desc LOOP,
R2,interface loop1, desc LOOP,

here’s the R1 file

hostname R1
!
interface loop1
ip address 1.1.1.1 255.255.255.255
shut
!
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc LAN
shut
!
logging buffered

Here’s the R2 file

hostname R2
!
interface loop1
ip addr 2.2.2.2 255.255.255.255
interface fa0/0
ip addr 192.168.1.1 255.255.255.0

And here’s the (perhaps) latest form of the script.

from collections import defaultdict
import json
import pprint
import collections
from operator import itemgetter
from ciscoconfparse import CiscoConfParse

with open('change.txt', 'r') as reader:
          configchangetext = reader.read()

          splittext = configchangetext.split('\n')
          my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
          for line in splittext:
            
            dataline = line.split(',')
            #for l in dataline:
              #print(l)
            my_dict[dataline[0]][dataline[1]][dataline[2]] = dataline[3]
            


#print(my_dict)
print('\n')
print('***************************************************************************************************************************')
print('*******************************IMPLEMENTATION DRAFT PLAN*******************************************************************')
print('***************************************************************************************************************************')
for k, v in my_dict.items():
  #print (str(k) + '\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_new_dict = my_dict[k].items()
  print('\n')
  for item in my_new_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')
print('\n\n\n')        	
        
print('***************************************************************************************************************************')
print('***************************END OF IMPLEMENTATION DRAFT PLAN****************************************************************')
print('***************************************************************************************************************************')
print('\n')

print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('****************************CONFIG SANITIZATION IN PROGRESS****************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
print('***************************************************************************************************************************')
realdevicelist = []
for i in my_dict.keys():
      #i add the devices to the list from dict main keys
  realdevicelist.append(i)
  for j in my_dict[i].keys():
      	#i don't do anything here
    pass
    #let's now use this list of keys (R1, R2, R3, R4 in this case) and using those names let's open the configs from desktop.
    #if a config is not found, an exception is caught
while True:
  try:
    x = 0
        #print(str(realdevicelist) + '  ..this is the list ')
    for x in range(x,len(realdevicelist)):
      with open(realdevicelist[x], 'r') as reader:
        config = reader.read()
        my_config_as_list = config.split("\n")
  except FileNotFoundError as e:
    print('\n')
    print('*******************************ERROR***************************************************************************')
    print(str(e) + '.This may mean that in the change file there is a router mentioned for which i do \nnot have a config. This error appeared in ' + realdevicelist[x])
    print('*******************************ERROR***************************************************************************')
    print('\n')
    del my_dict[realdevicelist[x]]
    realdevicelist.remove(realdevicelist[x])
    print('*******************************INFO****************************************************************************')
    print('An error has been found and removed. ')
    print('*******************************INFO****************************************************************************')
    print('\n')
    continue
  else:
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE FINISHED **************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('*******************************SANITISE RESULTS PHASE**********************************************************************')
    print('***************************************************************************************************************************')
    print('After removing all errors from the implementation plan, what follows is the sanitized implementation plan. \n This is not the final implementation plan yet, because i still need to compare the plan with actual configs.  ')
    print('\n\n\n')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    print('*******************************IMPLEMENTATION PLAN BEFORE COMPARE PHASE****************************************************')
    print('***************************************************************************************************************************')
    print('***************************************************************************************************************************')
    for k, v in my_dict.items():
    #print (str(k) + '\n')
      print('\n i am showing now how I want router config to be modified for each router: \n\n' + str(k))
      my_new_dict = my_dict[k].items()
      
      for item in my_new_dict:
      #print(item)#item is a tuple consisting of string and default dict
        x = 0
        for x in range (x, len(item)):
          newstringordict= item[x]
          if isinstance(newstringordict, str):
            print(newstringordict)
          else:
            for z in newstringordict.keys():
              print(z)
    print('\n')          
    print('***************************************************************************************************************************')
    print('*******************************END OF PLAN*********************************************************************************')
    print('***************************************************************************************************************************')
    
    break 

print('\n')          
print('***************************************************************************************************************************')
print('*******************************COMPARE PHASE*******************************************************************************')
print('***************************************************************************************************************************')
    

print('***************************************************************************************************************************')
print('*******************************FINAL CONFIG DELTA FOR IMPLEMENTATION*******************************************************')
print('***************************************************************************************************************************')
my_final_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
for router, config in my_dict.items():
    #print (str(router) + '\n')
    #print('i am on router ' + router + '\n')
    with open(router, 'r') as reader:
      config = reader.read()
      my_config_as_list = config.split("\n")
      parse = CiscoConfParse(my_config_as_list)
      #print('\n final config for ' + str(router) + ':' + ' \n\n' + str(router))

      my_new_dict = my_dict[router].items()
      
      for tpl in my_new_dict:
      #print(item)#cfg is a tuple consisting of interface string and defaultdict config values: 'shut' : ' '
        
        interface = tpl[0]
        cfgs = tpl[1]
        
        for key in cfgs.keys():
          #print('this is a key ' + key)

          #print('this is ' + interface + ' and this is its config ' + key)
          answer = parse.find_objects('^' + interface)
          if len(answer) > 0:
                #print('a subkey from the input config dictionary matches a parent object from the actual config. This does not tell us anything yet. Maybe the value will be modified, maybe not ') 
                second_answer = parse.find_objects_w_child(interface, key)
                if len(second_answer) > 0:
                  #print('i am on ' + router + ' in subkey ' + interface + ' in value ' + key + ' and I have found a match in the existing configuration of ' + router + ' so i am not going to implement this change')
                  pass
                else:
                  #print(' no match  in config of ' + router + ' found when on ' + router + ' in subkey ' + interface + ' in value ' + key + ' so i will implement this change ')
                  
                  my_final_dict[router][interface][key] = ' '
                  #print(router + '\n ' + interface + '\n ' + key + '\n')
                  #connect to router and implement interface, key


for k, v in my_final_dict.items():

  print('-------------------------------------------------------------------------------------------------------------------------')
  print('\ni am showing now how I FINALLY want router config to be modified for router ' + str(k) + '  ' + '\n\n')
  print('-------------------------------------------------------------------------------------------------------------------------')
  my_newfinal_dict = my_final_dict[k].items()
  print('\n')
  for item in my_newfinal_dict:
    #print(item)#item is a tuple consisting of string and default dict
    x = 0
    
    for x in range (x, len(item)):
      newstringordict= item[x]
      if isinstance(newstringordict, str):
        print(newstringordict)
      else:
        for k in newstringordict.keys():
          print(k)

This now produces the following output:

IMPLEMENTATION DRAFT PLAN
***
i am showing now how I want router config to be modified for router R1

interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
interface loop1
desc LOOP
i am showing now how I want router config to be modified for router R8

interface fa0/8
desc LAN
i am showing now how I want router config to be modified for router R9

interface loop1
desc LOOP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

END OF IMPLEMENTATION DRAFT PLAN***




CONFIG SANITIZATION IN PROGRESS



ERROR**
[Errno 2] No such file or directory: 'R8'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R8
ERROR**
INFO* An error has been found and removed. INFO*
ERROR**
[Errno 2] No such file or directory: 'R9'.This may mean that in the change file there is a router mentioned for which i do
not have a config. This error appeared in R9
ERROR**
INFO* An error has been found and removed. INFO*


***SANITISE FINISHED **




SANITISE RESULTS PHASE***

After removing all errors from the implementation plan, what follows is the sanitized implementation plan.
This is not the final implementation plan yet, because i still need to compare the plan with actual configs.


IMPLEMENTATION PLAN BEFORE COMPARE PHASE*


i am showing now how I want router config to be modified for each router:
R1
interface fa0/1
shut
interface fa0/0
desc LAN
shut
hsrp 10
service-policy QOS
i am showing now how I want router config to be modified for each router:
R2
interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface fa0/4
desc LAN
interface fa0/2
desc WAN
interface loop1
desc LOOP

END OF PLAN


COMPARE PHASE


FINAL CONFIG DELTA FOR IMPLEMENTATION**
***
i am showing now how I FINALLY want router config to be modified for router R1

interface fa0/0
hsrp 10
service-policy QOS
i am showing now how I FINALLY want router config to be modified for router R2

interface fa0/0
ip address 192.168.0.1 255.255.255.0
desc DMZ
interface loop1
desc LOOP

The next step is to test this in GNS3: I will change the code so that the script downloads actual configs of GNS routers, then I will add the part where the script should push out the compare delta to execute the changes.

Using a nested default dictionary to create a router configuration block

Normal python dictionaries allow a single value for a given key. However, a default dictionary allows a list as values, which solves the problem of non-unique keys.
An example which is not allowed in python.

vodka: gray, vodka: blue, vodka: red

Let’s look how we can deal with this using defaultdict.

from collections import defaultdict


myconfigstring = ('interface fa0/0, ip address 192.168.0.2 255.255.255.0, shutdown, desc LAN\ninterface fa0/1, ip address 192.168.1.2 255.255.255.0, shutdown, desc WAN')
data_list = [lines.split(",") for lines in myconfigstring.split("\n")]
full_dict = defaultdict(list)

for line in data_list:
    print(line)
    interface = line[0]
    ipaddress = line[1]
    state = line[2]
    desc = line[3]
    
    interfacedetails = (ipaddress, state, desc)
    full_dict[interface].append(interfacedetails)
print(full_dict)

This produces:

defaultdict(, {'interface fa0/0': [(' ip address 192.168.0.2 255.255.255.0', ' shutdown', ' desc LAN')], 'interface fa0/1': [(' ip address 192.168.1.2 255.255.255.0', ' shutdown', ' desc WAN')]})

However, this is not a nested one yet, because we might want to have a dict like this:
{alcohols: {vodkas: {red, blue, green}, beers: {strong, weak}}}

Let’s now look at how we can use this to populate a router configuration block.

my_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
my_dict['R1']['interface fa0/0']['ip address'] = '192.168.0.1 255.255.255.0'
my_dict['R1']['interface fa0/0']['desc'] = 'LAN'
my_dict['R2']['interface loop0/0']['ip address'] = '2.2.2.2 255.255.255.0'

This produces the desired result:

{"R1": {"interface fa0/0": {"ip address": "192.168.0.1 255.255.255.0", "desc": "LAN"}}, "R2": {"interface loop0/0": {"ip address": "2.2.2.2 255.255.255.0"}}}

Where do we go from here?

Now it’s time to rewrite our configuration change file and rewrite the method to create and populate the dictionary from that input file.

This is the original form of my method.

def makedictionaryfromchangefile():
     
    #looping through list of configs and parsing the config
    with open('change.txt', 'r') as reader:
          configchangetext = reader.read()
          stext = configchangetext.split('\n')
          mydict = dict(map(lambda s: s.split(','), stext))
          Dict = { }
          for k, v in mydict.items():
            v = dict(z.split(";") for z in v.split("?"))
            Dict[k] =  {}
            y = 0
            for y in range(y,len(v)):
              eachitem = list(v.items())[y]       
              Dict[k][eachitem[0]] = eachitem[1]
     
    return Dict

to be continued in Implementing idempotency in Unicon a.k.a do the desired changes already exist on the router? Part 3, Rewriting the dictionary

Sanity checks of whisky list in my cellar with while loop with Continue keyword in Python

Let’s say that 3 years ago i looked into my cellar and made a detailed inventory of what I found there. This list is 100% reliable. This is a nested dictionary.

{ 'whisky': { '1983': 'full',
              '1981': 'full',
              '1976': 'label damaged '},
  'vodka': {'2005': 'funny smell'},
  'champagne': {'2018': 'pricey'}}

However, when I went to the cellar I realized that I had drunk the champagne a long time ago already. I prepared two simple text files with the names of what I still have. The first file is called whisky, the second file is called vodka. The files are empty.

Now I want to get rid of this item in my nested dictionary because i don’t have it anymore. The continue phrase in Except clause will help me sanitize the list and go back to the main Try clause.

import pprint
  Dict = { 'whisky': { '1983': 'full',
              '1981': 'full',
              '1976': 'label damaged '},
  'vodka': {'2005': 'funny smell'},
  'champagne': {'2018': 'pricey'}}
#i'm gathering here list of alcohol types
dictbottlelist = []
for i in Dict.keys():
  dictbottlelist.append(i)
  for j in Dict[i].keys():
    pass

while True:
  #trying to open file with a specific alcohol type
  try:
    x = 0
    for x in range(x,len(dictbottlelist)):
      with open(dictbottlelist[x], 'r') as reader:
        pass
  #if the file with alcohol type doesn't exist, delete it from inventory and continue with the loop      
  except FileNotFoundError as e:
      
      print(str(e) + '.This bottle is GONE ' + dictbottlelist[x])
      del Dict[dictbottlelist[x]]
      dictbottlelist.remove(dictbottlelist[x])
      print('An error in the list has been found and removed. ')
      continue
  else:
    print('After removing all errors from inventory, this is what you have in your cellar. ')
    pp = pprint.PrettyPrinter(indent=4)
    pp.pprint(Dict)
    break     
this is what you thought you had in your cellar:
{ 'champagne': {'2018': 'pricey'},
'vodka': {'2005': 'funny smell'},
'whisky': {'1976': 'label damaged ', '1981': 'full', '1983': 'full'}}
[Errno 2] No such file or directory: 'champagne'.This bottle is GONE champagne
An error in the list has been found and removed.
After removing all errors from inventory, this is what you really have in your cellar.
{ 'vodka': {'2005': 'funny smell'},
'whisky': {'1976': 'label damaged ', '1981': 'full', '1983': 'full'}}

I will use this to sanitize my config change file in the other blog post…