Wyróżnione

Networks can be fun (for life)

john_whitelbow

Hi!

My name is Tomek De Wille. This is my network automation blog.

Jordan Peterson says: ”you can be so much more than you are”. I would like my blog to reflect this sentence. Let’s learn as much as possible in the time we have.

At the moment I would like to focus on automation frameworks for networks with occasional use of Python. I will be trying to present some python features using real-life examples, which I will be then using to create some automation scripts for my network.

The road goes ever on.

The last few months have been even busier than before. I’m in the process of changing my job, next month my daughter will be born so there’s a few things to do before that date, and i’ve recently taken up learning Arabic because it’s something i’ve always wanted to do. I sort of feel like Indiana Jones when i’m reading that.

Anyways, back to technical topics. I’m going to focus on Azure and automation in general from now on. I don’t think i can do a lot of writing this year considering all my life changes but i’ll give it a try. What i want to do right now is think (and read) how companies create a hybrid network. For example, if there’s a traditional DC and a mix of traditional and modern business partners (cloud and non-cloud), how are companies able to scale out their connectivity using as much automation as possible. I hate those excel files with requests for a new vpn, a modification of an old vpn etc. I feel this is the old way of doing things and there must be something better, like an approval-led automated workflow in some cloud tool which can deploy things like this in a matter of seconds.

Playing with Azure VPN gateway

Hello

It seems that the whole world is moving to the cloud, so i thought i’d give it a try. I’ve set up a small lab where I’m trying out a range of VPN possibilities with Azure.

I’ve tested so far:

  • ikev2 route-based VPN to Cisco SVTI
  • ikev1 policy-based VPN to cisco policy-based with multiple SAs
  • ikev2 route-based vpn with traffic selectors to Cisco policy-based

It all looks fine. The great thing is, within the $200 credit they give you for free, i was even able to set up a remote tshoot session with their engineer. I was really happy to have that session due to some weird issues with the azure ipsec gateway.

MTU issues in crypto map tunnel connections

Hello

A strange issue at work has come up so it’s back to the drawing board… i mean GNS3 🙂

The problem is that user cannot connect to an application. It seems that the application sends packets that are too large for the path MTU. After some research, it seems that this problem is described in scenario 8 from the following document.

https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html#t6

Long story short, PMTUD will break if ‚no ip unreachables’ is configured anywhere on the path or if some firewall blocks ICMP. One workaround is to actually clear DF bit on the crypto map entry with ‚set security-association df bit clear’ command.
GNS3 testing shows that this should work but whether the application will like it is a different story.

Off-topic || Learning Languages

And now for something completely different…
As you might know, I enjoy learning languages (normal languages, not like Python) so I like to read a book once in a while and this is my method of learning (or preserving) other languages. Every 4-5 years i try to learn a new language and then I incorporate it into my reading plan which for 2021 is 6 books in French, 4 books in Spanish, 2 books in German, so every month i try to read one book in one language and next month it’ll be a different language. The bias is towards French at the moment because A: that’s the most recent language I started and i’m still not proficient, B: I use English and German at work so i don’t have to read that much in them.
I started my system back in 1999 when I read my first books in English (I think it was Lord of the Rings and i was well proud of myself) and I kept adding other languages, German around 2007, Spanish in 2010 and French in 2018. I guess i could have squeezed in one more language but i had an interesting career turn between 2009 and 2011 so i kind of spent all my free time for a few years doing Cisco stuff.

Anyways, my ”reading system” works pretty stable now, with the obvious difficulty being finding enough interesting books.
For 2021 i’ve already ticked off one book in French and now I’m reading ‚Le sang des elfes’ now by Sapkowski at a healthy pace of 30 pages a day so i can finish it in around a month. In February I will be reading something in Spanish.
My plan for 2022-2026 will be to start with either Russian or Arabic. I guess that I can still learn these two languages within the next 15 years and then the system might collapse on account of the fact that in order to keep your level at C1/C2, you need to use the language frequently and i don’t think this is doable with more than 5 different languages plus the motivation factor plays a big role. I’ve noticed that it is getting more difficult to find a good reason to keep learning, because spending time with my son is more fun and i’m going to have a daughter now.


Design by contract

Hello
I’ve found this great package today, which allows you to set some constraints on your function parameters.

https://pypi.org/project/PyContracts/

Long story short, you can use the contracts like this:

@contract(a='int,>0', b='list[N],N>0', returns='list[N]')
def my_function(a, b):
    ...

or like this:

@contract
def my_function(a : 'int,>0', b : 'list[N],N>0') -> 'list[N]':
     # Requires b to be a nonempty list, and the return
     # value to have the same length.
     ...

I’m loving this!

Dq support in Cisco pyats

Today just a short entry. I’ve found that it’s much easier to use Dq to get your values from parsed outputs.

This is the traditional, ”complex way” of getting value for the key [‚version short’] of the ”subdictionary” of the main dictionary where the key is [‚version’]

testbed = Genie.init('testbed.yml')
r1 = testbed.devices['R1']
r1.connect(log_stdout=False)  
shver = r1.parse("show version")
print('software version is: ' + shver['version']['version_short’])

And this is the cool way: creating a query.

from genie.utils import Dq
from genie.testbed import load

routingTable = r1.parse('show ip route')
print(routingTable.q.contains('connected').get_values('routes'))
OUTPUT:
['200.200.200.200/32', '200.200.200.201/32', '192.168.122.0/24', '1.1.1.1/32', '100.100.0.0/24']


This could be used to find any values that are below or above some threshold: eigrp neighbors that have flapped recently, ”fresh” BGP routes, spanning tree events etc.

Using Robot Framework to create easy tests

Hello

PyATS, Genie, and unicon implement robot framework to create easy verifications for your network. Today i’ve created this simple robot testset that goes through a number of tests.

*** Settings ***
Library    ats.robot.pyATSRobot
Library    genie.libs.robot.GenieRobot
Library    pyats.aetest.container.TestContainer
Library    genie.libs.robot.GenieRobotApis
Library    unicon.robot.UniconRobot



*** Test Cases ***

Connect to HUB1 device
    use genie testbed "testbed_session2_slide25.yaml"
    connect to device "HUB1"
Verify that number of bgp routes on HUB1 device is 0
    verify count "0" "bgp routes" on device "HUB1"
Verify that Tunnels are UP on HUB1
    verify interface config no shutdown  device=HUB1  interface=tunnel1251
Disconnect from HUB1 device
    disconnect from device "HUB1"
Connect to ISP device
    use genie testbed "testbed_session2_slide25.yaml"
    connect to device "ISP1"
Verify number of BGP nei
   verify count "1" "bgp neighbors" on device "ISP1"
Disconnect from ISP device
    disconnect from device "ISP1"

The syntax for verifications can be checked at https://pubhub.devnetcloud.com/media/genie-feature-browser/docs/#/apis

One important gotcha is that there need to be two spaces between the name of the verification (for which you just remove the _ character for the name taken from the api website) and the args. Otherwise robot doesn’t know where the name ends and where args start.

I’ll be writing more about this but it’s clear this is an absolutely awesome tool.

tode@ubuntu:~/perfectscripts/generator/Session2$ robot second.robot
 Second                                                                        
 [ WARN ] Could not load the Datafile correctly                                
 Connect to HUB1 device                                                | PASS |
 Verify that number of bgp routes on HUB1 device is 0                  | PASS |
 Verify that Tunnels are UP on HUB1                                    | PASS |
 Disconnect from HUB1 device                                           | PASS |
 [ WARN ] Could not load the Datafile correctly                                
 Connect to ISP device                                                 | PASS |
 Verify number of BGP nei                                              | FAIL |
 Expected '1', but found '0'
 Disconnect from ISP device                                            | PASS |
 Second                                                                | FAIL |
 7 critical tests, 6 passed, 1 failed
 7 tests total, 6 passed, 1 failed
 Output:  /home/tode/perfectscripts/generator/Session2/output.xml
 Log:     /home/tode/perfectscripts/generator/Session2/log.html
 Report:  /home/tode/perfectscripts/generator/Session2/report.html

Genie parsers

Hello

One of many cool things about Genie is that it provides parsers, which take command output from Cisco and translate it into a dictionary that has easily accessible (programatically) data.
To do that, Genie uses really complex regex expressions to scrape data from the output and then puts the values it gets into a dictionary schema.Let’s have a look at an example of a Genie parser output that processes data from ‚show version’

software version is: 15.4
router has been up for: 4 minutes

I’ve achieved this with just a few lines:

shver = r1.parse("show version")
print('software version is: ' + shver['version']['version_short'])
print('router has been up for: ' + shver['version']['uptime'])

Another parser gets data from show ip eigrp neighbors

139.21.173.1 neighbor has been up for: 00:02:22

and the code is:

print(list(sheigrpnei['eigrp_instance']['1251']['vrf']['1251']['address_family']['ipv4']['eigrp_interface']['Tunnel1251']['eigrp_nbr'].keys())[0] + ' neighbor has been up for: ' + sheigrpnei['eigrp_instance']['1251']['vrf']['1251']['address_family']['ipv4']['eigrp_interface']['Tunnel1251']['eigrp_nbr']['139.21.173.1']['uptime'])

This may look a bit more complex but the first part of print gets a list (first element) of keys from the dictionary, and the second part of print gets the value of uptime from the dictionary. It’s just that in case of output of the command show ip eigrp vrf <vrf> neighbor, the schema prepared by the author is a bit longer because there are more values.

Unfortunately, not all parsers that I need are available out of the (Genie) box. Why? Because there is no box. This is not a commercial product so the project is community-driven. There are regular updates of the Genie package and more parsers are added but if you need a certain parser immediately, you have to write it yourself. Genie documentation is pretty comprehensive but I have found that the best way of doing something new is to copy existing working stuff and creatively mold it to obtain a new result.

For my daily work i need to be able to get data about crypto tunnels and there is no parser at this point for show crypto isakmp sa. Let’s create it then by looking at the output:

R1#show crypto isakmp sa
 IPv4 Crypto ISAKMP SA
 dst             src             state          conn-id status
 139.23.204.37   80.0.0.1        QM_IDLE           1001 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE (deleted)

This is the ”design phase” where you think about the end result to create a dictionary schema. What do you need in that final dictionary so that it meets your requirements? Because this was the first parser i’d ever created, I decided that I only needed two values: the destination and the state.

# Python
import re

# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Or, Optional

# parser utils
from genie.libs.parser.utils.common import Common

__all__ = ['ShowCryptoIsakmpSASchema']
class ShowCryptoIsakmpSASchema(MetaParser):
    '''
    Schema for:
    <show crypto isakmp sa>
    '''

    #* my first schema is very simple: we might rework it later.
    schema = {
        'dst': {
            Any(): {
                Optional('STATE'): str,
            }
        }
    }

I envisioned (in my mind’s eye 😀 ) my dictionary to look like this:

{'dst': {'139.23.204.37': {'STATE': 'QM_IDLE'}}}

Now that the schema is ready, i can proceed with creating the class that processes the output.

class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
    '''
    Parser for:
    <show crypto isakmp sa>
    '''

    cli_command = 'show crypto isakmp sa' 

    #* could add extra parameters if needed
    def cli(self,vrf = '', output = None):
        #if output is None:
           #out = self.device.execute(self.cli_command)
        #else:
           #out = self.cli_command
    #for testing i will prepare mock input
        out = '139.23.204.37   80.0.10.1       QM_IDLE           1001 ACTIVE'
        # initial variables
parsed_dict = {}

Now it’s time to change the brain gear because it is necessary to prepare a regex expression that will match the command output (mock input from the code above).

One hour later (and some extensive learning at testing at regex101.com)

p1 = re.compile(r'^(?P([0-9]{1,3}.){3}[0-9]{1,3})\s+(?P([0-9]{1,3}.){3}[0-9]{1,3})\s+(?P[\w\d-.]+)\s+(?P[0-9]+)\s+(?P[\w\d-.]+)$')

Now that your brain is at the right temperature, the rest is laughably easy. Let’s check if our regex matches any lines from the output (well, the only one we have):

for line in out.splitlines():
            line = line.strip()
        m = p1.match(line)


Then the logic is as follows, if the regex has found matches in the lines, let’s put the values from m into a special group dictionary. Finally, let’s assign values (or subdictionaries) from that groupdictionary into our final dictionary:

if m:
                groups = m.groupdict()
                #print('this is groups: ' + str(groups))

                dst = groups['dst']
                dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})

                state = groups['STATE']
                dst_dict.update({'STATE': state})
               
        return parsed_dict

Now the parser code is ready! Here’s the full code:

ode@ubuntu:~/showcryptoisakmp/tode/parser/ios$ cat showcryptoisakmp.py 
#******************************************************************************
#*                              Parser Template
#* ----------------------------------------------------------------------------
#* ABOUT THIS TEMPLATE - Please read
#*
#* - Any comments with "#*" in front of them (like this entire comment box) are
#*   for template clarifications only and should be removed from the final
#*   product.
#*
#* - Anything enclosed in <> must be replaced by the appropriate text for your
#*   application
#*
#* Support:
#*    pyats-support@cisco.com
#*
#* Description:
#*   This template file describes how to write a standard parser.
#*
#* Read More:
#*   For the complete and up-to-date user guide on parsers, visit:
#*   https://pubhub.devnetcloud.com/media/pyats-packages/docs/genie/cookbooks/index.html#genie-libs-parsers
#*   https://wiki.cisco.com/display/GENIE/Genie+Parsers+Deepdive
#*
#*   List of available Genie parsers:
#*   http://wwwin-pyats.cisco.com/cisco-shared/genie/latest/genie_libs/#/parsers
#*
#*******************************************************************************

#*******************************************************************************
#* DOCSTRINGS
#*
#*   All parsers should use the built-in Python docstrings functionality
#*   to define parser/class/method headers.
#*
#* Format:
#*   Docstring format should follow:
#*   URL= https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html
#*
#* Read More:
#*   Python Docstrings, PEP 257:
#*   URL= http://legacy.python.org/dev/peps/pep-0257/
#*******************************************************************************

'''
showcryptoisakmp.py

ios parsers for the following show commands:
    * <your commands here>

'''

# Python
import re

# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Or, Optional

# parser utils
from genie.libs.parser.utils.common import Common

# ==============================
# Schema for:
#    * '<show crypto isakmp sa>'
# ==============================

__all__ = ['ShowCryptoIsakmpSASchema']
class ShowCryptoIsakmpSASchema(MetaParser):
    '''
    Schema for:
    <show crypto isakmp sa>
    '''

    #* define your schema here
    schema = {
        'dst': {
            Any(): {
                Optional('STATE'): str,
            }
        }
    }

# ==============================
# Parser for:
#    * '<show crypto isakmp sa>'
# ==============================
class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
    '''
    Parser for:
    <show crypto isakmp sa>
    '''

    cli_command = 'show crypto isakmp sa' 

    #* could add extra parameters if needed
    def cli(self,vrf = '', output = None):
        #if output is None:
           #out = self.device.execute(self.cli_command)
        #else:
           #out = self.cli_command
        out = '139.23.204.37   80.0.10.1       QM_IDLE           1001 ACTIVE'
        # initial variables
        parsed_dict = {}

        p1 = re.compile(r'^(?P<dst>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<src>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<STATE>[\w\d\-\.]+)\s+(?P<CONNID>[0-9]+)\s+(?P<status>[\w\d\-\.]+)$')

        for line in out.splitlines():
            line = line.strip()
        m = p1.match(line)
        if m:
                groups = m.groupdict()
                #print('this is groups: ' + str(groups))

                dst = groups['dst']
                dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})

                state = groups['STATE']
                dst_dict.update({'STATE': state})
        #print(parsed_dict)        
        return parsed_dict    

However, our job is not fully done yet. Now we need to stop using mocked input and use real command output from the router.

class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
    '''
    Parser for:
    <show crypto isakmp sa>
    '''

    cli_command = 'show crypto isakmp sa' 

    #* could add extra parameters if needed
    def cli(self,vrf = '', output = None):
        if output is None:
           out = self.device.execute(self.cli_command)
        else:
           out = self.cli_command

        print('this is output i got from dictionary: ' + str(out))
        #out = '139.23.204.37   80.0.10.1       QM_IDLE           1001 ACTIVE'
        # initial variables
        parsed_dict = {}

        p1 = re.compile(r'^(?P<dst>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<src>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<STATE>[\w\d\-\.]+)\s+(?P<CONNID>[0-9]+)\s+(?P<status>[\w\d\-\.]+)$')

        for line in out.splitlines():
            line = line.strip()
            print('this is next line ' + line)
            m = p1.match(line)
            if m:
                groups = m.groupdict()
                print('this is groups: ' + str(groups))

                dst = groups['dst']
                dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})

                state = groups['STATE']
                dst_dict.update({'STATE': state})
        print(parsed_dict)        
        return parsed_dict    

Now the parser returns a dictionary, so the question is how to get specific values from that dictionary? let’s create a short script for that.

import json
from genie.conf import Genie
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
from genie.conf import Genie
from genie.conf.base import Device
from genie.conf.base import Testbed
from genie.conf.base.base import DeviceFeature
from genie.libs.conf.vrf import Vrf
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Optional
import tode

testbed = Genie.init('testbed.yml')
r1 = testbed.devices['R1']
r1.connect(log_stdout=False)

shcrypto = r1.parse("show crypto isakmp sa")
ipstring = str(list(shcrypto['dst'].keys())[0])
state = str(shcrypto['dst'][ipstring]['STATE'])
print('tunnel destination ' + ipstring + ' is in state ' + state )

ipstring = str(list(shcrypto['dst'].keys())[1])
state = str(shcrypto['dst'][ipstring]['STATE'])
print('tunnel destination ' + ipstring + ' is in state ' + state )

On my spoke routers i will never have more than two tunnels, so this should be enough, but of course it would be better to have a for loop to go through the whole length of the list rather than getting specific list positions. For now this will be enough, though.

Now let’s test this. For now the original parser includes a lot of ”debug” code that we don’t need but it’s included here for the sake of clarity:

this is output i got from dictionary: IPv4 Crypto ISAKMP SA
 dst             src             state          conn-id status
 139.23.204.37   80.0.0.1        QM_IDLE           1001 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE
 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE (deleted)
 IPv6 Crypto ISAKMP SA
 this is next line IPv4 Crypto ISAKMP SA
 this is next line dst             src             state          conn-id status
 this is next line 139.23.204.37   80.0.0.1        QM_IDLE           1001 ACTIVE
 this is groups: {'dst': '139.23.204.37', 'src': '80.0.0.1', 'STATE': 'QM_IDLE', 'CONNID': '1001', 'status': 'ACTIVE'}
 this is next line 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE
 this is groups: {'dst': '139.23.204.38', 'src': '80.0.0.1', 'STATE': 'MM_NO_STATE', 'CONNID': '0', 'status': 'ACTIVE'}
 this is next line 139.23.204.38   80.0.0.1        MM_NO_STATE          0 ACTIVE (deleted)
 this is next line 
 this is next line IPv6 Crypto ISAKMP SA
 {'dst': {'139.23.204.37': {'STATE': 'QM_IDLE'}, '139.23.204.38': {'STATE': 'MM_NO_STATE'}}}
 tunnel destination 139.23.204.37 is in state QM_IDLE
 tunnel destination 139.23.204.38 is in state MM_NO_STATE

What is really interesting is the last two lines. I have been successful in transforming string output to a dictionary and getting specific values from the dictionary.

#UPDATE 29.12.2020

I’ve actually spent some time writing (and rewriting) parsers, so I ended up having a nice script that returns the state of the tunnel just based on whatever I have in the description in the crypto map entry. Let’s say i have VPNs with my business partners and in the crypto map entry description i put their ID (e.g. PARTNER7923), i can do something like this:

myscript.py PARTNER7923

and the script returns the state of the Phase 1 (QM_IDLE).

This is however, not good enough. I want to be able to know the phase 2 SA’s to be able to tshoot some problems with specific sources and destinations (Do SA’s exist for the host pair? etc.)

I need a parser for show crypto session remote <peerIP> details, which is a bit tricky because the number of lines in the output is not known beforehand (depends on the number of SAs).

I ended up with the following regex:

(?P<PROFILE>(Profile.*))\s+(?P<UPTIME>(Uptime.*))\s+(?P<STATUS>(Session.status.*))\s+(?P<PEER>(Peer.*))\s+(?P<PeerD1>(.*))\s+(?P<PeerD2>(.*))\s+(?P<SESSIONID>(Session.*))\s+(?P<IPSEC>([\0-\377]*))

This matches on:

Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect, U - IKE Dynamic Route Update
S - SIP VPN

Interface: GigabitEthernet0/0/1.3138
Profile: PARTNERPROFILE7923
Uptime: 00:09:23
Session status: UP-ACTIVE
Peer: 100.0.0.1 port 500 fvrf: 3138 ivrf: 1251
      Phase1_id: 100.0.0.1
      Desc: (none)
  Session ID: 2450081
  IKEv2 SA: local 150.0.0.1/500 remote 100.0.0.1/500 Active
          Capabilities:F connid:24 lifetime:00:50:37
  IPSEC FLOW: permit ip 192.168.0.0/255.255.0.0 172.16.0.0/255.255.255.248
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 10 drop 0 life (KB/Sec) 4608000/2391
        Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4608000/2391

The next part is to create the actual parser (schema class and parser class)

Using Genielibs to create config

Hello

After reading the documentation, I realized that it is not necessary to create own methods (or j2 templates if you prefer the other way) to create the config. Instead, it is possible to use community-created classes, like vrf objects below.

For the first method I’m using a yaml config file with parameters, for the second method i’m injecting the parameters in the method ‚live’.

from genie.conf import Genie
from genie.conf.base import Device
from genie.conf.base import Testbed
from genie.conf.base.base import DeviceFeature
from genie.libs.conf.vrf import Vrf
from genie.libs.conf.static_routing.static_routing import StaticRouting
from pyats import topology
from genie.testbed import load
from genie.libs.conf.address_family import AddressFamily
from genie.libs.conf.bgp import RouteTarget



def Change17nov2020():
 
 with open('cisco_genie_cfg_loader_config_file.yml', 'rb') as f:
    conf = yaml.load(f.read())    
 
 r1 = load('testbed.yml').devices['R1']
 
 vrf1 = Vrf(name=conf['name'], description=conf['description'], rd=conf['rd'])
 
 r1.connect()
 r1.configure(str(vrf1.build_config(devices=[r1], apply=False)['R1']).split('\n'))

def Change18nov2020():
 r1 = load('testbed.yml').devices['R1']
 static_routing = StaticRouting()
 static_routing.device_attr[r1].vrf_attr['VRF1'].address_family_attr['ipv4'].route_attr['10.2.1.0/24'].next_hop_attr['192.168.1.2'].preference = 3

 
 r1.connect()
 r1.configure(str(static_routing.build_config(devices=[r1],apply=False['R1']).split('\n'))

Change17nov2020()
Change18nov2020()

Reverting the change is very easy, with the ‚unbuild_config’ method instead. Yes, reverting your change is as simple as adding two letters to the code.

Note that sometimes your out-of-the-box solution is not fully baked. You may occasionally need to ‚improve’ the genie libs a bit. In the example above, the vrf.py under ios-xe genie libraries didn’t have the address-family ipv4 command, which didn’t activate the routing table for the vrf. I ended up editing manually (under site packages) the vrf.py file to add the following line:

configurations.append_line(‚address-family ipv4 unicast’)

Alternatively (i think this is clearer), you can do this:

 def Change17nov2020():

   with open('cisco_genie_cfg_loader_config_file.yml', 'rb') as f:
     conf = yaml.load(f.read())    # load the config file
   testbed = load('testbed.yml')
   r1 = testbed.devices['R1']
   r1.connect()
   vrf1 = Vrf(name=conf['name'], description=conf['description'], rd=conf['rd'])
   r1.add_feature(vrf1)
   output = vrf1.build_config()

Putting it all together: OOB network programming, nested jinjas, decorators, slack, Genie

After a few months of configuring my test network using one-off scripts i have now realized that perhaps this was not entirely proper way of doing things.
In one of my previous blog posts i mentioned that configuring the router using that long change.txt wasn’t ”network as code” but rather network as a long messy text file. This was true, although it was a small step forward.
From now on I will try to combine all the things i’ve done before, that is Unicon, jinja templates but using the OOB approach and modifying the network using code, so that I have the object-oriented app logic in one file, change execution in another (importing the logic), jinja templates in another file (.j2) and jinja attributes in another file (.yml).


The endgoal will be to have the following workflow:
1. prepare the new config,

2. get current network configuration and state,

3. implement the config,

4. get new network configuration and state

5. get change notification with config delta in Slack.

6. Optional (start the workflow from Slack, the ”chatops” way)

First let’s create some basic logic.

class Router:

   def __init__(self, sship, sshuser, sshpass):
    self.config = []
    self.sship = sship
    self.sshuser = sshuser
    self.sshpass = sshpass

   def setIPaddress(self, intname, IP, mask):
    self.intname = intname
    self.ip = IP
    self.mask = mask
    commandlist = []
    commandlist.append(self.intname)
    commandlist.append('ip address ' + str(self.ip) + ' ' + str(self.mask))
    self.config.append(commandlist)
   
  

    def configure(self):
    	#todo

router = Router()
router.setIPaddress("int fa0/0","192.168.0.1","255.255.255.0")
print(router.config)

Now we have the OOB basic, I will take advantage of all the cool bits and pieces from the previous blog posts to create this full script including Unicon, jinja etc.

from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
import sys
import shutil
import diffios
import os
import time
import logging
from gtts import gTTS
import re

class Router:

   def __init__(self):
    self.config = '' 

   def setIPaddress(self, intname, IP, mask):
    self.intname = intname
    self.ip = IP
    self.mask = mask
    commandlist = []
    commandlist.append("interface " + str(self.intname))
    commandlist.append('ip address ' + str(self.ip) + ' ' + str(self.mask))
    self.config += commandlist

   def prepareACLconfig(self, datafile, template):
       self.datafile = datafile
       self.template = template
       accesslists = yaml.load(open(self.datafile), Loader=yaml.SafeLoader)
       env = Environment(loader = FileSystemLoader('.'), trim_blocks=True, autoescape=True)
       template = env.get_template(self.template)
       acl_config = template.render(data=accesslists)
       self.config += acl_config

   def configure(self):
    testbed = loader.load('testbed.yml')
    c = testbed.devices['R1']
    c.connect()
    output = c.configure(router.config)
    c.destroy()
    #todo
#and now let's test an actual config change with acl data in the .yml object and template in the .j2 object, with the GNS3 routers described in the testbed.yml object.
      
router = Router()
router.prepareACLconfig('vpnacl.yml', 'acl.j2')
router.configure()


while all the data files look like this:

tode@ubuntu:~/perfectscripts/generator$ cat vpnacl.yml
 accesslists:
 aclname: permit_www
  aclcommand: permit
  protocol: ip
  source: 192.168.122.100
  sourcewildcard: 0.0.0.0
  destination: 8.8.8.8
  destwildcard: 0.0.0.0  


 tode@ubuntu:~/perfectscripts/generator$ cat acl.j2
 {% for accesslist in data.accesslists %}
 ip access-list extended {{accesslist.aclname}}
 {{accesslist.aclcommand}} {{accesslist.protocol}} {{accesslist.source}} {{accesslist.sourcewildcard}} {{accesslist.destination}} {{accesslist.destwildcard}}
 {% endfor %}


 tode@ubuntu:~/perfectscripts/generator$ cat testbed.yml
 devices:
   R1:
     connections:
       cli:
         ip: 192.168.122.100
         port: 23 
         protocol: telnet
     credentials:
       default:
         password: cisco
         username: cisco
       enable:
         password: cisco
     os: ios
     type: ios

Decorators

Let’s add a simple function here that will do ‚show run’ before and after the change. To do that, i’ve implemented a simple decorator.

def diffdecorator(func):
    def wrapper(self):
        testbed = loader.load('testbed.yml')
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.bak', "w")
        f.write(output)
        f.close()
        c.destroy()
        func()
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.txt', "w")
        f.write(output)
        f.close()
        c.destroy()
        diff = diffios.Compare('R1.bak', 'R1.txt')
        deltaoutput = diff.delta()
        filename = 'R1_change_delta.bak'
        f = open(filename, "w")
        f.write(deltaoutput)
        f.close()
    return wrapper

   @diffdecorator
   def configure():
    testbed = loader.load('testbed.yml')
    c = testbed.devices['R1']
    c.connect()
    output = c.configure(router.config)
    c.destroy()
    

Nested Jinja template

Going further, i created an ospf.j2 file and a corresponding ospf.yml file. Now because the ospf can have multiple network statements, a nested jinja template was needed.

tode@ubuntu:~/perfectscripts/generator$ cat ospf.yml
ospf:
   - processnumber: 100
     routerid: 1.1.1.1
     network: 
       - address: 192.168.122.0 0.0.0.255 area 0
       - address: 1.1.1.1 0.0.0.0 area 0
tode@ubuntu:~/perfectscripts/generator$ cat ospf.j2
{% for element in data.ospf%}
  router ospf {{element.processnumber}}
  router-id {{element.routerid}}
  {% for entry in element.network%}
    network {{entry.address}}
  {% endfor %}
{% endfor %}


This has produced the following config:

tode@ubuntu:~/perfectscripts/generator$ python3 ng_oob.py 
   router ospf 100
   router-id 1.1.1.1
       network 192.168.122.0 0.0.0.255 area 0
       network 1.1.1.1 0.0.0.0 area 0

Next steps:

  • using pyats to check network state before change (another decorator)
  • decision about folder structure with .ymls, templates and script change history
  • how to implement idempotency (file-based, or code-based)
  • using Slack to trigger changes (reverse bot)

Adding Slack notification

Let’s rewrite the decorator again:

def diffdecorator(func):
    def wrapper(self):
        testbed = loader.load('testbed.yml')
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.bak', "w")
        f.write(output)
        f.close()
        c.destroy()
        func()
        c = testbed.devices['R1']
        c.connect(log_stdout=False)
        output = c.execute('show run')
        f = open('R1.txt', "w")
        f.write(output)
        f.close()
        c.destroy()
        diff = diffios.Compare('R1.bak', 'R1.txt')

        deltaoutput = diff.delta()
        p = get_notifier('slack')
        p.notify(webhook_url='https://hooks.slack.com/services/xxxxx/yyyyy/zzzzzz', message='config on this router has been changed!')
        p.notify(webhook_url='https://hooks.slack.com/services/xxxxx/yyyyy/zzzzzz', message=diff.delta())
        filename = 'R1_change_delta.bak'
        f = open(filename, "w")
        f.write(deltaoutput)
        f.close()
    return wrapper


Adding Genie magic

A while ago I was tasked with creating a report based on the uptime taken from output of ‚show version’ command plus software version. Preparing the regex was funny, and i was well proud of the way i did the ”screenscrubbing”, until I saw how you can do this easier.

The regex way for uptime doesn’t look that bad but what if Cisco changed the output of the show version command?

endresult = re.findall(r'control processor .*', result)

And here comes Genie. Initialising the testbed looks very similar to the unicon way ( Genie.init instead of loader.load )

     testbed = Genie.init('testbed.yml')
     r1 = testbed.devices['R1']
     r1.connect()
     shver = r1.parse("show version")
     
     print(shver['version']['version_short'])
     print(shver['version']['uptime'])

The output of this is:

12.4
 5 hours, 8 minutes

Now another thing we can do with Genie is compare the state of the network. This is different than a configuration diff. A Genie diff can for example show you which OSPF LSAs you received in between running two ”Genie learn commands”. Let me show you how it’s done:

genie learn ospf --testbed testbed.yml --device R1 --output learn
earning '['ospf']' on devices '['R1']'
 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.04s/it]
 +==============================================================================+
 | Genie Learn Summary for device R1                                            |
 +==============================================================================+
 |  Connected to R1                                                             |
 -   Log: learn/connection_R1.txt 
 Learnt feature 'ospf' 
 -  Ops structure:  learn/ospf_ios_R1_ops.txt 
 -  Device Console: learn/ospf_ios_R1_console.txt 
 ============================================================================== 

Then I run my python script which generates and implements the ospf config:

tode@ubuntu:~/perfectscripts/generator$ python3 ng_oob.py 
   router ospf 100
   router-id 1.1.1.1
       network 192.168.122.0 0.0.0.255 area 0
       network 1.1.1.1 0.0.0.0 area 0

(...) here i didn't include the unicon's screen output

Now i run the Genie learn again, this time i redirect the output to another folder:

tode@ubuntu:~/perfectscripts/generator$ genie diff learn learn2
 1it [00:00, 127.13it/s]
 +==============================================================================+
 | Genie Diff Summary between directories learn/ and learn2/                    |
 +==============================================================================+
 |  File: ospf_ios_R1_ops.txt                                                   |
 - Diff can be found at ./diff_ospf_ios_R1_ops.txt 
 

Inside the diff file is the following:

tode@ubuntu:~/perfectscripts/generator$ cat diff_ospf_ios_R1_ops.txt 
 --- learn/ospf_ios_R1_ops.txt
 +++ learn2/ospf_ios_R1_ops.txt
 +info: 
 + vrf: 
 + default: 
 + address_family: 
 ipv4: 
 instance: 
 100: 
 areas: 
 0.0.0.0: 
 area_id: 0.0.0.0
 area_type: normal
 database: 
 lsa_types: 
 1: 
 lsa_type: 1
 lsas: 
 1.1.1.1 1.1.1.1: 
 adv_router: 1.1.1.1
 lsa_id: 1.1.1.1
 ospfv2: 
 body: 
 router: 
 links: 
 1.1.1.1: 
 link_data: 255.255.255.255
 link_id: 1.1.1.1
 topologies: 
 0: 
 metric: 1
 mt_id: 0
 type: stub network
 192.168.122.0: 
 link_data: 255.255.255.0
 link_id: 192.168.122.0
 topologies: 
 0: 
 metric: 1
 mt_id: 0
 type: stub network
 num_of_links: 2
 header: 
 adv_router: 1.1.1.1
 age: 42
 checksum: 0xE54B
 length: 48
 lsa_id: 1.1.1.1
 option: None
 seq_num: 80000001
 type: 1
 interfaces: 
 FastEthernet0/0: 
 bfd: 
 enable: False
 cost: 1
 dead_interval: 40
 demand_circuit: False
 dr_ip_addr: 192.168.122.100
 dr_router_id: 1.1.1.1
 enable: True
 hello_interval: 10
 hello_timer: 00:00:03
 interface_type: broadcast
 lls: True
 name: FastEthernet0/0
 passive: False
 priority: 1
 retransmit_interval: 5
 state: dr
 transmit_delay: 1
 Loopback1: 
 bfd: 
 enable: False
 cost: 1
 demand_circuit: False
 enable: True
 interface_type: loopback
 name: Loopback1
 mpls: 
 te: 
 enable: False
 statistics: 
 area_scope_lsa_cksum_sum: 0x00E54B
 area_scope_lsa_count: 1
 spf_runs_count: 1
 bfd: 
 enable: False
 graceful_restart: 
 cisco: 
 enable: False
 type: cisco
 ietf: 
 enable: False
 type: ietf
 mpls: 
 ldp: 
 autoconfig: False
 autoconfig_area_id: 0.0.0.0
 nsr: 
 enable: False
 preference: 
 single_value: 
 all: 110
 router_id: 1.1.1.1
 spf_control: 
 paths: 4
 throttle: 
 spf: 
 hold: 10000
 maximum: 10000
 start: 5000
 stub_router: 
 always: 
 always: False
 external_lsa: False
 include_stub: False
 summary_lsa: False 

What i’m going to do is include this learn process before and after each change, on top of the diffios change, so that we can have more proof that our change has been significant.

However, we can use this Genie diff also if no change has taken place, or if the change has been external to our administrative domain. Something breaks down even if we didn’t change anything, so how about running a Genie diff to check if we now have fewer ospf routes than before? or fewer bgp routes? It’s never been easier to troubleshoot a network problem.

…to be continued…