Hello
One of many cool things about Genie is that it provides parsers, which take command output from Cisco and translate it into a dictionary that has easily accessible (programatically) data.
To do that, Genie uses really complex regex expressions to scrape data from the output and then puts the values it gets into a dictionary schema.Let’s have a look at an example of a Genie parser output that processes data from ‚show version’
software version is: 15.4
router has been up for: 4 minutes
I’ve achieved this with just a few lines:
shver = r1.parse("show version")
print('software version is: ' + shver['version']['version_short'])
print('router has been up for: ' + shver['version']['uptime'])
Another parser gets data from show ip eigrp neighbors
139.21.173.1 neighbor has been up for: 00:02:22
and the code is:
print(list(sheigrpnei['eigrp_instance']['1251']['vrf']['1251']['address_family']['ipv4']['eigrp_interface']['Tunnel1251']['eigrp_nbr'].keys())[0] + ' neighbor has been up for: ' + sheigrpnei['eigrp_instance']['1251']['vrf']['1251']['address_family']['ipv4']['eigrp_interface']['Tunnel1251']['eigrp_nbr']['139.21.173.1']['uptime'])
This may look a bit more complex but the first part of print gets a list (first element) of keys from the dictionary, and the second part of print gets the value of uptime from the dictionary. It’s just that in case of output of the command show ip eigrp vrf <vrf> neighbor, the schema prepared by the author is a bit longer because there are more values.
Unfortunately, not all parsers that I need are available out of the (Genie) box. Why? Because there is no box. This is not a commercial product so the project is community-driven. There are regular updates of the Genie package and more parsers are added but if you need a certain parser immediately, you have to write it yourself. Genie documentation is pretty comprehensive but I have found that the best way of doing something new is to copy existing working stuff and creatively mold it to obtain a new result.
For my daily work i need to be able to get data about crypto tunnels and there is no parser at this point for show crypto isakmp sa. Let’s create it then by looking at the output:
R1#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
139.23.204.37 80.0.0.1 QM_IDLE 1001 ACTIVE
139.23.204.38 80.0.0.1 MM_NO_STATE 0 ACTIVE
139.23.204.38 80.0.0.1 MM_NO_STATE 0 ACTIVE (deleted)
This is the ”design phase” where you think about the end result to create a dictionary schema. What do you need in that final dictionary so that it meets your requirements? Because this was the first parser i’d ever created, I decided that I only needed two values: the destination and the state.
# Python
import re
# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Or, Optional
# parser utils
from genie.libs.parser.utils.common import Common
__all__ = ['ShowCryptoIsakmpSASchema']
class ShowCryptoIsakmpSASchema(MetaParser):
'''
Schema for:
<show crypto isakmp sa>
'''
#* my first schema is very simple: we might rework it later.
schema = {
'dst': {
Any(): {
Optional('STATE'): str,
}
}
}
I envisioned (in my mind’s eye 😀 ) my dictionary to look like this:
{'dst': {'139.23.204.37': {'STATE': 'QM_IDLE'}}}
Now that the schema is ready, i can proceed with creating the class that processes the output.
class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
'''
Parser for:
<show crypto isakmp sa>
'''
cli_command = 'show crypto isakmp sa'
#* could add extra parameters if needed
def cli(self,vrf = '', output = None):
#if output is None:
#out = self.device.execute(self.cli_command)
#else:
#out = self.cli_command
#for testing i will prepare mock input
out = '139.23.204.37 80.0.10.1 QM_IDLE 1001 ACTIVE'
# initial variables
parsed_dict = {}
Now it’s time to change the brain gear because it is necessary to prepare a regex expression that will match the command output (mock input from the code above).
One hour later (and some extensive learning at testing at regex101.com)
p1 = re.compile(r'^(?P([0-9]{1,3}.){3}[0-9]{1,3})\s+(?P([0-9]{1,3}.){3}[0-9]{1,3})\s+(?P[\w\d-.]+)\s+(?P[0-9]+)\s+(?P[\w\d-.]+)$')
Now that your brain is at the right temperature, the rest is laughably easy. Let’s check if our regex matches any lines from the output (well, the only one we have):
for line in out.splitlines():
line = line.strip()
m = p1.match(line)
Then the logic is as follows, if the regex has found matches in the lines, let’s put the values from m into a special group dictionary. Finally, let’s assign values (or subdictionaries) from that groupdictionary into our final dictionary:
if m:
groups = m.groupdict()
#print('this is groups: ' + str(groups))
dst = groups['dst']
dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})
state = groups['STATE']
dst_dict.update({'STATE': state})
return parsed_dict
Now the parser code is ready! Here’s the full code:
ode@ubuntu:~/showcryptoisakmp/tode/parser/ios$ cat showcryptoisakmp.py
#******************************************************************************
#* Parser Template
#* ----------------------------------------------------------------------------
#* ABOUT THIS TEMPLATE - Please read
#*
#* - Any comments with "#*" in front of them (like this entire comment box) are
#* for template clarifications only and should be removed from the final
#* product.
#*
#* - Anything enclosed in <> must be replaced by the appropriate text for your
#* application
#*
#* Support:
#* pyats-support@cisco.com
#*
#* Description:
#* This template file describes how to write a standard parser.
#*
#* Read More:
#* For the complete and up-to-date user guide on parsers, visit:
#* https://pubhub.devnetcloud.com/media/pyats-packages/docs/genie/cookbooks/index.html#genie-libs-parsers
#* https://wiki.cisco.com/display/GENIE/Genie+Parsers+Deepdive
#*
#* List of available Genie parsers:
#* http://wwwin-pyats.cisco.com/cisco-shared/genie/latest/genie_libs/#/parsers
#*
#*******************************************************************************
#*******************************************************************************
#* DOCSTRINGS
#*
#* All parsers should use the built-in Python docstrings functionality
#* to define parser/class/method headers.
#*
#* Format:
#* Docstring format should follow:
#* URL= https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html
#*
#* Read More:
#* Python Docstrings, PEP 257:
#* URL= http://legacy.python.org/dev/peps/pep-0257/
#*******************************************************************************
'''
showcryptoisakmp.py
ios parsers for the following show commands:
* <your commands here>
'''
# Python
import re
# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Or, Optional
# parser utils
from genie.libs.parser.utils.common import Common
# ==============================
# Schema for:
# * '<show crypto isakmp sa>'
# ==============================
__all__ = ['ShowCryptoIsakmpSASchema']
class ShowCryptoIsakmpSASchema(MetaParser):
'''
Schema for:
<show crypto isakmp sa>
'''
#* define your schema here
schema = {
'dst': {
Any(): {
Optional('STATE'): str,
}
}
}
# ==============================
# Parser for:
# * '<show crypto isakmp sa>'
# ==============================
class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
'''
Parser for:
<show crypto isakmp sa>
'''
cli_command = 'show crypto isakmp sa'
#* could add extra parameters if needed
def cli(self,vrf = '', output = None):
#if output is None:
#out = self.device.execute(self.cli_command)
#else:
#out = self.cli_command
out = '139.23.204.37 80.0.10.1 QM_IDLE 1001 ACTIVE'
# initial variables
parsed_dict = {}
p1 = re.compile(r'^(?P<dst>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<src>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<STATE>[\w\d\-\.]+)\s+(?P<CONNID>[0-9]+)\s+(?P<status>[\w\d\-\.]+)$')
for line in out.splitlines():
line = line.strip()
m = p1.match(line)
if m:
groups = m.groupdict()
#print('this is groups: ' + str(groups))
dst = groups['dst']
dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})
state = groups['STATE']
dst_dict.update({'STATE': state})
#print(parsed_dict)
return parsed_dict
However, our job is not fully done yet. Now we need to stop using mocked input and use real command output from the router.
class ShowCryptoIsakmpSA(ShowCryptoIsakmpSASchema):
'''
Parser for:
<show crypto isakmp sa>
'''
cli_command = 'show crypto isakmp sa'
#* could add extra parameters if needed
def cli(self,vrf = '', output = None):
if output is None:
out = self.device.execute(self.cli_command)
else:
out = self.cli_command
print('this is output i got from dictionary: ' + str(out))
#out = '139.23.204.37 80.0.10.1 QM_IDLE 1001 ACTIVE'
# initial variables
parsed_dict = {}
p1 = re.compile(r'^(?P<dst>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<src>([0-9]{1,3}\.){3}[0-9]{1,3})\s+(?P<STATE>[\w\d\-\.]+)\s+(?P<CONNID>[0-9]+)\s+(?P<status>[\w\d\-\.]+)$')
for line in out.splitlines():
line = line.strip()
print('this is next line ' + line)
m = p1.match(line)
if m:
groups = m.groupdict()
print('this is groups: ' + str(groups))
dst = groups['dst']
dst_dict = parsed_dict.setdefault('dst', {}).setdefault(dst, {})
state = groups['STATE']
dst_dict.update({'STATE': state})
print(parsed_dict)
return parsed_dict
Now the parser returns a dictionary, so the question is how to get specific values from that dictionary? let’s create a short script for that.
import json
from genie.conf import Genie
from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader
from genie.conf import Genie
from genie.conf.base import Device
from genie.conf.base import Testbed
from genie.conf.base.base import DeviceFeature
from genie.libs.conf.vrf import Vrf
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Optional
import tode
testbed = Genie.init('testbed.yml')
r1 = testbed.devices['R1']
r1.connect(log_stdout=False)
shcrypto = r1.parse("show crypto isakmp sa")
ipstring = str(list(shcrypto['dst'].keys())[0])
state = str(shcrypto['dst'][ipstring]['STATE'])
print('tunnel destination ' + ipstring + ' is in state ' + state )
ipstring = str(list(shcrypto['dst'].keys())[1])
state = str(shcrypto['dst'][ipstring]['STATE'])
print('tunnel destination ' + ipstring + ' is in state ' + state )
On my spoke routers i will never have more than two tunnels, so this should be enough, but of course it would be better to have a for loop to go through the whole length of the list rather than getting specific list positions. For now this will be enough, though.
Now let’s test this. For now the original parser includes a lot of ”debug” code that we don’t need but it’s included here for the sake of clarity:
this is output i got from dictionary: IPv4 Crypto ISAKMP SA
dst src state conn-id status
139.23.204.37 80.0.0.1 QM_IDLE 1001 ACTIVE
139.23.204.38 80.0.0.1 MM_NO_STATE 0 ACTIVE
139.23.204.38 80.0.0.1 MM_NO_STATE 0 ACTIVE (deleted)
IPv6 Crypto ISAKMP SA
this is next line IPv4 Crypto ISAKMP SA
this is next line dst src state conn-id status
this is next line 139.23.204.37 80.0.0.1 QM_IDLE 1001 ACTIVE
this is groups: {'dst': '139.23.204.37', 'src': '80.0.0.1', 'STATE': 'QM_IDLE', 'CONNID': '1001', 'status': 'ACTIVE'}
this is next line 139.23.204.38 80.0.0.1 MM_NO_STATE 0 ACTIVE
this is groups: {'dst': '139.23.204.38', 'src': '80.0.0.1', 'STATE': 'MM_NO_STATE', 'CONNID': '0', 'status': 'ACTIVE'}
this is next line 139.23.204.38 80.0.0.1 MM_NO_STATE 0 ACTIVE (deleted)
this is next line
this is next line IPv6 Crypto ISAKMP SA
{'dst': {'139.23.204.37': {'STATE': 'QM_IDLE'}, '139.23.204.38': {'STATE': 'MM_NO_STATE'}}}
tunnel destination 139.23.204.37 is in state QM_IDLE
tunnel destination 139.23.204.38 is in state MM_NO_STATE
What is really interesting is the last two lines. I have been successful in transforming string output to a dictionary and getting specific values from the dictionary.
#UPDATE 29.12.2020
I’ve actually spent some time writing (and rewriting) parsers, so I ended up having a nice script that returns the state of the tunnel just based on whatever I have in the description in the crypto map entry. Let’s say i have VPNs with my business partners and in the crypto map entry description i put their ID (e.g. PARTNER7923), i can do something like this:
myscript.py PARTNER7923
and the script returns the state of the Phase 1 (QM_IDLE).
This is however, not good enough. I want to be able to know the phase 2 SA’s to be able to tshoot some problems with specific sources and destinations (Do SA’s exist for the host pair? etc.)
I need a parser for show crypto session remote <peerIP> details, which is a bit tricky because the number of lines in the output is not known beforehand (depends on the number of SAs).
I ended up with the following regex:
(?P<PROFILE>(Profile.*))\s+(?P<UPTIME>(Uptime.*))\s+(?P<STATUS>(Session.status.*))\s+(?P<PEER>(Peer.*))\s+(?P<PeerD1>(.*))\s+(?P<PeerD2>(.*))\s+(?P<SESSIONID>(Session.*))\s+(?P<IPSEC>([\0-\377]*))
This matches on:
Crypto session current status
Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect, U - IKE Dynamic Route Update
S - SIP VPN
Interface: GigabitEthernet0/0/1.3138
Profile: PARTNERPROFILE7923
Uptime: 00:09:23
Session status: UP-ACTIVE
Peer: 100.0.0.1 port 500 fvrf: 3138 ivrf: 1251
Phase1_id: 100.0.0.1
Desc: (none)
Session ID: 2450081
IKEv2 SA: local 150.0.0.1/500 remote 100.0.0.1/500 Active
Capabilities:F connid:24 lifetime:00:50:37
IPSEC FLOW: permit ip 192.168.0.0/255.255.0.0 172.16.0.0/255.255.255.248
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 10 drop 0 life (KB/Sec) 4608000/2391
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4608000/2391
The next part is to create the actual parser (schema class and parser class)
