Customising Ansible transport plugins to traverse custom jumpservers PART 1

Hi

The problem to solve here is as follows: Ansible only supports standard SSH proxies, but it doesn’t support custom jump servers. In my case, i ssh to the jump server but later i need to type in a command ‚connect mytargetrouter<ENTER>’ .
Let’s see how (and if) this can be solved.

Normally I document final results only. Today, however, i’d like to show that sometimes the path to the result can be equally interesting and that a failure can lead (eventually, after days of thinking and research) to good conclusions.

After I managed to set up Nornir to connect to my gns3 routers (one of the previous blog entries), I wanted to try and see if it is also capable of traversing a non-ssh proxy (in my environment i connect to a proxy where i write ”connect MYROUTER<ENTER>” and after up to 30 seconds i get to myrouter).
The way I approached it was completely wrong, but I only understood this after I’d run the script.

from nornir import InitNornir
from nornir.plugins.tasks.networking import napalm_get
from nornir.plugins.functions.text import print_result
import json
from nornir.plugins.tasks.networking import napalm_configure
from nornir.plugins.tasks import networking
from nornir.plugins.tasks.networking import netmiko_send_command
import sys
from netmiko import ConnectHandler
import time
from netmiko import redispatch
from netmiko import Netmiko
from jinja2 import Environment, FileSystemLoader
import yaml

def task_manages_connection_manually(task, router):
  task.host.open_connection("netmiko", configuration=task.nornir.config)
  p = task.run(
  task=netmiko_send_command, command_string ='connect ' + router + ' \n', use_timing=True)

  r = task.run(
  task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True) 
  
  sr = task.run(
  task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True)
  
  tr = task.run(task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True) 
  
  ur = task.run(task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True)

  task.host.close_connection("netmiko")


nr = InitNornir(config_file="./config.yaml", logging={"file": "mylogs", "level": "debug"} )
task_result = nr.run(task=task_manages_connection_manually, router='mytestrouter')
print_result(task_result)

The problem here was the unknown and everchanging delay that I experience on my jumpserver. By playing a bit with the use_timing variable I was able to get this to work but only if I inserted some fake tasks (tr, ur above .) that take some time to execute. I don’t care whether they will really execute. They are here just to give us time.

So now nornir was able to connect to my target router and get the output of show ip interface brief command (sometimes even twice, which is generally not desirable when you run any scripts) and for 10 seconds I was happy but then I realized that i’d accomplished nothing.  My hosts.yaml only had the proxy, not the target router. Even though i will be able to move the args to init method, nornir now gives me totally no advantage over netmiko. It thinks it connects to one target host only (proxy) and is not aware of the real target. I can no longer use any filtering, groups etc etc. This is obviously a bad idea. A few hours wasted.

I had to come up with something better. I gave up on Nornir for the time being and decided to be bold. If Ansible cannot use a custom (i.e. not a supported ssh bastion) proxy by default, why not customise the connection plugins?
I had a look at the network_cli.py of Ansible at the following path:

tode@ubuntu:/usr/lib/python3/dist-packages/ansible/plugins/connection$

Because network_cli uses paramiko, i then looked at the paramiko_ssh.py, where I found this connect_uncached def:

def _connect_uncached(self):
''' activates the connection object '''

if paramiko is None:
raise AnsibleError("paramiko is not installed: %s" % to_native(PARAMIKO_IMPORT_ERR))

port = self._play_context.port or 22
display.vvv("ESTABLISH PARAMIKO SSH CONNECTION FOR USER: %s on PORT %s TO %s" % (self._play_context.remote_user, port, self._play_context.remote_addr),
host=self._play_context.remote_addr)

ssh = paramiko.SSHClient()

# override paramiko's default logger name
if self._log_channel is not None:
ssh.set_log_channel(self._log_channel)

self.keyfile = os.path.expanduser("~/.ssh/known_hosts")

if self.get_option('host_key_checking'):
for ssh_known_hosts in ("/etc/ssh/ssh_known_hosts", "/etc/openssh/ssh_known_hosts"):
try:
# TODO: check if we need to look at several possible locations, possible for loop
ssh.load_system_host_keys(ssh_known_hosts)
break
except IOError:
pass # file was not found, but not required to function
ssh.load_system_host_keys()

ssh_connect_kwargs = self._parse_proxy_command(port)


ssh.set_missing_host_key_policy(MyAddPolicy(self._new_stdin, self))

allow_agent = True

if self._play_context.password is not None:
allow_agent = False

try:
key_filename = None
if self._play_context.private_key_file:
key_filename = os.path.expanduser(self._play_context.private_key_file)

# paramiko 2.2 introduced auth_timeout parameter
if LooseVersion(paramiko.__version__) >= LooseVersion('2.2.0'):
ssh_connect_kwargs['auth_timeout'] = self._play_context.timeout

ssh.connect(
self._play_context.remote_addr.lower(),
username=self._play_context.remote_user,
allow_agent=allow_agent,
look_for_keys=self.get_option('look_for_keys'),
key_filename=key_filename,
password=self._play_context.password,
timeout=self._play_context.timeout,
port=port,
**ssh_connect_kwargs
)

Now the plan is as follows: to remove the self._play_context values from ssh.connect and instead hardcode my proxy user and password. Finally, I will send a connect command in the ssh channel: ssh.exec_command(‚connect MYROUTER’). This seems easy enough.

After making the changes in the paramiko_ssh.py I was able to get to my jump host and I could see that Ansible got to the jump host prompt but the ssh.exec_command(cmd) to go to my target router didn’t seem to have any effect. In the end ansible closed the channel. A step back, then. I needed to create a manual paramiko script and test it in GNS3 to see what is happening.

Let’s create a simple Paramiko script and start a router in GNS3

import paramiko
import time
s = paramiko.SSHClient()
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
s.connect(hostname='192.168.122.100', username='cisco', password='cisco', port='22')
stdin, stdout, stderr = s.exec_command('show ip interface brief')
time.sleep(2)
stdout=stdout.readlines()
output = ""
for line in stdout:
output=output+line
print(output)
s.close()

BTW without the workaround with time.sleep() you will encounter a bug:

Traceback (most recent call last):
File "/home/tode/.local/lib/python3.8/site-packages/paramiko/file.py", line 66, in __del__
File "/home/tode/.local/lib/python3.8/site-packages/paramiko/channel.py", line 1392, in close
File "/home/tode/.local/lib/python3.8/site-packages/paramiko/channel.py", line 991, in shutdown_write
File "/home/tode/.local/lib/python3.8/site-packages/paramiko/channel.py", line 967, in shutdown
File "/home/tode/.local/lib/python3.8/site-packages/paramiko/transport.py", line 1846, in _send_user_message
AttributeError: 'NoneType' object has no attribute 'time'

With time.sleep(2) it runs just fine:

tode@ubuntu:~/paramikoscript$ python3 simple.py

Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 192.168.122.100 YES NVRAM up up 
FastEthernet1/0 unassigned YES NVRAM administratively down down 
FastEthernet2/0 unassigned YES NVRAM administratively down down 
Loopback2001 200.200.200.200 YES NVRAM up up 
Loopback2002 200.200.200.201 YES NVRAM up up

 

ok so now back to the ssh paramiko file in ansible plugins:

ssh.connect(
'1.1.1.1',
username='cisco',
allow_agent=allow_agent,
look_for_keys=self.get_option('look_for_keys'),
key_filename=key_filename,
password='cisco',
timeout=60,
port=722,
**ssh_connect_kwargs
)
ssh.exec_command('connect testrouter\r\n')
time.sleep(10)

 

Let’s test this now… However, this still fails (channel closed error). I nearly got discouraged and stopped working on this for a day because I had no idea why the command didn’t work. It turned out that the reason is that my jump server doesn’t support Paramiko’s exec_command. Looks like I may have to use invoke_shell() . Let’s test this invoke_shell thing again in GNS3

import logging
import paramiko
import time

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("paramiko").setLevel(logging.DEBUG)
paramiko.util.log_to_file("paramiko.log")
s = paramiko.SSHClient()
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
s.connect(hostname='192.168.122.100', username='cisco', password='cisco', port='22')
chan = s.invoke_shell()
chan.send(' show ip interface brief | redirect disk0:showip.txt\n')
time.sleep(2)
s.close()

Running the script…

tode@ubuntu:~/paramikoscript$ python3 simple.py 
DEBUG:paramiko.transport:starting thread (client mode): 0x38429df0
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.7.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-1.99-Cisco-1.25
INFO:paramiko.transport:Connected (version 1.99, client Cisco-1.25)
DEBUG:paramiko.transport:kex algos:['diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['aes128-cbc', '3des-cbc', 'aes192-cbc', 'aes256-cbc'] server encrypt:['aes128-cbc', '3des-cbc', 'aes192-cbc', 'aes256-cbc'] client mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-md5', 'hmac-md5-96'] server mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-md5', 'hmac-md5-96'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: diffie-hellman-group-exchange-sha1
DEBUG:paramiko.transport:HostKey agreed: ssh-rsa
DEBUG:paramiko.transport:Cipher agreed: aes128-cbc
DEBUG:paramiko.transport:MAC agreed: hmac-sha1
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:Got server p (2048 bits)
DEBUG:paramiko.transport:kex engine KexGex specified hash_algo <built-in function openssl_sha1>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-rsa host key for [192.168.122.100]:22: b'd59685e80107be2bdfcc1686a83fc22e'
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (password) successful!
DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
DEBUG:paramiko.transport:[chan 0] Max packet out: 4096 bytes
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok

The result is a new file on disk0 of my testrouter with the output of show ip int brief. This is good. So maybe now I can implement this in the ansible plugin.
This is the full scope of the changes in the paramiko ansible plugin. I’m using 1.1.1.1 with port 722 as my proxy and testrouter as my target router. The actual IP of the testrouter is not important (=it doesn’t have to be defined in the hosts.yaml or any inventory) because the jumpserver is fully aware of all host mappings.

ssh.connect(
'1.1.1.1',
username='cisco',
allow_agent=allow_agent,
look_for_keys=self.get_option('look_for_keys'),
key_filename=key_filename,
password='cisco',
timeout=60,
port=722,
**ssh_connect_kwargs
)
channel = ssh.invoke_shell()
channel.send('connect testrouter\n')
time.sleep(20)
channel.send('show ip interface brief | redirect bootflash:ship.txt\n')

Just another look at the playbook that I will run again:

- name: show ip interface brief
gather_facts: false
hosts: testrouter 
tasks:
- name: "get ship output"
ios_command: 
commands: "show ip interface brief"
register: cli_result

However, when I tested the playbook I got a FATAL error saying that Ansible cannot elevate the privileges on the jump server (it tries to do that after the ssh_connect)
This is because Ansible cannot run the BECOME method on the jump host, so I disabled (in my group_vars) the ansible_become variable with: ansible_become : no

Now I was able to use the playbook!

tode@ubuntu:~/ansiblefolder$ ansible-playbook showipintbrief.yml -vvv
ansible-playbook 2.9.6
config file = /home/tode/ansiblefolder/ansible.cfg
configured module search path = ['/home/tode/.local/lib/python3.8/site-packages/napalm_ansible/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 3.8.2 (default, Jul 16 2020, 14:00:26) [GCC 9.3.0]
Using /home/tode/ansiblefolder/ansible.cfg as config file
host_list declined parsing /home/tode/ansiblefolder/inventory.yml as it did not pass its verify_file() method
script declined parsing /home/tode/ansiblefolder/inventory.yml as it did not pass its verify_file() method
Parsed /home/tode/ansiblefolder/inventory.yml inventory source with ini plugin

PLAYBOOK: showipintbrief.yml *******************************************************************************************************************************************
1 plays in showipintbrief.yml

PLAY [show ip interface brief] *****************************************************************************************************************************************
META: ran handlers

TASK [get ship output] *************************************************************************************************************************************************
task path: /home/tode/ansiblefolder/showipintbrief.yml:5
<192.168.122.100> ESTABLISH LOCAL CONNECTION FOR USER: tode
<192.168.122.100> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990 `" && echo ansible-tmp-1598528447.646881-107080737393990="` echo /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/network/ios/ios_command.py
<192.168.122.100> PUT /home/tode/.ansible/tmp/ansible-local-6990397lydnf/tmppwshwp3x TO /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/AnsiballZ_ios_command.py
<192.168.122.100> EXEC /bin/sh -c 'chmod u+x /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/ /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/AnsiballZ_ios_command.py && sleep 0'
<192.168.122.100> EXEC /bin/sh -c '/usr/bin/python3 /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/AnsiballZ_ios_command.py && sleep 0'
<192.168.122.100> EXEC /bin/sh -c 'rm -f -r /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/ > /dev/null 2>&1 && sleep 0'
ok: [testrouter] => {
"changed": false,
"invocation": {
"module_args": {
"auth_pass": null,
"authorize": null,
"commands": [
"show ip interface brief"
],
"host": null,
"interval": 1,
"match": "all",
"password": null,
"port": null,
"provider": null,
"retries": 10,
"ssh_keyfile": null,
"timeout": null,
"username": null,
"wait_for": null
}
},
"stdout": [
"404 Command Not Found"
],
"stdout_lines": [
[
"404 Command Not Found"
]
]
}
META: ran handlers
META: ran handlers

PLAY RECAP *************************************************************************************************************************************************************
testrouter : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

and sure enough, the txt file with the output of show ip interface brief appeared on the bootflash of my target router. Mission accomplished.

But what exactly happened here? The ansible debug stdout shows ”404 command not found”. This means that the command from the playbook was issued on the jump host instead of the target router. The only reason ansible succeeded was because the only task was to issue the show command and register the result.
What does this mean? It means there is another python file to modify: somewhere where Ansible opens the channel within the ssh connection. This file will need to be aware of the playbook’s variables like the router names etc. Unfortunately, the actual SSH connection is only to the jump host and this ssh connection is returned in paramiko_ssh.py. Modifying Ansible is far from done at this point.

Good news is that Ansible managed to actually issue the command on the jumphost and got some output so opening a channel is done somewhere correctly irrespective of the operating system on the host. In theory, an alternative to channel.py modification then exists: we could even make a playbook where the first task would be to connect to the ”inventory host”. But i’m not sure i like this workaround.

to be continued…

 

 

 

 

 

 

 

 

ChatOps – Ansible gets status from router, notifies Slack channel

Hello

Just a teaser of the idea that i’m toying with while i’m still working hard on the automation post. The idea of chatops is that your automation scripts should send status notifications (or any other notifications) to a common workspace, such as Slack. I would love to see daily statuses of my routers in my slack channel:
R1: ok
R2: ok
R3: nok

The other direction is also possible (oh the chaos an attacker could wreak upon the network with this one) : it is possible to actually run scripts from Slack.

For now i’ve just registered my slack account, created an app, and ran a test curl command from my VM to the slack channel. I’ll try to built a slack notification into my ansible scripts now.

chatops

Obviously I couldn’t leave this at this unfinished stage 😀 I took my ansible playbook and modified it:

- name: add_entry_to_acl 
  hosts: testrouter
  tasks:
   - name: add_new_entry
     ios_config:
     lines:
     - "{{ acl20 }}" 
     parents: ip access-list extended permit_www
     before: ip access-list extended permit_www 
     save_when: modified
   - name: send notification to Slack
     local_action:
     module: slack
     token: <here enter your slack webhook token>
     channel: "#things"
     msg: "Name of the host is {{ ansible_net_hostname }} and the software version is {{ ansible_net_version }} while the platform is {{ ansible_net_model }}"

 

tode@ubuntu:~/ansiblefolder$ ansible-playbook aclplaybook.yml

PLAY [add_entry_to_acl] ********************************************************

TASK [Gathering Facts] *********************************************************
[WARNING]: Ignoring timeout(10) for ios_facts
[WARNING]: default value for `gather_subset` will be changed to `min` from
`!config` v2.11 onwards
ok: [testrouter]

TASK [add_new_entry] ***********************************************************
changed: [testrouter]

TASK [show clock] **************************************************************
ok: [testrouter]

TASK [send notification to Slack] **********************************************
ok: [testrouter -> localhost]

PLAY RECAP *********************************************************************
testrouter : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

 

and voila:

final_chat

 

Cisco IOS-XE ikev2 Denial of Service vulnerability, June 2020, fixed in 16.9.4

Hello

This may not exactly be breaking news but just to let you know that this ikev2 CVE has been out there since June. Unfortunately, there is no workaround. Even if you implement your crypto call control with max SAs, it simply means that this attack will fill up the maximum number of SA’s. I guess it’s time to upgrade your routers again…

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ikev2-9p23Jj2a

 

IPsec preferred peer + reverse route static

Hello

A bit of a pause with automation this week. I’ve been assigned a task to set up a redundant crypto-based VPN, a task which is in fact quite easy but for some weird reason Cisco documentation doesn’t describe how to do this in the peer preferred feature.
The problem is here that you can’t set up static routes with a track object because there’s nothing to link the track object to if you don’t have logical tunnel interfaces. Of course, there’s always EEM scripts that could activate routes if it sees ”tunnel up/down” events in the syslog but… let’s be serious. Besides, I was curious to see how fast this redundancy works.
vpn_redundancy

R1 has 1.1.1.1, both R2 and R3 have 2.2.2.2 as their loopback to simulate redundant LANs behind them (e.g. distributed data center)

crypto isakmp policy 10
encr 3des
authentication pre-share
group 5
crypto isakmp key cisco address 0.0.0.0
crypto isakmp keepalive 10
crypto ipsec transform-set MYSET esp-3des esp-sha-hmac
mode tunnel
crypto map MYMAP 10 ipsec-isakmp
set peer 101.1.1.1 default !!!by default tunnel goes to R2
set peer 102.1.1.1 !!! if keepalives (DPD) to R2 fail, tunnel is initiated to R3
set transform-set MYSET
match address MYACL
reverse-route static !!!this installs routes to 2.2.2.2 dynamically based on ACL and state of DPD
crypto map MYMAP


ip access-list ext MYACL
permit ip host 1.1.1.1 2.2.2.2

int loopback0
ip addr 1.1.1.1 255.255.255.255

 

Now let’s ping 2.2.2.2 from R1’s 1.1.1.1 when R2 and R3 are up with crypto isakmp debug on.

IOU1#ping 2.2.2.2 source 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1

*Aug 7 03:11:44.860: ISAKMP:(0): SA request profile is (NULL)
*Aug 7 03:11:44.860: ISAKMP: Created a peer struct for 101.1.1.1, peer port 500
*Aug 7 03:11:44.860: ISAKMP: New peer created peer = 0xF2F1E4D8 peer_handle = 0x80000006
*Aug 7 03:11:44.860: ISAKMP: Locking peer struct 0xF2F1E4D8, refcount 1 for isakmp_initiator
*Aug 7 03:11:44.860: ISAKMP: local port 500, remote port 500
*Aug 7 03:11:44.860: ISAKMP: set new node 0 to QM_IDLE
*Aug 7 03:11:44.860: ISAKMP: Find a dup sa in the avl tree during calling isadb_insert sa = F2F30F50
*Aug 7 03:11:44.860: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Aug 7 03:11:44.860: ISAKMP:(0):found peer pre-shared key matching 101.1.1.1
*Aug 7 03:11:44.860: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Aug 7 03:11:44.860: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Aug 7 03:11:44.860: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Aug 7 03:11:44.860: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Aug 7 03:11:44.860: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Aug 7 03:11:44.860: ISAKMP:(0):Old State = IKE_READY New State = IKE_I_MM1

*Aug 7 03:11:44.860: ISAKMP:(0): beginning Main Mode exchange
*Aug 7 03:11:44.860: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:11:44.860: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Aug 7 03:11:44.863: ISAKMP (0): received packet from 101.1.1.1 dport 500 sport 500 Global (I) MM_NO_STATE
*Aug 7 03:11:44.863: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:11:44.863: ISAKMP:(0):Old State = IKE_I_MM1 New State = IKE_I_MM2

*Aug 7 03:11:44.863: ISAKMP:(0): processing SA payload. message ID = 0
*Aug 7 03:11:44.863: ISAKMP:(0): processing vendor id payload
*Aug 7 03:11:44.863: ISAKMP:(0): vendor ID seems Unity/DPD but major 69 mismatch
*Aug 7 03:11:44.863: ISAKMP (0): vendor ID is NAT-T RFC 3947
*Aug 7 03:11:44.863: ISAKMP:(0):found peer pre-shared key matching 101.1.1.1
*Aug 7 03:11:44.863: ISAKMP:(0): local preshared key found
*Aug 7 03:11:44.863: ISAKMP : Scanning profiles for xauth ...
*Aug 7 03:11:44.863: ISAKMP:(0):Checking ISAKMP transform 1 against priority 10 policy
*Aug 7 03:11:44.863: ISAKMP: encryption 3DES-CBC
*Aug 7 03:11:44.863: ISAKMP: hash SHA
*Aug 7 03:11:44.863: ISAKMP: default group 5
*Aug 7 03:11:44.863: ISAKMP: auth pre-share
*Aug 7 03:11:44.863: ISAKMP: life type in seconds
*Aug 7 03:11:44.863: ISAKMP: life duration (VPI) of 0x0 0x1 0x51 0x80
*Aug 7 03:11:44.863: ISAKMP:(0):atts are acceptable. Next payload is 0
*Aug 7 03:11:44.863: ISAKMP:(0):Acceptable atts:actual life: 0
*Aug 7 03:11:44.863: ISAKMP:(0):Acceptable atts:life: 0
*Aug 7 03:11:44.863: ISAKMP:(0):Fill atts in sa vpi_length:4
*Aug 7 03:11:44.863: ISAKMP:(0):Fill atts in sa life_in_seconds:86400
*Aug 7 03:11:44.863: ISAKMP:(0):Returning Actual lifetime: 86400
*Aug 7 03:11:44.863: ISAKMP:(0)::Started lifetime timer: 86400.

*Aug 7 03:11:44.863: ISAKMP:(0): processing vendor id payload
*Aug 7 03:11:44.863: ISAKMP:(0): vendor ID seems Unity/DPD but major 69 mismatch
*Aug 7 03:11:44.863: ISAKMP (0): vendor ID is NAT-T RFC 3947
*Aug 7 03:11:44.863: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Aug 7 03:11:44.863: ISAKMP:(0):Old State = IKE_I_MM2 New State = IKE_I_MM2

*Aug 7 03:11:44.863: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_SA_SETUP
*Aug 7 03:11:44.863: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Aug 7 03:11:44.863: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Aug 7 03:11:44.863: ISAKMP:(0):Old State = IKE_I_MM2 New State = IKE_I_MM3

*Aug 7 03:11:44.882: ISAKMP (0): received packet from 101.1.1.1 dport 500 sport 500 Global (I) MM_SA_SETUP
*Aug 7 03:11:44.882: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:11:44.882: ISAKMP:(0):Old State = IKE_I_MM3 New State = IKE_I_MM4

*Aug 7 03:11:44.882: ISAKMP:(0): processing KE payload. message ID = 0
*Aug 7 03:11:44.896: ISAKMP:(0): processing NONCE payload. message ID = 0
*Aug 7 03:11:44.896: ISAKMP:(0):found peer pre-shared key matching 101.1.1.1
*Aug 7 03:11:44.896: ISAKMP:(1004): processing vendor id payload
*Aug 7 03:11:44.896: ISAKMP:(1004): vendor ID is Unity
*Aug 7 03:11:44.896: ISAKMP:(1004): processing vendor id payload
*Aug 7 03:11:44.896: ISAKMP:(1004): vendor ID is DPD
*Aug 7 03:11:44.896: ISAKMP:(1004): processing vendor id payload
*Aug 7 03:11:44.896: ISAKMP:(1004): speaking to another IOS box!
*Aug 7 03:11:44.896: ISAKMP:received payload type 20
*Aug 7 03:11:44.896: ISAKMP (1004): His hash no match - this node outside NAT
*Aug 7 03:11:44.896: ISAKMP:received payload type 20
*Aug 7 03:11:44.896: ISAKMP (1004): No NAT Found for self or peer
*Aug 7 03:11:44.896: ISAKMP:(1004):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Aug 7 03:11:44.896: ISAKMP:(1004):Old State = IKE_I_MM4 New State = IKE_I_MM4

*Aug 7 03:11:44.897: ISAKMP:(1004):Send initial contact
*Aug 7 03:11:44.897: ISAKMP:(1004):SA is doing pre-shared key authentication using id type ID_IPV4_ADDR
*Aug 7 03:11:44.897: ISAKMP (1004): ID payload
next-payload : 8
type : 1
address : 100.1.1.1
protocol : 17
port : 500
length : 12
*Aug 7 03:11:44.897: ISAKMP:(1004):Total payload length: 12
*Aug 7 03:11:44.897: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_KEY_EXCH
*Aug 7 03:11:44.897: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:11:44.897: ISAKMP:(1004):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Aug 7 03:11:44.897: ISAKMP:(1004):Old State = IKE_I_MM4 New State = IKE_I_MM5

*Aug 7 03:11:44.899: ISAKMP (1004): received packet from 101.1.1.1 dport 500 sport 500 Global (I) MM_KEY_EXCH
*Aug 7 03:11:44.899: ISAKMP:(1004): processing ID payload. message ID = 0
*Aug 7 03:11:44.899: ISAKMP (1004): ID payload
next-payload : 8
type : 1
address : 101.1.1.1
protocol : 17
port : 500
length : 12
*Aug 7 03:11:44.899: ISAKMP:(0):: peer matches *none* of the profiles
*Aug 7 03:11:44.899: ISAKMP:(1004): processing HASH payload. message ID = 0
*Aug 7 03:11:44.899: ISAKMP:(1004):SA authentication status:
authenticated
*Aug 7 03:11:44.899: ISAKMP:(1004):SA has been authenticated with 101.1.1.1
*Aug 7 03:11:44.899: ISAKMP: Trying to insert a peer 100.1.1.1/101.1.1.1/500/, and inserted successfully F2F1E4D8.
*Aug 7 03:11:44.899: ISAKMP:(1004):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:11:44.899: ISAKMP:(1004):Old State = IKE_I_MM5 New State = IKE_I_MM6

*Aug 7 03:11:44.900: ISAKMP:(1004):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Aug 7 03:11:44.900: ISAKMP:(1004):Old State = IKE_I_MM6 New State = IKE_I_MM6

*Aug 7 03:11:44.909: ISAKMP:(1004):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Aug 7 03:11:44.909: ISAKMP:(1004):Old State = IKE_I_MM6 New State = IKE_P1_COMPLETE

*Aug 7 03:11:44.909: ISAKMP:(1004):IKE_DPD is enabled, initializing timers
*Aug 7 03:11:44.909: ISAKMP:(1004):beginning Quick Mode exchange, M-ID of 3736979590
*Aug 7 03:11:44.909: ISAKMP:(1004):QM Initiator gets spi
*Aug 7 03:11:44.909: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:11:44.909: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:11:44.909: ISAKMP:(1004):Node 3736979590, Input = IKE_MESG_INTERNAL, IKE_INIT_QM
*Aug 7 03:11:44.909: ISAKMP:(1004):Old State = IKE_QM_READY New State = IKE_QM_I_QM1
*Aug 7 03:11:44.909: ISAKMP:(1004):Input = IKE_MESG_INTERNAL, IKE_PHASE1_COMPLETE
*Aug 7 03:11:44.909: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

*Aug 7 03:11:44.913: ISAKMP (1004): received packet from 101.1.1.1 dport 500 sport 500 Global (I) QM_IDLE
*Aug 7 03:11:44.913: ISAKMP:(1004): processing HASH payload. message ID = 3736979590
*Aug 7 03:11:44.913: ISAKMP:(1004): processing SA payload. message ID = 3736979590
*Aug 7 03:11:44.913: ISAKMP:(1004):Checking IPSec proposal 1
*Aug 7 03:11:44.913: ISAKMP: transform 1, ESP_3DES
*Aug 7 03:11:44.913: ISAKMP: attributes in transform:
*Aug 7 03:11:44.913: ISAKMP: encaps is 1 (Tunnel)
*Aug 7 03:11:44.913: ISAKMP: SA life type in seconds
*Aug 7 03:11:44.913: ISAKMP: SA life duration (basic) of 3600
*Aug 7 03:11:44.913: ISAKMP: SA life type in kilobytes
*Aug 7 03:11:44.913: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0
*Aug 7 03:11:44.913: ISAKMP: authenticator is HMAC-SHA
*Aug 7 03:11:44.913: ISAKMP:(1004):atts are acceptable.
*Aug 7 03:11:44.913: ISAKMP:(1004): processing NONCE payload. message ID = 3736979590
*Aug 7 03:11:44.913: ISAKMP:(1004): processing ID payload. message ID = 3736979590
*Aug 7 03:11:44.913: ISAKMP:(1004): processing ID payload. message ID = 3736979590
*Aug 7 03:11:44.913: ISAKMP:(1004):Node 3736979590, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
*Aug 7 03:11:44.913: ISAKMP:(1004):Old State = IKE_QM_I_QM1 New State = IKE_QM_IPSEC_INSTALL_AWAIT
*Aug 7 03:11:44.913: ISAKMP: Failed to find peer index node to update peer_info_list
*Aug 7 03:11:44.914: ISAKMP:(1004):Received IPSec Install callback... proceeding with the negotiation
*Aug 7 03:11:44.914: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:11:44.914: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:11:44.914: ISAKMP:(1004):deleting node -557987706 error FALSE reason "No Error"
*Aug 7 03:11:44.914: ISAKMP:(1004):Node 3736979590, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
*Aug 7 03:11:44.914: ISAKMP:(1004):Old State = IKE_QM_IPSEC_INSTALL_AWAIT New State = IKE_QM_PHASE2_COMPLETE.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 8/12/18 ms
IOU1#
*Aug 7 03:11:57.029: ISAKMP (1004): received packet from 101.1.1.1 dport 500 sport 500 Global (I) QM_IDLE
*Aug 7 03:11:57.029: ISAKMP: set new node 2039185025 to QM_IDLE
*Aug 7 03:11:57.029: ISAKMP:(1004): processing HASH payload. message ID = 2039185025
*Aug 7 03:11:57.029: ISAKMP:(1004): processing NOTIFY DPD/R_U_THERE protocol 1
spi 0, message ID = 2039185025, sa = 0xF2F30F50
*Aug 7 03:11:57.029: ISAKMP:(1004):deleting node 2039185025 error FALSE reason "Informational (in) state 1"
*Aug 7 03:11:57.029: ISAKMP:(1004):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
*Aug 7 03:11:57.029: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

*Aug 7 03:11:57.029: ISAKMP:(1004):DPD/R_U_THERE received from peer 101.1.1.1, sequence 0x6EEFB371
*Aug 7 03:11:57.029: ISAKMP: set new node 1411802357 to QM_IDLE
*Aug 7 03:11:57.029: ISAKMP:(1004):Sending NOTIFY DPD/R_U_THERE_ACK protocol 1
spi 4087798040, message ID = 1411802357
*Aug 7 03:11:57.029: ISAKMP:(1004): seq. no 0x6EEFB371
*Aug 7 03:11:57.029: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:11:57.029: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:11:57.030: ISAKMP:(1004):purging node 1411802357
*Aug 7 03:11:57.030: ISAKMP:(1004):Input = IKE_MESG_FROM_PEER, IKE_MESG_KEEP_ALIVE
IOU1#
*Aug 7 03:11:57.030: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

Now on R2 let’s shut down the public facing interface eth0/0 and see what happens on R1.

IOU1#ping 2.2.2.2 source 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
.....
Success rate is 0 percent (0/5)

*Aug 7 03:14:55.573: ISAKMP: DPD received KMI message.
*Aug 7 03:14:55.573: ISAKMP: set new node 1236715696 to QM_IDLE
*Aug 7 03:14:55.573: ISAKMP:(1004):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 4073138204, message ID = 1236715696
*Aug 7 03:14:55.573: ISAKMP:(1004): seq. no 0x484909A3
*Aug 7 03:14:55.573: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:14:55.574: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:14:55.574: ISAKMP:(1004):purging node 1236715696
IOU1#show ip route
*Aug 7 03:14:57.586: ISAKMP:(1004):DPD incrementing error counter (1/5)
*Aug 7 03:14:57.586: ISAKMP: set new node -140430850 to QM_IDLE
*Aug 7 03:14:57.586: ISAKMP:(1004):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 4087797952, message ID = 4154536446
*Aug 7 03:14:57.586: ISAKMP:(1004): seq. no 0x484909A4
*Aug 7 03:14:57.586: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:14:57.586: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:14:57.586: ISAKMP:(1004):purging node -140430850
*Aug 7 03:14:57.586: ISAKMP:(1004):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
IOU1#
*Aug 7 03:14:57.586: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

IOU1#
*Aug 7 03:14:59.592: ISAKMP:(1004):DPD incrementing error counter (2/5)
*Aug 7 03:14:59.592: ISAKMP: set new node 2080816415 to QM_IDLE
*Aug 7 03:14:59.592: ISAKMP:(1004):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 4087797952, message ID = 2080816415
*Aug 7 03:14:59.592: ISAKMP:(1004): seq. no 0x484909A5
*Aug 7 03:14:59.592: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:14:59.592: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:14:59.592: ISAKMP:(1004):purging node 2080816415
*Aug 7 03:14:59.592: ISAKMP:(1004):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
IOU1#
*Aug 7 03:14:59.592: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

IOU1#
*Aug 7 03:15:02.855: ISAKMP:(1004):DPD incrementing error counter (3/5)
*Aug 7 03:15:02.855: ISAKMP: set new node 527308109 to QM_IDLE
*Aug 7 03:15:02.855: ISAKMP:(1004):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 4087797952, message ID = 527308109
*Aug 7 03:15:02.855: ISAKMP:(1004): seq. no 0x484909A6
*Aug 7 03:15:02.855: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:15:02.855: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:15:02.855: ISAKMP:(1004):purging node 527308109
*Aug 7 03:15:02.855: ISAKMP:(1004):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
IOU1#
*Aug 7 03:15:02.855: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

IOU1#
*Aug 7 03:15:04.855: ISAKMP:(1004):DPD incrementing error counter (4/5)
*Aug 7 03:15:04.855: ISAKMP: set new node -345256241 to QM_IDLE
*Aug 7 03:15:04.855: ISAKMP:(1004):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 4087797952, message ID = 3949711055
*Aug 7 03:15:04.855: ISAKMP:(1004): seq. no 0x484909A7
*Aug 7 03:15:04.855: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:15:04.855: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:15:04.855: ISAKMP:(1004):purging node -345256241
*Aug 7 03:15:04.855: ISAKMP:(1004):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
IOU1#
*Aug 7 03:15:04.855: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

IOU1#
*Aug 7 03:15:06.860: ISAKMP:(1004):DPD incrementing error counter (5/5)
*Aug 7 03:15:06.860: ISAKMP:(1004):peer 101.1.1.1 not responding!
*Aug 7 03:15:06.860: ISAKMP:(1004):peer does not do paranoid keepalives.

*Aug 7 03:15:06.860: ISAKMP:(1004):deleting SA reason "End of ipsec tunnel" state (I) QM_IDLE (peer 101.1.1.1)
*Aug 7 03:15:06.860: ISAKMP:(1004):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
*Aug 7 03:15:06.860: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

*Aug 7 03:15:06.860: ISAKMP: Failed to find peer index node to update peer_info_list
*Aug 7 03:15:06.860: ISAKMP: set new node 186812859 to QM_IDLE
*Aug 7 03:15:06.861: ISAKMP:(1004): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:15:06.861: ISAKMP:(1004):Sending an IKE IPv4 Packet.
*Aug 7 03:15:06.861: ISAKMP:(1004):purging node 186812859
*Aug 7 03:15:06.861: ISAKMP:(1004):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
*Aug 7 03:15:06.861: ISAKMP:(1004):Old State = IKE_P1_COMPLETE New State = IKE_DEST_SA

*Aug 7 03:15:06.861: ISAKMP:(1004):deleting SA reason "End of ipsec tunnel" state (I) QM_IDLE (peer 101.1.1.1)
*Aug 7 03:15:06.861: ISAKMP: Unlocking peer struct 0xF2F1E4D8 for isadb_mark_sa_deleted(), count 0
IOU1#
*Aug 7 03:15:06.861: ISAKMP:(1004):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:15:06.861: ISAKMP:(1004):Old State = IKE_DEST_SA New State = IKE_DEST_SA

*Aug 7 03:15:06.988: ISAKMP: Deleting peer node by peer_reap for 101.1.1.1: F2F1E4D8
*Aug 7 03:15:06.988: ISAKMP: ignoring request to send delete notify (no ISAKMP sa) src 100.1.1.1 dst 101.1.1.1 for SPI 0xA9FAAD12
IOU1#ping 2.2.2.2 source 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1

*Aug 7 03:15:26.674: ISAKMP:(0): SA request profile is (NULL)
*Aug 7 03:15:26.674: ISAKMP: Created a peer struct for 101.1.1.1, peer port 500
*Aug 7 03:15:26.674: ISAKMP: New peer created peer = 0xF2F1E4D8 peer_handle = 0x80000007
*Aug 7 03:15:26.674: ISAKMP: Locking peer struct 0xF2F1E4D8, refcount 1 for isakmp_initiator
*Aug 7 03:15:26.674: ISAKMP: local port 500, remote port 500
*Aug 7 03:15:26.674: ISAKMP: set new node 0 to QM_IDLE
*Aug 7 03:15:26.674: ISAKMP: Find a dup sa in the avl tree during calling isadb_insert sa = F4AB36C8
*Aug 7 03:15:26.674: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Aug 7 03:15:26.674: ISAKMP:(0):found peer pre-shared key matching 101.1.1.1
*Aug 7 03:15:26.674: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Aug 7 03:15:26.674: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Aug 7 03:15:26.674: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Aug 7 03:15:26.674: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Aug 7 03:15:26.674: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Aug 7 03:15:26.674: ISAKMP:(0):Old State = IKE_READY New State = IKE_I_MM1

*Aug 7 03:15:26.674: ISAKMP:(0): beginning Main Mode exchange
*Aug 7 03:15:26.674: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:15:26.674: ISAKMP:(0):Sending an IKE IPv4 Packet......
Success rate is 0 percent (0/5)
IOU1#
*Aug 7 03:15:36.676: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Aug 7 03:15:36.676: ISAKMP (0): incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
*Aug 7 03:15:36.676: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Aug 7 03:15:36.676: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:15:36.676: ISAKMP:(0):Sending an IKE IPv4 Packet.
IOU1#
*Aug 7 03:15:46.684: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Aug 7 03:15:46.684: ISAKMP (0): incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Aug 7 03:15:46.684: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Aug 7 03:15:46.684: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:15:46.684: ISAKMP:(0):Sending an IKE IPv4 Packet.
IOU1#
*Aug 7 03:15:56.676: ISAKMP: set new node 0 to QM_IDLE
*Aug 7 03:15:56.676: ISAKMP:(0):SA is still budding. Attached new ipsec request to it. (local 100.1.1.1, remote 101.1.1.1)
*Aug 7 03:15:56.676: ISAKMP: Error while processing SA request: Failed to initialize SA
*Aug 7 03:15:56.676: ISAKMP: Error while processing KMI message 0, error 2.
*Aug 7 03:15:56.693: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Aug 7 03:15:56.693: ISAKMP (0): incrementing error counter on sa, attempt 3 of 5: retransmit phase 1
*Aug 7 03:15:56.693: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
IOU1#
*Aug 7 03:15:56.693: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:15:56.693: ISAKMP:(0):Sending an IKE IPv4 Packet.
IOU1#
*Aug 7 03:16:06.698: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Aug 7 03:16:06.698: ISAKMP (0): incrementing error counter on sa, attempt 4 of 5: retransmit phase 1
*Aug 7 03:16:06.698: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Aug 7 03:16:06.698: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:16:06.698: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Aug 7 03:16:06.865: ISAKMP:(1004):purging SA., sa=F2F30F50, delme=F2F30F50
IOU1#
*Aug 7 03:16:16.699: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Aug 7 03:16:16.699: ISAKMP (0): incrementing error counter on sa, attempt 5 of 5: retransmit phase 1
*Aug 7 03:16:16.699: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Aug 7 03:16:16.699: ISAKMP:(0): sending packet to 101.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:16:16.699: ISAKMP:(0):Sending an IKE IPv4 Packet.
IOU1#
*Aug 7 03:16:26.682: ISAKMP:(0):peer does not do paranoid keepalives.

*Aug 7 03:16:26.682: ISAKMP:(0):deleting SA reason "P1 delete notify (in)" state (I) MM_NO_STATE (peer 101.1.1.1)
*Aug 7 03:16:26.682: ISAKMP:(0): SA request profile is (NULL)
*Aug 7 03:16:26.682: ISAKMP: Created a peer struct for 102.1.1.1, peer port 500
*Aug 7 03:16:26.682: ISAKMP: New peer created peer = 0xF4A92B30 peer_handle = 0x80000008
*Aug 7 03:16:26.682: ISAKMP: Locking peer struct 0xF4A92B30, refcount 1 for isakmp_initiator
*Aug 7 03:16:26.682: ISAKMP: local port 500, remote port 500
*Aug 7 03:16:26.682: ISAKMP: set new node 0 to QM_IDLE
*Aug 7 03:16:26.682: ISAKMP:(0):insert sa successfully sa = F2F30F50
*Aug 7 03:16:26.682: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Aug 7 03:16:26.682: ISAKMP:(0):found peer pre-shared key matching 102.1.1.1
*Aug 7 03:16:26.682: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Aug 7 03:16:26.682: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Aug 7 03:16:26.682: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Aug 7 03:16:26.682: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Aug 7 03:16:26.682: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Aug 7 03:16:26.682: ISAKMP:(0):Old State = IKE_READY New State = IKE_I_MM1

*Aug 7 03:16:26.682: ISAKMP:(0): beginning Main Mode exchange
*Aug 7 03:16:26.682: ISAKMP:(0): sending packet to 102.1.1.1 my_port 500 peer_port 500 (I) MM_NO_STATE
*Aug 7 03:16:26.682: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Aug 7 03:16:26.682: ISAKMP:(0):deleting SA reason "P1 delete notify (in)" state (I) MM_NO_STATE (peer 101.1.1.1)
*Aug 7 03:16:26.682: ISAKMP: Unlocking peer struct 0xF2F1E4D8 for isadb_mark_sa_deleted(), count 0
*Aug 7 03:16:26.682: ISAKMP: Deleting peer node by peer_reap for 101.1.1.1: F2F1E4D8
*Aug 7 03:16:26.683: ISAKMP:(0):deleting node 189642018 error FALSE reason "IKE deleted"
*Aug 7 03:16:26.683: ISAKMP:(0):deleting node -162170267 error FALSE reason "IKE deleted"
*Aug 7 03:16:26.683: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
*Aug 7 03:16:26.683: ISAKMP:(0):Old State = IKE_I_MM1 New State = IKE_DEST_SA

*Aug 7 03:16:26.684: ISAKMP (0): received packet from 102.1.1.1 dport 500 sport 500 Global (I) MM_NO_STATE
*Aug 7 03:16:26.684: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:16:26.684: ISAKMP:(0):Old State = IKE_I_MM1 New State = IKE_I_MM2

*Aug 7 03:16:26.684: ISAKMP:(0): processing SA payload. message ID = 0
*Aug 7 03:16:26.684: ISAKMP:(0): processing vendor id payload
*Aug 7 03:16:26.684: ISAKMP:(0): vendor ID seems Unity/DPD but major 69 mismatch
*Aug 7 03:16:26.684: ISAKMP (0): vendor ID is NAT-T RFC 3947
*Aug 7 03:16:26.684: ISAKMP:(0):found peer pre-shared key matching 102.1.1.1
*Aug 7 03:16:26.684: ISAKMP:(0): local preshared key found
*Aug 7 03:16:26.684: ISAKMP : Scanning profiles for xauth ...
*Aug 7 03:16:26.684: ISAKMP:(0):Checking ISAKMP transform 1 against priority 10 policy
*Aug 7 03:16:26.684: ISAKMP: encryption 3DES-CBC
*Aug 7 03:16:26.684: ISAKMP: hash SHA
*Aug 7 03:16:26.684: ISAKMP: default group 5
*Aug 7 03:16:26.684: ISAKMP: auth pre-share
*Aug 7 03:16:26.684: ISAKMP: life type in seconds
*Aug 7 03:16:26.684: ISAKMP: life duration (VPI) of 0x0 0x1 0x51 0x80
*Aug 7 03:16:26.684: ISAKMP:(0):atts are acceptable. Next payload is 0
*Aug 7 03:16:26.684: ISAKMP:(0):Acceptable atts:actual life: 0
*Aug 7 03:16:26.684: ISAKMP:(0):Acceptable atts:life: 0
*Aug 7 03:16:26.684: ISAKMP:(0):Fill atts in sa vpi_length:4
*Aug 7 03:16:26.684: ISAKMP:(0):Fill atts in sa life_in_seconds:86400
*Aug 7 03:16:26.684: ISAKMP:(0):Returning Actual lifetime: 86400
*Aug 7 03:16:26.684: ISAKMP:(0)::Started lifetime timer: 86400.

*Aug 7 03:16:26.684: ISAKMP:(0): processing vendor id payload
*Aug 7 03:16:26.684: ISAKMP:(0): vendor ID seems Unity/DPD but major 69 mismatch
*Aug 7 03:16:26.684: ISAKMP (0): vendor ID is NAT-T RFC 3947
*Aug 7 03:16:26.684: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Aug 7 03:16:26.684: ISAKMP:(0):Old State = IKE_I_MM2 New State = IKE_I_MM2

*Aug 7 03:16:26.684: ISAKMP:(0): sending packet to 102.1.1.1 my_port 500 peer_port 500 (I) MM_SA_SETUP
*Aug 7 03:16:26.684: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Aug 7 03:16:26.684: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Aug 7 03:16:26.684: ISAKMP:(0):Old State = IKE_I_MM2 New State = IKE_I_MM3

*Aug 7 03:16:26.698: ISAKMP (0): received packet from 102.1.1.1 dport 500 sport 500 Global (I) MM_SA_SETUP
*Aug 7 03:16:26.698: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:16:26.698: ISAKMP:(0):Old State = IKE_I_MM3 New State = IKE_I_MM4

*Aug 7 03:16:26.698: ISAKMP:(0): processing KE payload. message ID = 0
*Aug 7 03:16:26.709: ISAKMP:(0): processing NONCE payload. message ID = 0
*Aug 7 03:16:26.709: ISAKMP:(0):found peer pre-shared key matching 102.1.1.1
*Aug 7 03:16:26.709: ISAKMP:(1005): processing vendor id payload
*Aug 7 03:16:26.709: ISAKMP:(1005): vendor ID is Unity
*Aug 7 03:16:26.709: ISAKMP:(1005): processing vendor id payload
*Aug 7 03:16:26.709: ISAKMP:(1005): vendor ID is DPD
*Aug 7 03:16:26.709: ISAKMP:(1005): processing vendor id payload
*Aug 7 03:16:26.709: ISAKMP:(1005): speaking to another IOS box!
*Aug 7 03:16:26.709: ISAKMP:received payload type 20
*Aug 7 03:16:26.709: ISAKMP (1005): His hash no match - this node outside NAT
*Aug 7 03:16:26.709: ISAKMP:received payload type 20
*Aug 7 03:16:26.709: ISAKMP (1005): No NAT Found for self or peer
*Aug 7 03:16:26.709: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Aug 7 03:16:26.709: ISAKMP:(1005):Old State = IKE_I_MM4 New State = IKE_I_MM4

*Aug 7 03:16:26.709: ISAKMP:(1005):Send initial contact
*Aug 7 03:16:26.709: ISAKMP:(1005):SA is doing pre-shared key authentication using id type ID_IPV4_ADDR
*Aug 7 03:16:26.709: ISAKMP (1005): ID payload
next-payload : 8
type : 1
address : 100.1.1.1
protocol : 17
port : 500
length : 12
*Aug 7 03:16:26.709: ISAKMP:(1005):Total payload length: 12
*Aug 7 03:16:26.709: ISAKMP:(1005): sending packet to 102.1.1.1 my_port 500 peer_port 500 (I) MM_KEY_EXCH
*Aug 7 03:16:26.709: ISAKMP:(1005):Sending an IKE IPv4 Packet.
*Aug 7 03:16:26.710: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Aug 7 03:16:26.710: ISAKMP:(1005):Old State = IKE_I_MM4 New State = IKE_I_MM5

*Aug 7 03:16:26.711: ISAKMP (1005): received packet from 102.1.1.1 dport 500 sport 500 Global (I) MM_KEY_EXCH
*Aug 7 03:16:26.711: ISAKMP:(1005): processing ID payload. message ID = 0
*Aug 7 03:16:26.711: ISAKMP (1005): ID payload
next-payload : 8
type : 1
address : 102.1.1.1
protocol : 17
port : 500
length : 12
*Aug 7 03:16:26.711: ISAKMP:(0):: peer matches *none* of the profiles
*Aug 7 03:16:26.711: ISAKMP:(1005): processing HASH payload. message ID = 0
*Aug 7 03:16:26.711: ISAKMP:(1005):SA authentication status:
authenticated
*Aug 7 03:16:26.711: ISAKMP:(1005):SA has been authenticated with 102.1.1.1
*Aug 7 03:16:26.711: ISAKMP: Trying to insert a peer 100.1.1.1/102.1.1.1/500/, and inserted successfully F4A92B30.
*Aug 7 03:16:26.711: ISAKMP:(1005):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Aug 7 03:16:26.711: ISAKMP:(1005):Old State = IKE_I_MM5 New State = IKE_I_MM6

*Aug 7 03:16:26.711: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Aug 7 03:16:26.711: ISAKMP:(1005):Old State = IKE_I_MM6 New State = IKE_I_MM6

*Aug 7 03:16:26.717: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Aug 7 03:16:26.717: ISAKMP:(1005):Old State = IKE_I_MM6 New State = IKE_P1_COMPLETE

*Aug 7 03:16:26.717: ISAKMP:(1005):IKE_DPD is enabled, initializing timers
*Aug 7 03:16:26.717: ISAKMP:(1005):beginning Quick Mode exchange, M-ID of 777919085
*Aug 7 03:16:26.717: ISAKMP:(1005):QM Initiator gets spi
*Aug 7 03:16:26.717: ISAKMP:(1005): sending packet to 102.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:16:26.717: ISAKMP:(1005):Sending an IKE IPv4 Packet.
*Aug 7 03:16:26.717: ISAKMP:(1005):Node 777919085, Input = IKE_MESG_INTERNAL, IKE_INIT_QM
*Aug 7 03:16:26.717: ISAKMP:(1005):Old State = IKE_QM_READY New State = IKE_QM_I_QM1
*Aug 7 03:16:26.717: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PHASE1_COMPLETE
*Aug 7 03:16:26.717: ISAKMP:(1005):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

*Aug 7 03:16:26.719: ISAKMP (1005): received packet from 102.1.1.1 dport 500 sport 500 Global (I) QM_IDLE
*Aug 7 03:16:26.719: ISAKMP:(1005): processing HASH payload. message ID = 777919085
*Aug 7 03:16:26.719: ISAKMP:(1005): processing SA payload. message ID = 777919085
*Aug 7 03:16:26.719: ISAKMP:(1005):Checking IPSec proposal 1
*Aug 7 03:16:26.719: ISAKMP: transform 1, ESP_3DES
*Aug 7 03:16:26.719: ISAKMP: attributes in transform:
*Aug 7 03:16:26.719: ISAKMP: encaps is 1 (Tunnel)
*Aug 7 03:16:26.719: ISAKMP: SA life type in seconds
*Aug 7 03:16:26.719: ISAKMP: SA life duration (basic) of 3600
*Aug 7 03:16:26.719: ISAKMP: SA life type in kilobytes
*Aug 7 03:16:26.719: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0
*Aug 7 03:16:26.719: ISAKMP: authenticator is HMAC-SHA
*Aug 7 03:16:26.719: ISAKMP:(1005):atts are acceptable.
*Aug 7 03:16:26.719: ISAKMP:(1005): processing NONCE payload. message ID = 777919085
*Aug 7 03:16:26.719: ISAKMP:(1005): processing ID payload. message ID = 777919085
*Aug 7 03:16:26.719: ISAKMP:(1005): processing ID payload. message ID = 777919085
*Aug 7 03:16:26.719: ISAKMP:(1005):Node 777919085, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
*Aug 7 03:16:26.719: ISAKMP:(1005):Old State = IKE_QM_I_QM1 New State = IKE_QM_IPSEC_INSTALL_AWAIT
*Aug 7 03:16:26.720: ISAKMP: Failed to find peer index node to update peer_info_list
*Aug 7 03:16:26.720: ISAKMP:(1005):Received IPSec Install callback... proceeding with the negotiation
*Aug 7 03:16:26.720: ISAKMP:(1005): sending packet to 102.1.1.1 my_port 500 peer_port 500 (I) QM_IDLE
*Aug 7 03:16:26.720: ISAKMP:(1005):Sending an IKE IPv4 Packet.
*Aug 7 03:16:26.720: ISAKMP:(1005):deleting node 777919085 error FALSE reason "No Error"
*Aug 7 03:16:26.720: ISAKMP:(1005):Node 777919085, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
IOU1#
*Aug 7 03:16:26.720: ISAKMP:(1005):Old State = IKE_QM_IPSEC_INSTALL_AWAIT New State = IKE_QM_PHASE2_COMPLETE
IOU1#


 

It’s interesting to see how much time elapsed: a lot! I shut down the interface at around 3:14:40. The DPD realized there’s a problem at 3:14:55 so it started sending keepalives every 2 seconds. It gave up at 3:15:06. This is already 26 seconds. I pinged 2.2.2.2 again at 3:15:26, R2 didn’t respond so isakmp gave up after another minute and only then did it probe R3. This gives us around 90 seconds of delay in case of a failure of the primary vpn peer.

 

Implementation and verification solutions

Hello

I’ve been looking at existing automation solutions for a while now and it seems I have the following options now:

IMPLEMENTATION:

  • netmiko + yaml
  • Unicon (pyATS transport plugin) + yaml.
  • Ansible
  • nornir
  • napalm

VERIFICATION AND DATA EXTRACTION:

  • pyATS
  • netmiko
  • napalm

The problem that I’m dealing at work is that i have a custom bastion that i need to traverse to get to my managed network devices. I will therefore try to implement a few designs (acl add, acl delete, vlan add, vpn add, route add etc.) using each of the implementation solutions, first without and then with the custom proxy. I’ll try to compare the effort to implement and verify. (And yes, I do realize that the final goal should be a continouos integration solution like Jenkins but hey, Rome wasn’t build in a day)

 

 

 

Implementing ACLs using automation libraries

Hello

Over the next few weeks I’d like to test a few automation libraries to see which ones are easier to use than others. I hope this can be a useful introduction for those without much prior experience with automation or coding in general.
I don’t have much experience with python as I only did some programming in java a long time ago so this entry has a lot of code borrowed from all over the internet 🙂 We are all learning here so if you see something particularly noobish, feel free to laugh out loud.

I first launched GNS3 on ubuntu 20.04 and prepared a simple topology

GNS3_netmiko

I connected ubuntu to gns3 using the nat cloud. R1 is 192.168.122.100.

NETMIKO

My goal was to implement a simple ACL on R1. To do that, i’ve prepared an acl.j2 template file:

{% for accesslist in data.accesslists %}
ip access-list extended {{accesslist.aclname}}
{{accesslist.aclcommand}} {{accesslist.protocol}} {{accesslist.source}} {{accesslist.sourcewildcard}} {{accesslist.destination}} {{accesslist.destwildcard}}
{% endfor %}

Then I prepared the acl data in the acl.yml file:

accesslists:
- aclname: permit_www
aclcommand: permit
protocol: ip
source: 192.168.122.100
sourcewildcard: 0.0.0.0
destination: 8.8.8.8
destwildcard: 0.0.0.0

Finally, the python script.

import sys
from netmiko import ConnectHandler
import time
from netmiko import redispatch
from netmiko import Netmiko
from jinja2 import Environment, FileSystemLoader
import yaml

def connect(hostip):

net_connect = ConnectHandler(device_type='cisco_ios', host=hostip, username='cisco', password='cisco', secret='cisco')
net_connect.find_prompt()
net_connect.enable()
accesslists = yaml.load(open('acl.yml'), Loader=yaml.SafeLoader)
env = Environment(loader = FileSystemLoader)
template = env.get_template('acl.j2') 
acl_config = template.render(data=accesslists) 
print(acl_config) 
print(f"Logged into {hostip} successfully") 
output = net_connect.send_config_set(acl_config.split("\n")) 
#split method returns a list, that can be used in the send_config_set
if __name__ == "__main__": 
ip = sys.argv[1] 
connect(ip)


I then ran the script:

tode@ubuntu:~/nmikoymlproj$ python3 firstscript.py 192.168.122.100
ip access-list extended permit_www
permit ip 192.168.122.100 0.0.0.0 8.8.8.8 0.0.0.0

Logged into 192.168.122.100 successfully


 

UNICON

from unicon import Connection
import time
from jinja2 import Environment, FileSystemLoader
import yaml

#this prepares the config
accesslists = yaml.load(open('acl2.yml'), Loader=yaml.SafeLoader)
env = Environment(loader = FileSystemLoader('.'), trim_blocks=True, autoescape=True)
template = env.get_template('acl.j2')
acl2_config = template.render(data=accesslists)
preoutput = acl2_config.split("\n")

#this prepares the connection
c = Connection(hostname='R1', start=['telnet ' + router1], os='ios', credentials={'default': {'username': 'cisco', 'password': 'cisco'}, 'secret': 'cisco'},)
c.connect()

#This checks how many commands the config set has. This is because unicon's configure method adds ''end'' after each command. So in case of nested commands we need to push the whole set of commands (e.g. interface loop0,ip addr 1.1.1.1 255.255.255.255). So command is a list, to which I push all elements of the config, then the whole list is pushed to the router.

x = len(preoutput)
y=0
command = []
while y<x:
  command.append(preoutput[x-x+y])
  y = y+1

#this actually configures the router
print(command)output = c.configure(command)

and the output…

tode@ubuntu:~/nmikoymlproj$ python3 uniconscript.py

2020-08-03 04:12:14,672: %UNICON-INFO: +++ R1 logfile /tmp/R1-20200803T041214670.log +++

2020-08-03 04:12:14,672: %UNICON-INFO: +++ Unicon plugin ios +++
ip access-list extended permit_www
permit ip 192.168.122.220 0.0.0.0 9.9.9.9 0.0.0.0

Logged into 192.168.122.100 successfully
Trying 192.168.122.100...


2020-08-03 04:12:14,706: %UNICON-INFO: +++ connection to spawn: telnet 192.168.122.100, id: 140510945062576 +++

2020-08-03 04:12:14,707: %UNICON-INFO: connection to R1
Connected to 192.168.122.100.
Escape character is '^]'.


User Access Verification

Username: cisco
Password: 
R1#

2020-08-03 04:12:15,391: %UNICON-INFO: +++ initializing handle +++

2020-08-03 04:12:15,391: %UNICON-INFO: +++ R1: executing command 'term length 0' +++
term length 0
R1#

2020-08-03 04:12:15,510: %UNICON-INFO: +++ R1: executing command 'term width 0' +++
term width 0
R1#

2020-08-03 04:12:15,670: %UNICON-INFO: +++ R1: executing command 'show version' +++
show version
Cisco IOS Software, 7200 Software (C7200-ADVENTERPRISEK9-M), Version 12.4(24)T5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport

(...)

Configuration register is 0x2102

R1#

2020-08-03 04:12:15,820: %UNICON-INFO: +++ R1: config +++
config term
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#no logging console
R1(config)#line console 0
R1(config-line)#exec-timeout 0
R1(config-line)#end
R1#
['ip access-list extended permit_www', 'permit ip 192.168.122.220 0.0.0.0 9.9.9.9 0.0.0.0', '']

2020-08-03 04:12:16,007: %UNICON-INFO: +++ R1: config +++
config term
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ip access-list extended permit_www
R1(config-ext-nacl)#permit ip 192.168.122.220 0.0.0.0 9.9.9.9 0.0.0.0
R1(config-ext-nacl)#end
R1#

 

Now let’s take the hosts into an inventory yml file:

from unicon import Connection
from jinja2 import Environment, FileSystemLoader
import yaml
from pyats.topology import loader

#this prepares the config and command sets
accesslists = yaml.load(open('acl2.yml'), Loader=yaml.SafeLoader)
env = Environment(loader = FileSystemLoader('.'), trim_blocks=True, autoescape=True)
template = env.get_template('acl.j2')
acl2_config = template.render(data=accesslists)
preoutput = acl2_config.split("\n")
x = len(preoutput)
y=0
command = []
while y<x:
  command.append(preoutput[x-x+y])
  y = y+1

#this loads the testbed topology and connects
testbed = loader.load("testbed.yml")
c = testbed.devices['R1']
c.connect()

#actual configuration
output = c.configure(command)

where the inventory file looks as follows:

devices:
    R1:
       connections:
          cli:
             ip: 192.168.122.100
             port: 23 
             protocol: telnet
      credentials:
         default:
             password: cisco
             username: cisco
         enable:
              password: cisco
     os: ios
     type: ios

 

Now this code is still not ideal because too many things are hardcoded:

  • acl.j2 template
  • acl.yml file with the config
  • name of router from the testbed

So let’s do move them out into sys.args + a bit of refactoring to have everything neat

def prepareconfig(template,configdata):
  accesslists = yaml.load(open(configdata), Loader=yaml.SafeLoader)
  env = Environment(loader = FileSystemLoader('.'), trim_blocks=True, autoescape=True)
  template = env.get_template(template)
  realconfig = template.render(data=accesslists)
  preoutput = realconfig.split("\n")
  x = len(preoutput)
  y=0
  command = []
  while y<x:
     command.append(preoutput[x-x+y])
     y = y+1
  return command

def connect(testbedfile,devicename):
  testbed = loader.load(testbedfile)
  c = testbed.devices[devicename]
  c.connect()
  return c

def configure(command,connection):
  thiscommand = command
  c = connection
  output = c.configure(thiscommand)

##USAGE   python3 script.py acl.j2 acl.yml R1 testbed.yml

if __name__ == "__main__":
  template = sys.argv[1]
  configdata = sys.argv[2]
  devicename = sys.argv[3]
  testbedfile = sys.argv[4]
  configure(prepareconfig(template,configdata),connect(testbedfile,devicename))



This seems quite easy, but for every feature it is necessary to come up with a .j2 template and prepare data according to the template.

ANSIBLE

- name: add_entry_to_acl 
hosts: testrouter
tasks:
- name: add_new_entry
ios_config: 
lines:
- permit ip 10.10.10.10 0.0.0.0 11.11.11.11 0.0.0.0
parents: ip access-list extended permit_www
before: ip access-list extended permit_www
match: exact
authorize: yes

well ain’t that faster than typing up all that python code myself…

Obviously there are things hardcoded in this so let’s do some more work here.

I’ve added an acl20 var to my testrouter.yml under host_vars and changed the playbook in the following way:

- name: modify_acl_entries
  hosts: testrouter
  tasks:
  - name: add_new_entry
    ios_config: 
    lines:
     - "{{ acl20 }}" 
    parents: ip access-list extended permit_www
    before: ip access-list extended permit_www
    save_when: modified

and the output is…

tode@ubuntu:~/ansiblefolder$ ansible-playbook aclplaybook.yml

PLAY [modify_acl_entries] ********************************************************

TASK [add_new_entry] ***********************************************************
[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host testrouter should use 
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with 
prior Ansible releases. A future Ansible release will default to using the 
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.9/reference_appendices/interpreter_discovery.html for more information. This 
feature will be removed in version 2.12. Deprecation warnings can be disabled 
by setting deprecation_warnings=False in ansible.cfg.
changed: [testrouter]

PLAY RECAP *********************************************************************
testrouter : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0


 

Napalm

I’ve had no previous experience with this one so let’s start from scratch: First, the installation:

pip3 install napalm

Now, on the router in GNS3 you need to have a working file system, on my 7200 router I had to increase the size of disk0 under router template, then issue: format disk0: and you’re good to go.

I created an acltest.cfg file with a simple acl:

ip access-list extended napalm_acl

 permit host 1.2.3.4 host 7.8.9.10

Then I created a simple python napalm script:

from napalm.base import get_network_driver

driver = get_network_driver('ios')
dev = driver('192.168.122.100', 'cisco', 'cisco',)

dev.open()
dev.load_merge_candidate(filename='acltest.cfg')
dev.commit_config()
dev.close()

ok, quite easy, but there are so many things hardcoded here. Let’s create an inventory file first:

{
   "r1": {
            "IP": "192.168.122.100",
            "type": "ios",
            "user": "cisco",
            "password": "cisco"
         },
    "r2": {
            "IP": "192.168.122.101",
            "type": "ios",
            "user": "cisco",
            "password": "cisco"
          }
}

and now let’s rework the script a bit

from napalm.base import get_network_driver
import sys
import json

hostname = sys.argv[1]
acl_file = sys.argv[2]


with open("inventory.json", "r") as f:
    dev_db = json.load(f)
dev_param = dev_db[hostname.lower()]
driver = get_network_driver(dev_param['type'])
with driver(dev_param['IP'], dev_param['user'], dev_param['password']) as device:

device.open()
device.load_merge_candidate(acl_file)
device.commit_config()
device.close()

and now let’s run the script

tode@ubuntu:~/napalmproj$ python3 napalm_script.py R1 acltest.cfg

Alternatively, I could use here j2 templates, process the template and data, output a config file and use it instead of having a ”ready config” (useful if the operator is not familiar with ios). Obviously now it is possible to mix and match all the approaches that i’ve gone thru so far.

 

Ansible+Napalm

 

Update@17.08.2020
This one was surprisingly difficult to set up but i really liked the result. I created a config file that i wanted to merge with the running config on the target device.

ip access-list ext ansnap
permit ip 16.17.18.19 0.0.0.0 20.21.22.23 0.0.0.0

I also prepared my inventory.yml file

[gnsrouter]
testrouter

In group_vars, i created gnsrouter.yml with the following settings:

ansible_connection : network_cli
ansible_network_os : ios
ansible_python_interpreter : "/usr/bin/python3"

 

I then created the playbook:

- name: merge_config
hosts: testrouter
gather_facts: no 
tasks:
- name: whatever
  napalm_install_config:
   hostname: '192.168.122.100'
   username: 'cisco'
   password: 'cisco'
   dev_os: "ios"
   optional_args:
    port: 22
   config_file: 'config_file' 
   commit_changes: True
   replace_config: False
   get_diffs: True
   diff_file: 'R1diff.txt'

 

Now i run the playbook:

tode@ubuntu:~/ansiblefolder$ ansible-playbook ansnap.yml

PLAY [merge_config] ************************************************************

TASK [whatever] ****************************************************************
changed: [testrouter]

PLAY RECAP *********************************************************************
testrouter : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

And the result is the diff file:

tode@ubuntu:~/ansiblefolder$ cat R1diff.txt 
+ip access-list ext ansnap
+ permit ip 16.17.18.19 0.0.0.0 20.21.22.23 0.0.0.0

I’ll try to see what more I can do with this…

 

Nornir + Netmiko

To create this script, i got my .j2 and .yml acl files from the Netmiko script. I prepared the config.yaml file in the following fashion:

tode@ubuntu:~/nornirfolder$ cat config.yaml
---
core:
num_workers: 20

inventory:
plugin: nornir.plugins.inventory.simple.SimpleInventory
options:
host_file: "inventory/hosts.yaml"
group_file: "inventory/groups.yaml"
defaults_file: "inventory/defaults.yaml"

The files look as follows:

tode@ubuntu:~/nornirfolder/inventory$ cat hosts.yaml 
---
testrouter:
hostname: 192.168.122.100
port: 22
groups:
- cisco_ios
- testroutergroup
tode@ubuntu:~/nornirfolder/inventory$ cat groups.yaml 
---
testroutergroup:
groups:
- cisco_ios

cisco_ios:
platform: ios
tode@ubuntu:~/nornirfolder/inventory$ cat defaults.yaml 
---
username: cisco
password: cisco

Then it was quite easy once I worked out how to use the run method.

from nornir.plugins.tasks.networking import napalm_get
from nornir.plugins.functions.text import print_result
import json
from nornir.plugins.tasks import networking
import sys
from netmiko import ConnectHandler
import time 
from netmiko import Netmiko 
from jinja2 import Environment, FileSystemLoader 
import yaml


#this creates the acl_config string
env = Environment(loader = FileSystemLoader('.'), trim_blocks=True, autoescape=True) 
template = env.get_template('aclnr.j2') 
accesslists = yaml.load(open('aclnr.yml'), Loader=yaml.SafeLoader)
acl_config = template.render(data=accesslists) 
print(acl_config)

nr = InitNornir(config_file="./config.yaml")
task_result = nr.run(task=networking.netmiko_send_config, config_commands=acl_config.splitlines())
print_result(task_result)

 

Nornir + Napalm

A similar example, but with a file config and a manual connection def (i might need this later to do a manual jump through my proxy)

from nornir import InitNornir
from nornir.plugins.tasks.networking import napalm_get
from nornir.plugins.functions.text import print_result
import json
from nornir.plugins.tasks.networking import napalm_configure
from nornir.plugins.tasks import networking
import sys
from netmiko import ConnectHandler
import time 
from netmiko import redispatch 
from netmiko import Netmiko 
from jinja2 import Environment, FileSystemLoader 
import yaml


def task_manages_connection_manually(task):
task.host.open_connection("napalm", configuration=task.nornir.config)
r = task.run(
task=napalm_configure, filename='aclconfig', 
)
task.host.close_connection("napalm")
nr = InitNornir(config_file="./config.yaml")
task_result = nr.run(task=task_manages_connection_manually,)
print_result(task_result)

tode@ubuntu:~/nornirfolder$ python3 manualscript.py

 task_manages_connection_manually************************************************
* testrouter ** changed : True *************************************************
vvvv task_manages_connection_manually ** changed : False vvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_configure ** changed : True ---------------------------------------- INFO
+ip access-list ext napalmconfg
+permit ip 4.7.9.11 0.0.0.0 5.3.1.2 0.0.0.0
^^^^ END task_manages_connection_manually ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We can see that the lines have been added (+)

Achieving idempotency here can be done by writing the config file in exactly the same way as it will appear on the cisco router. I corrected the aclconfig file like this and ran the script once again. This time the status of napalm_configure task is: changed: False

ip access-list extended napalmconfg
 permit ip host 4.7.9.11 host 5.3.1.2

tode@ubuntu:~/nornirfolder$ python3 manualscript.py 

task_manages_connection_manually************************************************
* testrouter ** changed : False ************************************************
vvvv task_manages_connection_manually ** changed : False vvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_configure ** changed : False --------------------------------------- INFO
^^^^ END task_manages_connection_manually ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tode@ubuntu:~/nornirfolder$

 

 

 

 

 

 

 

 

 

 

 

 

 

pyATS + unicon = great way of jumping through SSH proxies and bastions

Hello

For the last year i’ve been trying to find a way to implement yaml-based configs on my routers located behind a non-typical jumphost. On this jumphost i need to type „connect <hostname>” to get to my routers. Ansible cannot do this by default because it uses paramiko or native SSH to connect to devices.
Today i’ve found the Unicon library used in pyAts. The code turned out to be super easy:

from unicon import Connection

proxy_conn = Connection(hostname='mybastion', start=['ssh myusername@1.1.1.1 -p 722'], os='linux', credentials={'default': {'username': 'admin', 'password': 'mystupidpassword'}})

c = Connection(hostname='Router1', start=['connect Router1'], os='ios', proxy_connections=[proxy_conn])
c.connect()

As easy as pie. This works on my ubuntu 16.04 with python 3.8. Unfortunately, this works neither in Windows python nor on Ubuntu WSL. (I’m pretty sure it will work in WSL 2 when my windows is updated to windows 10 2004 version, we’ll see).
The next step will be to do the same in yaml, then prepare a playbook in yaml (or whatever pyats calls the task lists).

The example on unicon website looks like this so i suspect something similar:

devices:
  jumphost:
    os: linux
    type: linux
    connections:
      cli:
        protocol: ssh
        ip: 1.1.1.1
        port: 722
  Router:
    os: ios
    type: router
    connections:
      defaults:
        class: unicon.Unicon
      cli:
        command: connect Router1
        proxy: jumphost

Screen scraping made dignified with textfsm templates

Hello

This time I tried out some ready-made templates that some folks prepare to process cisco show command outputs:

from ntc_templates.parse import parse_output
vlan_output = (
„VLAN Name Status Ports\n”
„—- ——————————– ——— ——————————-\n”
„1 default active Gi0/1\n”
„10 Management active \n”
„50 VLan50 active Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5,\n”
” Fa0/6, Fa0/7, Fa0/8\n”
)
vlan_parsed = parse_output(platform=”cisco_ios”, command=”show vlan”, data=vlan_output)

!!!the output gave us a list of dictionaries that we need to loop through.
In the first step we see that length of vlan_parsed is 3, so we can loop through that.

When we are inside the first loop, we have dictionary objects, so we can use the .items function to go through that, just remembering that because we are inside of the first loop we need to call this by list[x].

x = 0
while x <len(vlan_parsed):
  print()
  print()
  print('NEXT VLAN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
  print()
  for key, value in vlan_parsed[x].items():
     print('---------------------------------')
     print(key, ' : ', value)
  x += 1

And voila!

NEXT VLAN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

———————————
status : active
———————————
name : default
———————————
vlan_id : 1
———————————
interfaces : [‚Gi0/1’]
NEXT VLAN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

———————————
status : active
———————————
name : Management
———————————
vlan_id : 10
———————————
interfaces : []
NEXT VLAN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


 

And yes, i know it’s still screen scraping. But kind of easier because you don’t have to do the regex and string manipulation yourself.

Next example was a bit more difficult because some templates don’t allow for headers. If something doesn’t work, look in your /home/.local/lib/python3.5/site-packages/ntc_templates/templates

In this case after „start’ we have just a record line, so in my show ip interface brief i removed the headers and just pasted the output.

cat cisco_ios_show_ip_interface_brief.textfsm
Value INTF (\S+)
Value IPADDR (\S+)
Value STATUS (up|down|administratively down)
Value PROTO (up|down)

Start
^${INTF}\s+${IPADDR}\s+\w+\s+\w+\s+${STATUS}\s+${PROTO} -> Record
# Capture time-stamp if vty line has command time-stamping turned on
^Load\s+for\s+
^Time\s+source\s+is


 

So here’s my final python script

templates.parse import parse_output

ship_output = (

"GigabitEthernet0/0/0 15.10.8.107 YES NVRAM up up\n"
"Loopback1255 111.21.186.11 YES NVRAM up up\n"
"Tunnel1251 111.21.173.11 YES NVRAM up up\n"
"Tunnel1255 111.21.187.11 YES NVRAM up up\n"
"Vlan1 192.168.8.11 YES NVRAM up up\n"


)
ship_parsed = parse_output(platform="cisco_ios", command="show ip interface brief", data=ship_output)

x = 0
while x <len(ship_parsed):
print()
print()
print('NEXT ONE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
print()
for key, value in ship_parsed[x].items():
print('---------------------------------')
print(key, ' : ', value)
x += 1

and the script output is…

NEXT ONE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------
proto : up
---------------------------------
status : up
---------------------------------
intf : GigabitEthernet0/0/0
---------------------------------
ipaddr : 15.10.8.107


NEXT ONE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------
proto : up
---------------------------------
status : up
---------------------------------
intf : Loopback1255
---------------------------------
ipaddr : 111.21.186.11


NEXT ONE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------
proto : up
---------------------------------
status : up
---------------------------------
intf : Tunnel1251
---------------------------------
ipaddr : 111.21.173.11


NEXT ONE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------
proto : up
---------------------------------
status : up
---------------------------------
intf : Tunnel1255
---------------------------------
ipaddr : 111.21.187.11


NEXT ONE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------
proto : up
---------------------------------
status : up
---------------------------------
intf : Vlan1
---------------------------------
ipaddr : 192.168.8.11
>>>
>>>

 

Netmiko redispatch with ssh bastion

Hello

In my network environment, to reach customer devices i need to go through a non-standard SSH proxy, where it’s not possible to do an SSH connection through the SSH connection to the proxy. I played with Netmiko last year but I didn’t know how to get through the ssh proxy such that i still could use Netmiko’s native set of methods once i connect to the final device. This turns out to be quite easy with the redispatch command.

In this script I can go through my devices and get the uptime from show version (words between „Uptime” up to „minutes”). This returns a list, so in order to create a nice output with the hostname and uptime, you need to insert the hostname at position 0 of the list (prepend).

from netmiko import ConnectHandler
import time
from netmiko import redispatch
import re

jumpserver={‚device_type’:’terminal_server’,’ip’:’11.1.1.1′,’username’:’username’,’password’:’cisco’,’port’:22,’global_delay_factor’:5,’session_log’: ‚output.txt’}

net_connect=ConnectHandler(**jumpserver)
net_connect.find_prompt()

net_connect.write_channel(„connect 172.16.0.1\n”)
time.sleep(2)
redispatch(net_connect, device_type=”cisco_ios”)
net_connect.enable
result = net_connect.send_command_timing(„show ver”)
endresult = re.findall(r’Uptime .*?minutes’, result)
endresult.insert(0,’DeviceR1′)
print(endresult)
net_connect.write_channel(„exit\n”)
time.sleep(1)

My own config generator and autoloader part 3 – Verification

Hello

In part 2 i uploaded the config using Telnet. Now, I don’t exactly trust telnet + my connection to the company install center is sometimes… not very trustworthy. Therefore, it would be good to verify the config against what was intended. To do that, i’ve extended the last upload script.

import pexpect
import os
import sys
from pathlib import Path

#now, the main method also includes a command that can be run, e.g. show run, creates a text file with the command output and copies it back to the windows folder. 

def createall(ip,port,consoleserver,argcommand):
   myarraypath = str(Path.home()).split('/')
   table = { 39 : None }
   myfinaluser = myarraypath[2].translate(table)
   print(myfinaluser)
   clearcommand = "truncate -s 0 c.txt"
   os.system(clearcommand)
   command = "tail --lines=+1 /mnt/c/Users/" + myfinaluser + "/Desktop/mycurrentscripts/test.txt >> /home/" + myfinaluser + "/c.txt"
   os.system(command)
   child = pexpect.spawn('telnet ' + ip + ' ' + port)
   argcommandnew = argcommand
   commandfile = open(argcommandnew, 'wb')
   child.timeout = 50
   child.delaybeforesend = 1.0
   myconsoleserver = consoleserver
   child.expect(consoleserver + ' login:')
   child.sendline('hereyourusername')
   child.expect('Password:')
   child.sendline('hereyourpassword')
   child.sendline()
   neverknow = child.expect(['#', '>'])
   if neverknow==1:
      child.sendline('en')
      child.expect('Password:')
      child.sendline('myenablepassword')
      child.expect('#')
   elif neverknow==0:
      child.sendline('configure terminal')
      child.expect('#')
      child.sendline('do terminal length 0')
      child.expect('#')
      child.logfile = commandfile
      child.sendline('do ' + argcommandnew)
      child.expect('(config)#')
   copycommand = 'cp ./' + commandfile + '/mnt/c/Users/' + myfinaluser + '/Desktop/mycurrentscripts/commandoutput'
   os.system(copycommand)
     # filename = 'c.txt'
     # with open(filename, encoding="utf8", errors='ignore') as file_object:
      #  lines = file_object.readlines()
       # for line in lines:
        #  line = line #+ "\015"
         # child.sendline(line)
#usage:
#python3 script.py 172.16.0.1 myconsoleserver 'show run'
if __name__ == "__main__":
  ip = sys.argv[1]
  port = sys.argv[2]
  consoleserver = sys.argv[3]
  argcommand = sys.argv[4]
  createall(ip, port, consoleserver, argcommand)


Now, you can use Notepad++ compare plugin to compare the two text files. At this point you will probably see that the two files are not identical due to peculiar formatting of the show run output. So then it’s time to go back to the template in the config python script and add extra spacing etc.

!
vrf definition 1
 rd 1:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition 2
 description OtherVRF
 rd 2:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition 3
 description VRF MGMT
 rd 3:1
 !
 address-family ipv4
 exit-address-family
!

Finally, don’t do”show run” but ”show run brief” to verify the config. This will remove the certificates from the output (less differences to look at in notepad). You will never get this 100% right, but in the end it looks pretty good.