Hi
The problem to solve here is as follows: Ansible only supports standard SSH proxies, but it doesn’t support custom jump servers. In my case, i ssh to the jump server but later i need to type in a command ‚connect mytargetrouter<ENTER>’ .
Let’s see how (and if) this can be solved.
Normally I document final results only. Today, however, i’d like to show that sometimes the path to the result can be equally interesting and that a failure can lead (eventually, after days of thinking and research) to good conclusions.
After I managed to set up Nornir to connect to my gns3 routers (one of the previous blog entries), I wanted to try and see if it is also capable of traversing a non-ssh proxy (in my environment i connect to a proxy where i write ”connect MYROUTER<ENTER>” and after up to 30 seconds i get to myrouter).
The way I approached it was completely wrong, but I only understood this after I’d run the script.
from nornir import InitNornir
from nornir.plugins.tasks.networking import napalm_get
from nornir.plugins.functions.text import print_result
import json
from nornir.plugins.tasks.networking import napalm_configure
from nornir.plugins.tasks import networking
from nornir.plugins.tasks.networking import netmiko_send_command
import sys
from netmiko import ConnectHandler
import time
from netmiko import redispatch
from netmiko import Netmiko
from jinja2 import Environment, FileSystemLoader
import yaml
def task_manages_connection_manually(task, router):
task.host.open_connection("netmiko", configuration=task.nornir.config)
p = task.run(
task=netmiko_send_command, command_string ='connect ' + router + ' \n', use_timing=True)
r = task.run(
task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True)
sr = task.run(
task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True)
tr = task.run(task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True)
ur = task.run(task=netmiko_send_command, command_string ='show ip interface brief\n', use_timing=True)
task.host.close_connection("netmiko")
nr = InitNornir(config_file="./config.yaml", logging={"file": "mylogs", "level": "debug"} )
task_result = nr.run(task=task_manages_connection_manually, router='mytestrouter')
print_result(task_result)
The problem here was the unknown and everchanging delay that I experience on my jumpserver. By playing a bit with the use_timing variable I was able to get this to work but only if I inserted some fake tasks (tr, ur above .) that take some time to execute. I don’t care whether they will really execute. They are here just to give us time.
So now nornir was able to connect to my target router and get the output of show ip interface brief command (sometimes even twice, which is generally not desirable when you run any scripts) and for 10 seconds I was happy but then I realized that i’d accomplished nothing. My hosts.yaml only had the proxy, not the target router. Even though i will be able to move the args to init method, nornir now gives me totally no advantage over netmiko. It thinks it connects to one target host only (proxy) and is not aware of the real target. I can no longer use any filtering, groups etc etc. This is obviously a bad idea. A few hours wasted.
I had to come up with something better. I gave up on Nornir for the time being and decided to be bold. If Ansible cannot use a custom (i.e. not a supported ssh bastion) proxy by default, why not customise the connection plugins?
I had a look at the network_cli.py of Ansible at the following path:
tode@ubuntu:/usr/lib/python3/dist-packages/ansible/plugins/connection$
Because network_cli uses paramiko, i then looked at the paramiko_ssh.py, where I found this connect_uncached def:
def _connect_uncached(self):
''' activates the connection object '''
if paramiko is None:
raise AnsibleError("paramiko is not installed: %s" % to_native(PARAMIKO_IMPORT_ERR))
port = self._play_context.port or 22
display.vvv("ESTABLISH PARAMIKO SSH CONNECTION FOR USER: %s on PORT %s TO %s" % (self._play_context.remote_user, port, self._play_context.remote_addr),
host=self._play_context.remote_addr)
ssh = paramiko.SSHClient()
# override paramiko's default logger name
if self._log_channel is not None:
ssh.set_log_channel(self._log_channel)
self.keyfile = os.path.expanduser("~/.ssh/known_hosts")
if self.get_option('host_key_checking'):
for ssh_known_hosts in ("/etc/ssh/ssh_known_hosts", "/etc/openssh/ssh_known_hosts"):
try:
# TODO: check if we need to look at several possible locations, possible for loop
ssh.load_system_host_keys(ssh_known_hosts)
break
except IOError:
pass # file was not found, but not required to function
ssh.load_system_host_keys()
ssh_connect_kwargs = self._parse_proxy_command(port)
ssh.set_missing_host_key_policy(MyAddPolicy(self._new_stdin, self))
allow_agent = True
if self._play_context.password is not None:
allow_agent = False
try:
key_filename = None
if self._play_context.private_key_file:
key_filename = os.path.expanduser(self._play_context.private_key_file)
# paramiko 2.2 introduced auth_timeout parameter
if LooseVersion(paramiko.__version__) >= LooseVersion('2.2.0'):
ssh_connect_kwargs['auth_timeout'] = self._play_context.timeout
ssh.connect(
self._play_context.remote_addr.lower(),
username=self._play_context.remote_user,
allow_agent=allow_agent,
look_for_keys=self.get_option('look_for_keys'),
key_filename=key_filename,
password=self._play_context.password,
timeout=self._play_context.timeout,
port=port,
**ssh_connect_kwargs
)
Now the plan is as follows: to remove the self._play_context values from ssh.connect and instead hardcode my proxy user and password. Finally, I will send a connect command in the ssh channel: ssh.exec_command(‚connect MYROUTER’). This seems easy enough.
After making the changes in the paramiko_ssh.py I was able to get to my jump host and I could see that Ansible got to the jump host prompt but the ssh.exec_command(cmd) to go to my target router didn’t seem to have any effect. In the end ansible closed the channel. A step back, then. I needed to create a manual paramiko script and test it in GNS3 to see what is happening.
Let’s create a simple Paramiko script and start a router in GNS3
import paramiko
import time
s = paramiko.SSHClient()
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
s.connect(hostname='192.168.122.100', username='cisco', password='cisco', port='22')
stdin, stdout, stderr = s.exec_command('show ip interface brief')
time.sleep(2)
stdout=stdout.readlines()
output = ""
for line in stdout:
output=output+line
print(output)
s.close()
BTW without the workaround with time.sleep() you will encounter a bug:
Traceback (most recent call last): File "/home/tode/.local/lib/python3.8/site-packages/paramiko/file.py", line 66, in __del__ File "/home/tode/.local/lib/python3.8/site-packages/paramiko/channel.py", line 1392, in close File "/home/tode/.local/lib/python3.8/site-packages/paramiko/channel.py", line 991, in shutdown_write File "/home/tode/.local/lib/python3.8/site-packages/paramiko/channel.py", line 967, in shutdown File "/home/tode/.local/lib/python3.8/site-packages/paramiko/transport.py", line 1846, in _send_user_message AttributeError: 'NoneType' object has no attribute 'time'
With time.sleep(2) it runs just fine:
tode@ubuntu:~/paramikoscript$ python3 simple.py Interface IP-Address OK? Method Status Protocol FastEthernet0/0 192.168.122.100 YES NVRAM up up FastEthernet1/0 unassigned YES NVRAM administratively down down FastEthernet2/0 unassigned YES NVRAM administratively down down Loopback2001 200.200.200.200 YES NVRAM up up Loopback2002 200.200.200.201 YES NVRAM up up
ok so now back to the ssh paramiko file in ansible plugins:
ssh.connect(
'1.1.1.1',
username='cisco',
allow_agent=allow_agent,
look_for_keys=self.get_option('look_for_keys'),
key_filename=key_filename,
password='cisco',
timeout=60,
port=722,
**ssh_connect_kwargs
)
ssh.exec_command('connect testrouter\r\n')
time.sleep(10)
Let’s test this now… However, this still fails (channel closed error). I nearly got discouraged and stopped working on this for a day because I had no idea why the command didn’t work. It turned out that the reason is that my jump server doesn’t support Paramiko’s exec_command. Looks like I may have to use invoke_shell() . Let’s test this invoke_shell thing again in GNS3
import logging
import paramiko
import time
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("paramiko").setLevel(logging.DEBUG)
paramiko.util.log_to_file("paramiko.log")
s = paramiko.SSHClient()
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
s.connect(hostname='192.168.122.100', username='cisco', password='cisco', port='22')
chan = s.invoke_shell()
chan.send(' show ip interface brief | redirect disk0:showip.txt\n')
time.sleep(2)
s.close()
Running the script…
tode@ubuntu:~/paramikoscript$ python3 simple.py DEBUG:paramiko.transport:starting thread (client mode): 0x38429df0 DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.7.1 DEBUG:paramiko.transport:Remote version/idstring: SSH-1.99-Cisco-1.25 INFO:paramiko.transport:Connected (version 1.99, client Cisco-1.25) DEBUG:paramiko.transport:kex algos:['diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['aes128-cbc', '3des-cbc', 'aes192-cbc', 'aes256-cbc'] server encrypt:['aes128-cbc', '3des-cbc', 'aes192-cbc', 'aes256-cbc'] client mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-md5', 'hmac-md5-96'] server mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-md5', 'hmac-md5-96'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False DEBUG:paramiko.transport:Kex agreed: diffie-hellman-group-exchange-sha1 DEBUG:paramiko.transport:HostKey agreed: ssh-rsa DEBUG:paramiko.transport:Cipher agreed: aes128-cbc DEBUG:paramiko.transport:MAC agreed: hmac-sha1 DEBUG:paramiko.transport:Compression agreed: none DEBUG:paramiko.transport:Got server p (2048 bits) DEBUG:paramiko.transport:kex engine KexGex specified hash_algo <built-in function openssl_sha1> DEBUG:paramiko.transport:Switch to new keys ... DEBUG:paramiko.transport:Adding ssh-rsa host key for [192.168.122.100]:22: b'd59685e80107be2bdfcc1686a83fc22e' DEBUG:paramiko.transport:userauth is OK INFO:paramiko.transport:Authentication (password) successful! DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes DEBUG:paramiko.transport:[chan 0] Max packet out: 4096 bytes DEBUG:paramiko.transport:Secsh channel 0 opened. DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
The result is a new file on disk0 of my testrouter with the output of show ip int brief. This is good. So maybe now I can implement this in the ansible plugin.
This is the full scope of the changes in the paramiko ansible plugin. I’m using 1.1.1.1 with port 722 as my proxy and testrouter as my target router. The actual IP of the testrouter is not important (=it doesn’t have to be defined in the hosts.yaml or any inventory) because the jumpserver is fully aware of all host mappings.
ssh.connect(
'1.1.1.1',
username='cisco',
allow_agent=allow_agent,
look_for_keys=self.get_option('look_for_keys'),
key_filename=key_filename,
password='cisco',
timeout=60,
port=722,
**ssh_connect_kwargs
)
channel = ssh.invoke_shell()
channel.send('connect testrouter\n')
time.sleep(20)
channel.send('show ip interface brief | redirect bootflash:ship.txt\n')
Just another look at the playbook that I will run again:
- name: show ip interface brief gather_facts: false hosts: testrouter tasks: - name: "get ship output" ios_command: commands: "show ip interface brief" register: cli_result
However, when I tested the playbook I got a FATAL error saying that Ansible cannot elevate the privileges on the jump server (it tries to do that after the ssh_connect)
This is because Ansible cannot run the BECOME method on the jump host, so I disabled (in my group_vars) the ansible_become variable with: ansible_become : no
Now I was able to use the playbook!
tode@ubuntu:~/ansiblefolder$ ansible-playbook showipintbrief.yml -vvv
ansible-playbook 2.9.6
config file = /home/tode/ansiblefolder/ansible.cfg
configured module search path = ['/home/tode/.local/lib/python3.8/site-packages/napalm_ansible/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 3.8.2 (default, Jul 16 2020, 14:00:26) [GCC 9.3.0]
Using /home/tode/ansiblefolder/ansible.cfg as config file
host_list declined parsing /home/tode/ansiblefolder/inventory.yml as it did not pass its verify_file() method
script declined parsing /home/tode/ansiblefolder/inventory.yml as it did not pass its verify_file() method
Parsed /home/tode/ansiblefolder/inventory.yml inventory source with ini plugin
PLAYBOOK: showipintbrief.yml *******************************************************************************************************************************************
1 plays in showipintbrief.yml
PLAY [show ip interface brief] *****************************************************************************************************************************************
META: ran handlers
TASK [get ship output] *************************************************************************************************************************************************
task path: /home/tode/ansiblefolder/showipintbrief.yml:5
<192.168.122.100> ESTABLISH LOCAL CONNECTION FOR USER: tode
<192.168.122.100> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990 `" && echo ansible-tmp-1598528447.646881-107080737393990="` echo /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/network/ios/ios_command.py
<192.168.122.100> PUT /home/tode/.ansible/tmp/ansible-local-6990397lydnf/tmppwshwp3x TO /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/AnsiballZ_ios_command.py
<192.168.122.100> EXEC /bin/sh -c 'chmod u+x /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/ /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/AnsiballZ_ios_command.py && sleep 0'
<192.168.122.100> EXEC /bin/sh -c '/usr/bin/python3 /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/AnsiballZ_ios_command.py && sleep 0'
<192.168.122.100> EXEC /bin/sh -c 'rm -f -r /home/tode/.ansible/tmp/ansible-local-6990397lydnf/ansible-tmp-1598528447.646881-107080737393990/ > /dev/null 2>&1 && sleep 0'
ok: [testrouter] => {
"changed": false,
"invocation": {
"module_args": {
"auth_pass": null,
"authorize": null,
"commands": [
"show ip interface brief"
],
"host": null,
"interval": 1,
"match": "all",
"password": null,
"port": null,
"provider": null,
"retries": 10,
"ssh_keyfile": null,
"timeout": null,
"username": null,
"wait_for": null
}
},
"stdout": [
"404 Command Not Found"
],
"stdout_lines": [
[
"404 Command Not Found"
]
]
}
META: ran handlers
META: ran handlers
PLAY RECAP *************************************************************************************************************************************************************
testrouter : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
and sure enough, the txt file with the output of show ip interface brief appeared on the bootflash of my target router. Mission accomplished.
But what exactly happened here? The ansible debug stdout shows ”404 command not found”. This means that the command from the playbook was issued on the jump host instead of the target router. The only reason ansible succeeded was because the only task was to issue the show command and register the result.
What does this mean? It means there is another python file to modify: somewhere where Ansible opens the channel within the ssh connection. This file will need to be aware of the playbook’s variables like the router names etc. Unfortunately, the actual SSH connection is only to the jump host and this ssh connection is returned in paramiko_ssh.py. Modifying Ansible is far from done at this point.
Good news is that Ansible managed to actually issue the command on the jumphost and got some output so opening a channel is done somewhere correctly irrespective of the operating system on the host. In theory, an alternative to channel.py modification then exists: we could even make a playbook where the first task would be to connect to the ”inventory host”. But i’m not sure i like this workaround.
to be continued…
