VPN automation project.

Hi

Some of you may know that for the last few months I’ve been working on some vpn stuff; the project is basically a glorified hardware refresh, where we send out new routers to our clients’ locations. We preconfigure these routers in our install center, ups them over to the client, connect via ssh, finish the config, job done. This is easy because it’s a dmvpn setup with very few changes on the hub. However, what we also do is changes to existing crypto site2site tunnels and it’s just painful how manual this stuff is:

  • i get a ticket with IP (source and destination)
  • i need to correlate this change with the correct crypto map entry
  • in the crypto map entry i find the crypto acl name
  • i look up the existing crypto map acl to see if this means that i need to add an acl line or not
  • if needed, i add a new line in the crypto acl

What i would like to have is an automated process where making changes on a website would autogenerate a new config, send an email with this config (embedded in a python script) to the network administrator, who would simply execute the python script. This could also be done using ansible.

Step 1: prepare a website with javascript where each client connection would be presented row by row
Step 2: prepare the config script, make sure that adding a row on the website adds a line in the config script
Step 2a: make sure that only relevant edits result in a new config (something like a check if a new crypto acl entry is actually needed)
Step 2b: old versions of the crypto acl need to be archived, name of editor must be visible
Step 3: make sure email with the content is sent to the administrator
Step 4: prepare python script such that the change doesn't have to be done manually
Step 5: Prepare a transaction such that if the tunnel is DOWN after the change, the change is reverted

This seems a bit challenging now that i’ve written it all down. I wonder if I’ll be able to do it.

SSH keys lost after reload on 16.9.2

Hello

I’m currently taking part in a project where we send out new routers to remote locations and on a number of occasions we had a problem where i couldn’t connect to my preconfigured router via ssh. I thought i was going crazy because this happened randomly and every time i had to console in to the router with the assistance of some onsite technician, which is always a hassle. And today i’ve found the confirmation in CSCvm54595: SSH keys can go missing if you write the config with do wr. Or you can upgrade to 16.9.3+.

I am not going mad after all. Cisco, this was not your finest (regression testing) hour.

 

Multi-sa support for VTIs

Hello

Tired of old crypto maps? ios-xe 16.12 has something for you, then.
https://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/214728-configure-multi-sa-virtual-tunnel-interf.html

This nifty little feature works just like crypto maps behind the scenes, but logically it’s a tunnel. All you need to do is create a tunnel with ip unnamed and add: tunnel protection policy ipv4 <crypto ACL name>:

interface Tunnel0
 ip unnumbered GigabitEthernet0/0/0
 tunnel source GigabitEthernet0/0/0
 tunnel mode ipsec ipv4
 tunnel destination 192.0.2.2
 tunnel protection ipsec policy ipv4 CACL
 tunnel protection ipsec profile PROF

Isn’t this just great?

I’ve tested this feature on 16.12.3 and it seems fine. So you can have a crypto map (your old VPN connections) on your outside interface and, slowly but surely, you can delete all your crypto map entries one by one, transforming them into new VTI tunnels.
One immediate benefit I can think of is the possibility to have QoS on your tunnels, plus you no longer have to deal with the fact that your tunnels are traffic-triggered. You don’t have to wait for your business partner to say: hey tunnel isn’t working, because you will see it in the status of your tunnels so you can monitor them with SNMP. How cool is that.

For now i’ve tested this on my CSR1000v in GNS3. Hopefully I’ll be able to convince someone at work to allow an upgrade on my ASRs so that I can see how this works in production.

I’ll try to see if this also works with DVTIs (virtual templates etc.).

Cool stuff i’ve found in the new core 300-401 coursebook

Hello

I’ve gone through 60% of the book so far (switching, routing, and most of wireless) and I must say it’s a bad book. There’s ton of theory, very little configuration. It may be good as a refresher book just before the exam but I think this approach will just produce paper tigers. But what do i know.

I’ve managed to find some new stuff, though. I’ll keep expanding this. I still have 12 chapters to go.

  1. BGP AIGP – cool stuff, if you have thousands of PEs, and if you’ve grouped them in private BGP AS but for some reason each BGP AS has a different IGP, AIGP lets BGP make routing decisions based on IGP metric. https://packetpushers.net/bgp-aigp/
  2. VRRP v3. This just adds ipv6 support and changes the config to address family config. Nothing fancy.

ISE binary certificate comparison mystery

Hello

I’m wondering if anyone has encountered the same problem with binary certicate comparison. The scenario is as follows:

  • the client hires a new employee
  • an AD account is created for that employee.
  • Meanwhile, a PC is prepared for that employee and an IT person logs to that PC for the first time. CA shares a copy of the certificate with AD.
  • The employee receives the PC but can’t log into the network
  • ISE says that no valid account has been found in AD for this certificate IF binary certificate comparison is used. If it is not used, ISE accepts the certificate as valid.

If the certificates are refreshed on the PC, all is well again.

My theory is that this is due to the fact that there are multiple domain controllers and I think that there might be problems if the domain controller receives the certificate from CA before the object is replicated on that branch domain controller and mapping of machine account to machine certificate fails on that controller. My theory is further corroborated by the following bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr51940

and the following forum entry:

https://www.jamf.com/jamf-nation/discussions/22807/race-condition-when-requesting-an-ad-certificate

 

 

What I do when i don’t do IP

Hi

You can totally skip this entry if you’re only interested in networks. It’s just that i’ve met a few new colleages recently who had the impression that I spent my whole life in the cisco world (and this is so not true! i spend the other half in junos 😀 ) so I’ve decided to show that there’s more to me than just the network guy.
So the other part of what I do is that I learn languages and read books in those languages. I’ve developed this system recently where I try to read 4 books in french, 1 in spanish and 1 in german to stop from getting all rusty. (I think i’d like to learn Arabic as well but it’s definitely not a goal for this decade. Maybe 2030-2040).

I don’t use my languages at work every day but I have been able to use my spanish to talk to clients from south america on a number of occasions and it really boosted my relationship with those clients. I use german occasionally and it’s been a massive help in my career because (unsurprisingly) few network people can speak german on a professional level. But overall I learn languages just to be able to read literature. I read really diverse stuff. Books by Graham Greene, Juli Zeh, Sapkowski, Stephen King, hell, i’ve even read 12 rules for life by Jordan Peterson lately. Although I don’t have the same drive to read for pleasure as in the past (which may be due to the fact that i read too much for work purposes and my brain is in overload most of the time), i can still manage one book a month. In the past i would have been mortified at sharing this poor(ish) result but times have changed and I’m as close to being proud of that as I can be. I have a full time job, i’m a dad and I still read one book a month. Do you get an ”average-at-all” badge if you’re the best dentist-driver? or the best baker among rock singers? or the most-overweight half-marathon participant ? Jack of many trades, master of none 🙂

CCNP and CCIE ENCOR 300-401 finally out

Hello

The book is finally out, at least on Safaribooks. I’ve read 1/3 so far and it’s a nice little refresher of things i read a long time ago and managed to forget, like what happens if you have MST between 2 switches with 2 links connecting them. Now you map vlan 10 instance 1 but leave vlan 20 in instance 0. Then you try to load balance the traffic by putting vlan 10 on upper link and vlan 20 on lower link only. Of course the vlan 20 traffic will be blocked because IST is a member of all interfaces 😀 so you end up with connectivity only on vlan 10 and hosts in vlan 20 lose connectivity. Of course, why would you not map all vlans to mst instances other than 0 ? laziness, i guess. I’ll try to read the whole book this week to see if it’s any good. I kind of expected more depth, but maybe this is the idea of the ”core” book? to keep it simple?

Anyways, i’ve been playing with very random things recently, like dmvpn with ikev2 and certificates between ASRs and new shiny c1116s. Something that is worth testing in gns3 is a weird situation where a branch router is reloaded and dmvpn gets stuck in NHRP state. What i’ve found is that the hub router mistakenly deletes both the old IKE and the new IKE, so it can never decrypt the data. It’s only when DPD on the branch router reinitializes IKE that NHRP starts working properly. So a quick and dirty fix of this bug is to lower DPD to 30 seconds but the bug is still there whenever a branch site loses power. oh well.
Another activity that i’ve been lucky to be assigned to was testing of 9336 nexus switches for a server block with an emphasis on vPC. It’s rare that you can get a free lab so I tried to test a lot of vPC features, like vPC autorecovery, layer3 peer router, peer switch etc. All in all the best assignment i’ve had in months.

Finally, i’d like to share a hilarious ”team work” situation i’ve seen recently; it’s quite recent so i hope nobody feels offended but i can’t help myself: the client reports (6 months ago) that a company laptop may have been stolen and asks the firewall team to block an internal IP from talking with the enterprise network. The brave fw team blocks this one IP and the matter is put to sleep. Now fast forward to October. A client PC can’t connect to certain IPs. The brave fw team receives the incident and logs a comment saying that everything is fine, because this deny action is as per client request 6 months earlier. The confused NOC team asks to remove the rule because it is no longer needed, the laptop was never stolen in the first place, and this is quite possibly a different laptop that happened to receive the same IP because this IP is in the dhcp scope. This request is rejected by the fw team who says that the IP address in question should simply be excluded from the DHCP scope instead because the rule was created based on customer request so it should stay. The ticket then detiorates into a senseless ping-pong (my opinion is so much better than yours etc.).
It is so sad that each team lives in their antinuke silo where everything is comfy and warm, with thick ITIL carpets and linen. And you do not leave your silo because your friends are inside and outside there could be wolves! and dragons! or worse – blood traitors! Giving in to your enemy request once is like inviting a vampire into your home. One never knows what will happen. It’s best to barricade the door and wait for the postnuclear winter. You can only leave your bunker according to ITIL rules. Does ITIL say to eat? No? then die of hunger we shall!