Wyróżnione

Networks can be fun (for life)

john_whitelbow

Hi!

I’m a NOC engineer with a passion for languages. I used to be a language teacher but I realized that this was not something that I could do all my life. I joined Nokia in 2009 and now, 10 years later, i’m a senior network engineer at another major IT company, trying to get better with each day of solving network cases.

I’m always open to new challenges so if you’re stuck with a problem, or need a new network design or a quick implementation, drop me a line ( dewilletomek@gmail.com ) or call me on +48601079955.

WLC firmwares 8.3.143.5 – 8.3.143.8 with critical bugs

Hello

Cisco TAC has recently published versions .9 and .10 of the 8.3.143 software for WLC, which resolve critical malloc failures of a whole range of newer access points in images known as 8.3MR4Esc, bug number https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvm18273

The symptom is clear: users can’t log in using their PSKs, msglog on the WLC says that PSK is incorrect even though users put in the right PSK, while for dot1x users it’s EAP timeouts for M0 or M2 messages etc. Once logged to the AP, you can see that the event log is full of mallocs and tracebacks.

If this happens to you, upgrade as soon as possible. However, it is only possible to get these versions from Cisco TAC. The funny thing is that we only upgraded to 8.3.143.7, because this TAC version resolved a TACACS vulnerability. #ilovecisco

Apparently the same bugs appear in 8.5.x versions, see the bug details at bugsearch site.

HP printers losing DNS entries

Hello

Users started complaining about not being able to use several HP printers in their company. Whenever that happened, they were not able to resolve their names. They were still able to log in to the printer via the IP address.

The company has an IPAM solution that sends DDNS updates to the DNS server, which is administered by a third party so we were not able to have a look at the DNS policies.

What we noticed after a while was that even though our standard DHCP lease was 7 days, HP printers would get 30 days! It turns out that the parameter ”default lease time” does not mean ”maximum lease time”. If a device asks for a longer lease, it gets it via DHCP option 51. Aaargh!

Once we found this, we asked the DNS admins what the scavenging policy was. It turned out that the DNS server considered dynamic entries as stale if they were not refreshed within 8 days. The math was easy: the lease was refreshed at 50% of the lease, so between the 9th day and 15th days of the lease, DNS could scavenge the DNS entry. The DNS admins were reluctant to say what the scavenging interval was.

The solution was to change the parameter „default lease time” to „maximum lease time” on the DHCP server.

Incidentally, the same applies to iphones so make sure you don’t have hundreds of 30-day leases for your BYOD guests. It’s quite easy to run out of IP addresses…

Applying hotfix to ISE

Hello

Today I was hotfixing my ISE because of CSCvm14030 vulnerability. Here’s the output.

The only difference between patching and hotfixing is that the command is „application install” rather than ”patch install”.

I chose to do this manually on all nodes rather than the GUI automatic mode because i wanted to be in control over when I hotfix what node.

myISE/admin# application install ise-apply-CSCvm14030_2.1.0.474_common_1-SPA.tar.gz MyREPO
Save the current ADE-OS running configuration? (yes/no) [yes] ? yes
Generating configuration…
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine…
Unbundling Application Package…
Verifying Application Signature…
Initiating Application Install…

Checking if CSCvm14030_2.1.0.474_common_1 is already applied
– Successful

Checking ISE version compatibility
– Current ISE Version: 2.1.0.474 Patch Version: 7
– Successful

Applying hot patch CSCvm14030_2.1.0.474_common_1
– Taking backup of integrity files
– Taking backup of patch related files
– Running hotpatch wrapper script
– Restarting ISE services

Stopping ISE Monitoring & Troubleshooting Log Collector…
grep: write error
Stopping ISE Monitoring & Troubleshooting Log Processor…
grep: write error
grep: write error
grep: write error
grep: write error
grep: write error
grep: write error
grep: write error
ISE PassiveID Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
ISE PassiveID Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
Stopping ISE Certificate Authority Service…
Stopping ISE EST Service…
ISE Sxp Engine Service is disabled
ISE TC-NAC Service is disabled
Stopping ISE Profiler Database…
Stopping ISE Indexing Engine…
grep: write error
Stopping ISE Monitoring & Troubleshooting Session Database…
ISE PassiveID Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
Stopping ISE Certificate Authority Service…
Stopping ISE EST Service…
ISE Sxp Engine Service is disabled
ISE TC-NAC Service is disabled
Stopping ISE Profiler Database…
Stopping ISE Indexing Engine…
grep: write error
Stopping ISE Monitoring & Troubleshooting Session Database…
Stopping ISE AD Connector…
grep: write error
Stopping ISE Database processes…
Starting ISE Monitoring & Troubleshooting Session Database…
Starting ISE Profiler Database…
grep: write error
Starting ISE Application Server…
Starting ISE Certificate Authority Service…Starting ISE EST Service…
Starting ISE Monitoring & Troubleshooting Log Processor…
Starting ISE Monitoring & Troubleshooting Log Collector…
Starting ISE Indexing Engine…
Starting ISE AD Connector…
Note: ISE Processes are initializing. Use ‚show application status ise’
CLI to verify all processes are in running state.

 

DMVPN NAT mystery

Hello

Today I came across an interesting case, where two out of three tunnels on one spoke (out of 50 spokes in total) were down. The third (working) tunnel was connected to the same two hubs but via mpls.

I checked:

  • tunnel configuration. It was identical in all other 40+ spokes
  • crypto isakmp and ipsec policies. Again, identical.
  • state of crypto: both ISAKMP and IPSEC were up, encrypting and decrypting tunnels

So now it was up to GRE and NHRP to do their job of establishing DMVPN.

When I checked the state of dmvpn on the hub, it showed UP, however, the spokes were showing their state as NHRP. What does it mean? After doing NHRP debug I knew that the spoke wasn’t getting the resolution reply for the two bad tunnels.

Finally, I saw that there were 2 NAT entries for GRE traffic. It was strange, because the traffic was sourced by the same host on the LAN and the destinations were (in both cases) the dmvpn hub.
I looked at the nat translation statement: it was a nat overload to the outside (public) interface.

As GRE doesn’t cooperate well with PAT, I decided to clear the entries, especially that the the source host was Incomplete in the ARP table.

After clearing the nats from the ip nat translation table, the tunnels went up.

What does it teach us? We should make a deny statement for GRE traffic in our NAT ACLs and only use TCP and UDP in nat statements rather than IP.

I’m wondering if the same issue will crash my DMVPN lab or not.

broken CCIE lab with BGP synchronization and OSPF

Hello

We have the following scenario:

Capture

IOU 4 has a static route 10.1.9.9 to IOU 9. It redistributes it into OSPF. IOU4 has OSPF adj to IOU5, IOU5 to IOU2.

IOU2 is a BGP neigh to IOU5 and also to IOU4. IOU4 and IOU5 are route reflector clients of IOU2.

The problem starts when we enable BGP synchronization on R5. What do we know about synchronization? the condition is the route in BGP needs to match the same route in IGP to avoid black holes. where R1>R2>R3 but BGP is only enabled on R1 and R3. R2 causes the black hole.

But having the routes match in BGP and IGP is not the only condition, i’m afraid… let’s have a look at this case:

R5 receives the ospf route to IOU9 from IOU4, but it receives the BGP route to IOU9 from IOU2! The route from the internal routing protocol will not get synchronized with the same route in BGP.

The only fix is to change bgp router-id on IOU2 to be the same as OSPF router-id on IOU4.

 

 

IOU images are buggy

Hello

I’m doing the labs again and so many things are not working out of the box. Today I configured three routers in an NSSA area and one of them insisted on being the translator. The problem was that it wasn’t an ABR. It thought mistakenly that it was connected to the backbone but show ip ospf topology showed status of its backbone area as INACTIVE.

The solution (as always) is to delete the route config and type it in again.

I wonder if the lab images are just as buggy during the exam.

Why vtp pruning is a bad idea

Hello

A while ago I considered implementing VTP with pruning on a client network. But it’s a bad idea because of the following (from Cisco documentation)

(In vtp v3) VTP pruning still applies only to VLANs 1 to 1005, and VLANs 1002 to 1005 are still reserved and cannot be modified.

(…)

VTP pruning is not designed to function in VTP transparent mode. If one or more switches in the network are in VTP transparent mode, you should do one of these:

Turn off VTP pruning in the entire network

Turn off VTP pruning by making all VLANs on the trunk of the switch upstream to the VTP transparent switch pruning ineligible.

There is a funny cornercase scenario related to this: if you have switch A>switch B>switch C, where switch B is a transparent vtp switch while A and C have vtp enabled with pruning, switch C may request (through B) to prune a vlan. Switch A will prune the vlan from the trunk to switch B (which may have needed it!)

In other words: avoid…