Applying hotfix to ISE

Hello

Today I was hotfixing my ISE because of CSCvm14030 vulnerability. Here’s the output.

The only difference between patching and hotfixing is that the command is „application install” rather than ”patch install”.

I chose to do this manually on all nodes rather than the GUI automatic mode because i wanted to be in control over when I hotfix what node.

myISE/admin# application install ise-apply-CSCvm14030_2.1.0.474_common_1-SPA.tar.gz MyREPO
Save the current ADE-OS running configuration? (yes/no) [yes] ? yes
Generating configuration…
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine…
Unbundling Application Package…
Verifying Application Signature…
Initiating Application Install…

Checking if CSCvm14030_2.1.0.474_common_1 is already applied
– Successful

Checking ISE version compatibility
– Current ISE Version: 2.1.0.474 Patch Version: 7
– Successful

Applying hot patch CSCvm14030_2.1.0.474_common_1
– Taking backup of integrity files
– Taking backup of patch related files
– Running hotpatch wrapper script
– Restarting ISE services

Stopping ISE Monitoring & Troubleshooting Log Collector…
grep: write error
Stopping ISE Monitoring & Troubleshooting Log Processor…
grep: write error
grep: write error
grep: write error
grep: write error
grep: write error
grep: write error
grep: write error
ISE PassiveID Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
ISE PassiveID Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
Stopping ISE Certificate Authority Service…
Stopping ISE EST Service…
ISE Sxp Engine Service is disabled
ISE TC-NAC Service is disabled
Stopping ISE Profiler Database…
Stopping ISE Indexing Engine…
grep: write error
Stopping ISE Monitoring & Troubleshooting Session Database…
ISE PassiveID Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
Stopping ISE Certificate Authority Service…
Stopping ISE EST Service…
ISE Sxp Engine Service is disabled
ISE TC-NAC Service is disabled
Stopping ISE Profiler Database…
Stopping ISE Indexing Engine…
grep: write error
Stopping ISE Monitoring & Troubleshooting Session Database…
Stopping ISE AD Connector…
grep: write error
Stopping ISE Database processes…
Starting ISE Monitoring & Troubleshooting Session Database…
Starting ISE Profiler Database…
grep: write error
Starting ISE Application Server…
Starting ISE Certificate Authority Service…Starting ISE EST Service…
Starting ISE Monitoring & Troubleshooting Log Processor…
Starting ISE Monitoring & Troubleshooting Log Collector…
Starting ISE Indexing Engine…
Starting ISE AD Connector…
Note: ISE Processes are initializing. Use ‚show application status ise’
CLI to verify all processes are in running state.

 

DMVPN NAT mystery

Hello

Today I came across an interesting case, where two out of three tunnels on one spoke (out of 50 spokes in total) were down. The third (working) tunnel was connected to the same two hubs but via mpls.

I checked:

  • tunnel configuration. It was identical in all other 40+ spokes
  • crypto isakmp and ipsec policies. Again, identical.
  • state of crypto: both ISAKMP and IPSEC were up, encrypting and decrypting tunnels

So now it was up to GRE and NHRP to do their job of establishing DMVPN.

When I checked the state of dmvpn on the hub, it showed UP, however, the spokes were showing their state as NHRP. What does it mean? After doing NHRP debug I knew that the spoke wasn’t getting the resolution reply for the two bad tunnels.

Finally, I saw that there were 2 NAT entries for GRE traffic. It was strange, because the traffic was sourced by the same host on the LAN and the destinations were (in both cases) the dmvpn hub.
I looked at the nat translation statement: it was a nat overload to the outside (public) interface.

As GRE doesn’t cooperate well with PAT, I decided to clear the entries, especially that the the source host was Incomplete in the ARP table.

After clearing the nats from the ip nat translation table, the tunnels went up.

What does it teach us? We should make a deny statement for GRE traffic in our NAT ACLs and only use TCP and UDP in nat statements rather than IP.

I’m wondering if the same issue will crash my DMVPN lab or not.

broken CCIE lab with BGP synchronization and OSPF

Hello

We have the following scenario:

Capture

IOU 4 has a static route 10.1.9.9 to IOU 9. It redistributes it into OSPF. IOU4 has OSPF adj to IOU5, IOU5 to IOU2.

IOU2 is a BGP neigh to IOU5 and also to IOU4. IOU4 and IOU5 are route reflector clients of IOU2.

The problem starts when we enable BGP synchronization on R5. What do we know about synchronization? the condition is the route in BGP needs to match the same route in IGP to avoid black holes. where R1>R2>R3 but BGP is only enabled on R1 and R3. R2 causes the black hole.

But having the routes match in BGP and IGP is not the only condition, i’m afraid… let’s have a look at this case:

R5 receives the ospf route to IOU9 from IOU4, but it receives the BGP route to IOU9 from IOU2! The route from the internal routing protocol will not get synchronized with the same route in BGP.

The only fix is to change bgp router-id on IOU2 to be the same as OSPF router-id on IOU4.

 

 

IOU images are buggy

Hello

I’m doing the labs again and so many things are not working out of the box. Today I configured three routers in an NSSA area and one of them insisted on being the translator. The problem was that it wasn’t an ABR. It thought mistakenly that it was connected to the backbone but show ip ospf topology showed status of its backbone area as INACTIVE.

The solution (as always) is to delete the route config and type it in again.

I wonder if the lab images are just as buggy during the exam.

Why vtp pruning is a bad idea

Hello

A while ago I considered implementing VTP with pruning on a client network. But it’s a bad idea because of the following (from Cisco documentation)

(In vtp v3) VTP pruning still applies only to VLANs 1 to 1005, and VLANs 1002 to 1005 are still reserved and cannot be modified.

(…)

VTP pruning is not designed to function in VTP transparent mode. If one or more switches in the network are in VTP transparent mode, you should do one of these:

Turn off VTP pruning in the entire network

Turn off VTP pruning by making all VLANs on the trunk of the switch upstream to the VTP transparent switch pruning ineligible.

There is a funny cornercase scenario related to this: if you have switch A>switch B>switch C, where switch B is a transparent vtp switch while A and C have vtp enabled with pruning, switch C may request (through B) to prune a vlan. Switch A will prune the vlan from the trunk to switch B (which may have needed it!)

In other words: avoid…

Asking the right questions… wifi issues

Hello

I’ve had a spate of cases recently where not even a single command was needed to solve the problem.

Today a client called me about weak/missing wifi signal in a conference room on floor 2. The problem was that i didn’t have any map so i didn’t know what i was looking for. We called a person onsite who was heading to that building. At the time that person was sure that there was (or had been) an access point there.

On arrival, it turned out that the access point in the hallway near the conference room was nowhere to be seen. What the technician did see was a brand new metal ceiling. Apparently that new metal ceiling was fixed up directly underneath the old ceiling. The access point seemed to be trapped between the two ceilings.

When I asked the technician to connect to wifi, i saw his mac address on another AP on floor 1 instead. It seems that weak signal from floor 1 was leaking to floor 2? Again, we had no way of confirming where both APs were.

Summing up, what do you need to solve cases like this:

  • confidence to ask the right question to the right person even if this means that that person needs to walk/drive to the site
  • preferably a map with rooms and APs marked
  • if you have Cisco Prime then you’re golden because you can see AP channel utilization stats etc. etc.

 

Logical thinking and even a little knowledge goes a long way

Hello

I had an interesting problem today. The customer has two lines: MPLS and internet with dmvpn running on both.

Suddenly the voice quality dropped, and we saw that the tunnel over MPLS is down.

The route to the DMVPN hub goes through the mpls line with the IP of the CE router being 172.16.0.1

However, the arp entry for 172.16.0.1 was incomplete. Clearly the line was faulty.

A ticket to to the telecom was raised but the answer was unexpected: you haven’t been using the line for the last seven days.

I was like: what do you mean we’re not using the line! we’re not using the line because it’s DOWN, that’s what it is…but i began to think about it: in every answer there’s a grain of truth…

So i pinged the broadcast 172.16.0.255 and VOILA – 172.16.0.2 responded. But what is 172.16.0.2??? Let’s experiment since the line is down anyway
I changed the static route to go through the .2 address and of course the line was working again.

It turned out that the telecom replaced the line a while ago and changed the ip address of the CE router.

Now my point is here: i only used logic and one simple ip route command. No big deal, yet nobody else has come up with the idea to look for other addresses on the subnet. This is great new for beginners in the IT world: keep your mind open, be on the ball and even limited knowledge can bring great results.