Today I learnt that you can easily make a network loop on a Dell server with an embedded switch, causing a network outage for the whole office.
The embedded switch has a few physical external ports and a lot of logical internal backplane ports. If you use port mirroring on the external port but you send the mirrored frame also on the external port, the copied (mirrored) frames will multiply like crazy, because the frame that wants to go out does go out but is first copied. Its copy also wants to go out is is first copied etc. etc. One frame quickly becomes a billion frames that go out to your LAN.
Another example of how you can cause a loop by using SPAN is by ”faking” RSPAN:
As you might remember, last month before the demo I had a weird problem where my Gembird power strip application was malfunctioning. In short, the power strip connects to the Internet, and on my cell phone I configured an application where I can remotely switch on/off sockets on the power strip. The problem was that for some reason I could only connect to the power strip when i was on the same WLAN. Otherwise, the application said that the device was offline about 30 minutes after I would leave home. I thought that it was an application/faulty unit problem until today. I have 4 of those power strips so i swapped the faulty one and configured another gembird. I logged out from my wifi and used normal Internet access on my phone. Sure enough, the app died after an hour. But I noticed a weird thing: the app started working again as soon as i connected to the same wifi that the power strip was on! So I figured it wasn’t a problem with the connection between the power strip and the cloud server, but rather between my phone and the cloud server! It appears that the connection is to the TCP port 5000 of the Gembird cloud server, which i guess is blocked by the ISP. However, if I use wifi at home to use the gembird app, the status is OK.
I wouldn’t actually be able to figure this out if it weren’t for the problem I had at work recently, where Meraki VPN site to site (their ”punching UDP holes” vpn where nobody has a public IP address) would only work on the landline internet, but it would fail if operating in 4g mode.
Internet access offered by mobile operators is not full (they only proxy some ports or simply block certain ports) or broken due to the way they do NAT. And I figure that IPv6 will solve all such problems.
- Get a cheap ICND1 Cisco course (chris bryant’s courses at udemy.com rock and they can be dirty cheap if there’s a good deal on). The current url is https://www.udemy.com/ccna-on-demand-video-boot-camp/
- Get a book with Cisco labs, e.g. https://www.amazon.com/101-Labs-Cisco-CCNA-Exam/dp/0955781523
- Get 4 cheap routers (2801 or 1841) and Rj45 cables at ebay. A good price is $50 a piece
- Get 3 cheap switches (3550 or 3560) at ebay. A good price is $40 for 3550 or $60-70 for 3560
- Get a Cisco console cable and DB9>USB adapter at ebay
- Start watching the ICND1 course. Try to practise everything you see there on your equipment.
- After you finish watching the course, do the labs from the book.
- Take the 100-105 Cisco exam (go to pearson vue cisco website and book the exam at your nearest exam center) – this step is optional but Cisco certificates can guarantee at least a job interview.
- Congrats! Now get an ICND2 book and repeat the process.
This will set you back around $700 ($150 for the vids and books, $400 for the gear and $150 for the exam), or even less if you get a good bargain at ebay/allegro.
Alternatively, book an ICND1 course at www.humanity.pl. This ensures that:
- you get 80 hours of learning, this includes lectures and workshops. Classes are delivered by an experienced network engineer
- you get 60+ hours of access to our equipment when you go home after the classes
- it’s less expensive than buying the equipment and books
- you know exactly what to do at each step
- if you come across a difficult problem during or after the course (we provide a free 3month post-course email troubleshooting service), you can ask the trainer
- you are thoroughly prepared for the exam
What do I mean by ”weird problems”? Some websites don’t load fully, you can access network shares but cannot actually open the files, your teamviewer sessions are suddenly disconnected etc. Mind you, lowering MTU on endhosts is not really a good solution because then you need to do it on all your endhosts. Typically, you will lower the MTU on your router, but what if you cannot access your router or the router doesn’t have the option to change the MTU size?
You typically have this problem when on a PPPoE connection, if MTU on the router is set to 1500 instead of 1492. However, it seems that UPC customers have this problem and lowering the MTU on the machines helps, too, because it makes more space for any headers and administrative overhead that is necessary to make protocols work.
Now why does lowering the MTU size help? Because endhosts can fragment a large segment of data better than intermediary routers. Some routers will just do a bad job of fragmenting/assembling the packet. But if the endhost sends packets that are small enough not to be fragmented anywhere on the path from host A to host B, nobody needs to do any fragmentation at all.
Unbelievably, there is a bug on Nexus 7000 that can prevent you from applying any ACLs to the interface. The result is that the following command gives TCAM allocation failures.
ip access-group MYACL in
The solution is to use the following command:
hardware access-list resource feature bank-mapping
I was quite proud of the fact that I found the solution in 20 minutes.
I’ve found a number of times that if you have a crypto map that is applied to an interface, changes made to the transform-set that is applied to this interface are not applied instantaneously. So e.g. you have this crypto map:
crypto map MYMAP ipsec-isakmp 10
set peer 18.104.22.168
set security-association lifetime kilobytes 4608000
set security-association lifetime seconds 3600
set transform-set MYSET
set pfs group5
match address MY_INTERESTING_TRAFFIC
Now if you change MYSET, your router may still send out the old MYSET. Solution no1:
- be patient, wait 15 minutes
- clear the crypto map, apply it again
- shut and unshut the interface
So if you’re troubleshooting a broken S2S VPN, make one change, wait 15 minutes, check, if it’s still not working only THEN make another change.
Today I spent 80 minutes in a troubleshooting session with an engineer from the remote end trying one thing after another and nothing worked. We ended the session and set up another one for the next day. I reverted the config to the original settings + made 1 change that should be ok (but it still wasn’t ok! ), went shopping, came back, and it just worked!
Reading through the new CCDP book, I came across an interesting fact. Turning off trunk mode negotiaion (switchport mode trunk) and DTP (switchport nonegotiate) can reduce the link transition time by 2 seconds!
Further, it is good to put your most important vlans as low-number vlans, because they are first to come up. This can save probably miliseconds rather than seconds but it’s still a timesaver.
I still have about 2 months to prepare for CCDP and everything is going as per plan. I’m about 2/3 into the CCIE R&S course ( partly to revise for CCDP, partly to prepare for CCIE R&S written), I’ve started reading the CCDP official guide. I may revisit the cbtnuggets CCDP course, too, because Jeremy has added some new units after the CCDP exam changed. If I still have time, I may read some CVDs (Cisco Validated Designs), but I don’t want to actually overprepare because 60-90 minutes a day is the absolute maximum I can do these days without getting fed up with my learning/life ratio.
Now the end of 2016 is drawing near so here’s a short summary:
- I’ve passed 5 exams (CCDA, JNCIS-ENT, SIMOS, SISAS, SITCS)
- I’ve changed my job and it turned out to be a good move
- I’ve found a partner for my business and in the last 4 months we’ve managed to push forward quite a bit (website, racks, leaflet design etc.), we’re in a discussion with a major IT player to be a technology partner for our courses
- I’ve had some interesting experiences like the 4 week stay in Oslo to help their local network provider
All in all, a really good year!
And here’s a tentative schedule for 2017:
- CCDP exam in late February (edit Feb 19: PASSED!!!)
- first ad campaign starting in mid March
- first network courses in mid April
- e-learning platform launch in May/June
- remote rack rental service launch in July
- 2nd run of network courses in September
- CCIE R&S written in October/November
- first CCIE lab attempt in December/January 2018
I’m really excited about this plan because i’ve never had so much stuff happening both in my personal and work life. I guess that I’ll be over the moon if I can do 75%. Anything above will be a tremendous achievement.
I’ve had a handful of funny experiences lately with people trying to ask me difficult questions so that I can prove that my CCNP was actually deserved.
Frankly, I don’t get it: where does that come from? I mean, it’s not like I put CCNP on my CV to show off, and even if I was showing off, isn’t your CV the right place where you can say what you’ve achieved?
So, yes – I’m proud of my CCNP, and no – it doesn’t make me an expert. I don’t think any certificate makes anyone an expert because it’s just a piece of paper. But boy is getting that piece of paper fun! To me, these exams are like checkpoints where you can measure how much time and effort it took to get to a certain level. I don’t print my certs and put them up on my wall, i don’t put them in my email footers, but why shouldn’t I put this stuff on my CV?
I will put my CCIE up on every wall in my apartment when I pass it, though. Just kidding. I won’t.
I might will.
I think I will.
Today I spent like whole day connecting rack 1. I made an absolute mess with the cabling because I was in a hurry but other than that the rack is ready. I had a bit of a setback with the LAN power strip because it failed right after I came back home so I’m gonna have to replace it before the demo on Thursday but it looks like I will be able to use the rack during the demo!
The next step is to set up VPN on the ASA firewall + install a PC that will serve as a monitoring station (logs, tftp, snmp, etc.).
The side effect of today’s work is that I have enough stuff for Rack3 – I bought some more routers in November and I unpacked them today.
The big plan is to start selling a rack rental service in January. It would be a pity to have a pile of equipment turned off 80% of the time.