/images/logo.png
A Network Blog by a Network Engineer

Cisco Embedded Event Manager and its usefullness for TAC

A few years ago I had a TAC case open with Cisco.  The case was centered around an unusually high CPU condition on a Cisco 3800ISR series router.  This router was only routing traffic between the Ethernet interfaces, doing some SNAT (stateful NAT), and basic BGP routing – so the CPU should not have been in the 90%+ range.  Well, TAC wanted me to run some commands every 60 minutes 24×7 for a few days to see what was going on.  Now we all now that that is not an easy thing to do, so what I decided to do was grab an existing EEM script from Cisco’s website and modify it to provide all the information that the TAC engineer wanted.
Needless to say, the case was solved about 2 days later when we upgraded the code per their suggestion because of the identification of the bug. 🙂  I am guessing that having a router send you the logs and command output every hour either provided them the information they needed, or more likely, the engineer realized that  – well, you know.
So, for this post I have decided to post the EEM configuration as well as the EEM script in case you ever need to do something similar.  If I can save you some time, I am all for that.  I still use this script today – albeit in a different manner.  I now use it to get “Daily Health Checks” from the routers in the field.  Every day at noon they send me standard output commands so that we can check on them without having to log into each and every one of them.
So, without further typing – here is the script broken down: (to download this script, you can click here – HealthCheck_BLOG.tcl
Router Configuration:
First thing we will need to do is configure the Event Manager variables on the router:
The variables that we will use are:

Cisco IP Router DHCP Server and Client

Configuring DHCP on routers is not something that is normally
done in a production network.  Typically in a production network
DHCP servers are either Windows, Linux, or some other vendors
implementation.  Occasionally you will need to configure a Cisco
device to provided DHCP for initial purposes - say a new location
build-out, a lab network, or even just perhaps for your home
networking.
For this post, I will show you how to:
 - Configure your route to be a DHCP Server
 - Configure your router to be a client
 - Configure DHCP Reservations for a specific client
 - Release / Renew your DHCP lease
 - Configure DHCP options (150 for example)
Here is the network information we are going to use:
 Network   - 192.168.1.0/24
 Gateway   - 192.168.1.1
 DNS       - 192.168.1.2 and 192.168.1.3
 Domain    - FryGuyLab.Net
 Cisco UCM - 192.168.1.5 and 192.168.1.6
 WINS      - 192.168.1.2
 Node Type - Hybrid

So, first lets setup the DHCP Server router (R2)
Rack1R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Rack1R2(config)#ip dhcp pool DHCP_POOL_1
Rack1R2(dhcp-config)#network 192.168.1.0 /24
Rack1R2(dhcp-config)#default-router 192.168.1.1
Rack1R2(dhcp-config)#dns-server 192.168.1.2 192.168.1.3
Rack1R2(dhcp-config)#domain-name FryGuyBlog.Net
Rack1R2(dhcp-config)#option 150 ip 192.168.1.5 192.168.1.6
Rack1R2(dhcp-config)#netbios-node-type h-node
Rack1R2(dhcp-config)#netbios-name-server 192.168.1.2
Rack1R2(dhcp-config)#exit

Now, don't forget to exclude addresses if necessary. 
Here will exclude 192.168.1.1 - 192.168.1.10
Rack1R2(config)#ip dhcp excluded-address 192.168.1.1 192.168.1.10
Rack1R2(config)#
Now we can go over to R1 and configure DHCP and then enable
the G0/0 interface on the router.
Rack1R1#show run int g0/0
Building configuration...
Current configuration : 103 bytes
!
interface GigabitEthernet0/0
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
end

Now we can configure the interface:
Rack1R1(config)#int g0/0
Rack1R1(config-if)#ip address dhcp
Rack1R1(config-if)#no shut
Rack1R1(config-if)#exit
Rack1R1(config)#exit
Rack1R1#
*Nov 15 18:37:55.302: %SYS-5-CONFIG_I: Configured from console
 by console
*Nov 15 18:37:56.102: %LINK-5-CHANGED: Interface GigabitEthernet0/0,
changed state to reset
Rack1R1#
*Nov 15 18:37:58.866: %LINK-3-UPDOWN: Interface GigabitEthernet0/0,
changed state to up
*Nov 15 18:37:59.866: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet0/0, changed state to up
Rack1R1#
*Nov 15 18:38:05.490: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0
 assigned DHCP address 192.168.1.13, mask 255.255.255.0, hostname Rack1R1
As you can see, we received a DHCP address of 192.168.1.13/24 from the
server. To check the DHCP Lease information on the client, issue the
show dhcp lease command
Rack1R1#show dhcp lease
Temp IP addr: 192.168.1.13  for peer on Interface: GigabitEthernet0/0
Temp  sub net mask: 255.255.255.0
 DHCP Lease server: 192.168.1.1, state: 3 Bound
 DHCP transaction id: 859
 Lease: 86400 secs,  Renewal: 43200 secs,  Rebind: 75600 secs
Temp default-gateway addr: 192.168.1.1
 Next timer fires after: 11:58:27
 Retry count: 0   Client-ID: cisco-001c.f6e6.6a90-Gi0/0
 Client-ID hex dump: 636973636F2D303031632E663665362E
 366139302D4769302F30
 Hostname: Rack1R1

Now to check the connectivity, lets PING the other router:
Rack1R1#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Rack1R1#
To check the active leases on the server, issue the sh ip dhcp
binding command:
Rack1R2#sh ip dhcp binding
Bindings from all pools not associated with VRF:
IP address          Client-ID/              Lease expiration        Type
 Hardware address/
 User name
192.168.1.13        0063.6973.636f.2d30.    Nov 16 2010 06:25 PM    Automatic
                    3031.632e.6636.6536.
                    2e36.6139.302d.4769.
                    302f.30
Rack1R2#
Now for reservations. 
We will configure the DHCP server (R2) to assign 192.168.1.254/24
 to R3's F0/0 interface.
On the DHCP Server (R2) - configure a special DHCP pool just
for that client:
Rack1R2#conf t
Rack1R2(config)#ip dhcp pool R3
Rack1R2(config)#host 192.168.1.254 255.255.255.0
Rack1R2(config)#client-identifier 0100.0bfd.5661.e0

To find out the Cleint Identifier to use, on R3 look at the Fa0/0
MAC address and then pre-pend 01 to the mac address. 
(see IETF RFC1700 Hardware table below)
Rack1R3#sh int f0/0
FastEthernet0/0 is up, line protocol is up
 Hardware is AmdFE, address is 000b.fd56.61e0
On R3, configure the Fa0/0 interface as follows:
Rack1R3(config)#interface FastEthernet0/0
Rack1R3(config)#ip address dhcp client-id FastEthernet0/0
Rack1R3(config)#no shut
Rack1R3(config)#exit
Rack1R3#
You will then see:
*Mar 13 23:29:28.142: %DHCP-6-ADDRESS_ASSIGN: Interface FastEthernet0/0
assigned DHCP address 192.168.1.254, mask 255.255.255.0, hostname Rack1R3
And on R2 you will see under the DHCP binding:
Bindings from all pools not associated with VRF:
IP address          Client-ID/              Lease expiration        Type
 Hardware address/
 User name
192.168.1.13        0063.6973.636f.2d30.    Nov 16 2010 06:25 PM    Automatic
 3031.632e.6636.6536.
 2e36.6139.302d.4769.
 302f.30
192.168.1.254       0100.0bfd.5661.e0       Infinite                Manual
Rack1R2#

IETF Hardware Table from RFC1700
http://www.ietf.org/rfc/rfc1700.txt
Number Hardware Type (hrd)                           References
------ -----------------------------------           ----------
 1 Ethernet (10Mb)                                    [JBP]
 2 Experimental Ethernet (3Mb)                        [JBP]
 3 Amateur Radio AX.25                                [PXK]
 4 Proteon ProNET Token Ring                          [JBP]
 5 Chaos                                              [GXP]
 6 IEEE 802 Networks                                  [JBP]
 7 ARCNET                                             [JBP]
 8 Hyperchannel                                       [JBP]
 9 Lanstar                                             [TU]
 10 Autonet Short Address                             [MXB1]
 11 LocalTalk                                         [JKR1]
 12 LocalNet (IBM PCNet or SYTEK LocalNET)             [JXM]
 13 Ultra link                                        [RXD2]
 14 SMDS                                              [GXC1]
 15 Frame Relay                                        [AGM]
 16 Asynchronous Transmission Mode (ATM)              [JXB2]
 17 HDLC                                               [JBP]
 18 Fibre Channel                            [Yakov Rekhter]
 19 Asynchronous Transmission Mode (ATM)      [Mark Laubach]
 20 Serial Line                                        [JBP]
 21 Asynchronous Transmission Mode (ATM)              [MXB1]
Now, if you need to force release an IP address on a client, you can issue the command:
Rack1R1# release dhcp g0/0

If you need to renew the address, you can use the command
Rack1R1# renew dhcp g0/0

IP Expert CCIE SP Class last week

Well, finally had a chance to take a Cisco CCIE Service Provider class last week – and I am very glad I did.  I know I was not ready for the lab when I took it the other month, and it was my hope that this class would help me fill in the pieces.  I have to say, I feel much better about what I know and what I need to brush up on.
Many of the more complex topics are now much easier to understand and configure. The way that Marko explained and diagramed the concepts was wonderful.  He knows his stuff, and what really helped is that he has some real-world experience in deploying these technologies.  He is able to articulate them into a real-world deployment, so understanding the concept (Multicast for example) makes much more sense when you can relate it to something you understand.  It was a loooooong week of class with few breaks.
Each day was about 12 hours long – some lecture some labbing.  The lecture was not “power-point waterboarding” but more of interactive communication and understanding.  He would take a topic, explain it in lecture, and then actually configure it on the screen.  We broke configs, tweaked configs, and even reinterpreted things in order to see if we could get them work in alternate ways (when the possibility existed).  After he would show us the topic and configs, it was then up to us to do our labs for a few hours.
At first these labs looked easy, but you quikly realized that they where not.  You had to really read and understand what they where asking.  What is nice about that is that is how the real test is rumored to be.  At first it might look easy, but you need to understand it to realize what they are asking.  What was also nice about this was that it really helped you learn and understand a topic and topology.  You needed to understand the interactions between the technologies as to not break something you have already done.
The final two labs you get to do are 8-hours (yeah right) labs. These labs are close to what you can expect on the real lab  in the ways of complexity and such, so they are a good guage at your understanding of the topics.  I did not finish my first lab during class and, unfortunately, had to skip out on the second one (Friday).  I would have loved to spend the day there doing the lab but instead chose to get home to family.  Since I have the lab workbook from class, I will so do the second lab one this week and see what happens.
All in all I am very glad I was able to attend the class, I learned alot, but best of all – met some great people who also have the same goal to pass the SP lab.
 
 
 
 
 
 
 

off to class!

Well, today is the day. I am heading out to Columbus, OH to attend a CCIE SP bootcamp class hosted by IPExpert. Looking forward to this as I know there are some things I need to learn.
I am not taking the CCIE SP test just to get “another CCIE” but to actually learn more about the technology and how I can apply it to the place I work at.

CCIE SP Bootcamp scheduled

Been a little quiet lately, trying to brush up on things and get ready for class next week.  I have a CCIE Service Provider bootcamp scheduled with IPExpert on 11/8-12 in Columbus, OH. Really looking forward to the class, especially since Marko (@icemarkon on Twitter) is one of the teachers.
See what happens!  I will post my thoughts and reflections once the class is complete.
 
 
 

Multi-VRF redistribution (a.k.a. route leaking between VRFs)

I was trying to figure out how to redistribute routes between some VRFs for a project that I am working on.  I won’t get into details of the project or specifics, but ultimately the goal was to be able to leak routes between the VRFs so that we can gain access to the management ports on some devices.  Since VRFs are separate routing tables that co-exist within the same router and cannot see each other, getting communication between them can be difficult at best.
Using some the features of PE routers, you can overcome this limitation by using BGP to redistribute between them.
Below is the diagram that we will be using for this config.  What we will do is configure R3 to redistribute routes between VRF RED and VRF BLUE to enable connectivity.

R7 and R8 are just basic configs like you would do on an ordinary router
Basic R7 Configuration:
!
interface Loopback0
ip address 10.1.7.7 255.255.255.0
!
interface FastEthernet0/0
ip address 10.1.37.7 255.255.255.0
!
router eigrp 100
network 10.0.0.0
no auto-summary
!
Basic R8 Configuration:
!
interface Loopback0
ip address 10.1.8.8 255.255.255.0
!
interface FastEthernet0/1
ip address 10.1.38.8 255.255.255.0
!
router eigrp 200
network 10.0.0.0
no auto-summary
!
Now when it comes to R3, we need to be a little creative.  Here we will place the interfaces in the appropriate VRFs and configure EIGRP and BGP.
Note:  Fa0/0 will be in VRF BLUE and Fa0/1 will be in VRF RED
R3 Configuration items:
We will define VRF RED and BLUE here and also assign it an RD (route distinguisher) to ID these routes
Link – http://en.wikipedia.org/wiki/Route_distinguisher
VRF BLUE will have a RD of 1:1 and VRF RED will have a RD of 1:2.
The command route-target will allow us to import and export based on the RD assigned to a given network.
ip vrf BLUE
rd 1:1
route-target export 1:1
route-target import 1:1
route-target import 1:2
ip vrf RED
rd 1:2
route-target export 1:2
route-target import 1:2
route-target import 1:1
Next step is to configure the interfaces in the appropriate VRFs
interface FastEthernet0/0
ip vrf forwarding BLUE
ip address 10.1.37.3 255.255.255.0
__
interface FastEthernet0/1
ip vrf forwarding RED
ip address 10.1.38.3 255.255.255.0
Since this router will be running EIGRP under different VRFs, we will need to great a top-level AS and then get into the address-family commands to configure the VRF EIGRP AS
router eigrp 1
no auto-summary
!
address-family ipv4 vrf RED ! <- This will allow us to configure EIGRP for VRF RED
__network 10.1.38.3 0.0.0.0
no auto-summary
autonomous-system 200 !<- This is the EIGRP AS number for VRF RED
exit-address-family
!
address-family ipv4 vrf BLUE !<- This will allow us to configure EIGRP for VRF BLUE
__network 10.1.37.3 0.0.0.0
no auto-summary
autonomous-system 100 !<- This is the EIGRP AS number for VRF BLUE
exit-address-family
!
The next step is going to be to configure a lone BGP session where we can redistribute and place the routes in a VPNV4 environment.  Basically we are creating a lone SP network where VPNV4 routes are “exchanged”.  You do not need a neighbor for this, just a BGP session configured and IPV4 VRF families.
router bgp 1
no synchronization
bgp log-neighbor-changes
no auto-summary
!
address-family ipv4 vrf RED !<- Here we will configure BGP for VRF RED and redistribute routes learned from EIGRP in
redistribute eigrp 200
no auto-summary
no synchronization
exit-address-family
!
address-family ipv4 vrf BLUE !<-  Here we configure BGP for VRF BLUE and then redistribute routes learned from EIGRP in
redistribute connected
redistribute eigrp 100
no auto-summary
no synchronization
exit-address-family
!
Lastly we will need to configure EIGRP to redistribute BGP learned routes:
router eigrp 1
!
address-family ipv4 vrf VRF2
redistribute bgp 1 metric 1 1 1 1 1500
exit-address-family
!
address-family ipv4 vrf VRF1
redistribute bgp 1 metric 1 1 1 1 1500
exit-address-family
Once you have done that, R7 can now ping R8

Static ARP entries on NX-OS

Oh the joys of Microsoft Network Load Balancing (NLB).  One of the wonderful ways that it is usually configured is using a Multicast MAC address utilizing a Unicast IP address.  Needless to say, most network devices do not like that and thus have problems when devices outside the local LAN attempt to connect to the NLB address.  In order to work around this problem you need to configure a static ARP entry on the gateway so that the devices know how to get to the server.
On IOS, this is not so hard as you configure the a static ARP entry (arp 10.1.1.99 0300.5e11.1111).  That command there on the gateway will usually get things working. You may also need to define where that MAC address lives – and the command to do that would be mac-address-table static 0300.5e11.1111 vlan 200 interface fa2/3 fa2/4 This information was obtained from a Cisco document can be found here if you need more information – http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml
Now, on the NX-OS it is a little different as you need to configure the static ARP entry under the VLAN interface in order for it to work:
interface Vlan100
no shutdown
description [—-[ Standard VM VLAN ]—-]
no ip redirects
ip address 10.1.1.2/24
ip arp 10.1.1.99 0300.5e11.1111
N7K1# sh ip arp | inc 1111
10.1.1.99       –      0300.5e11.1111   Vlan100

Explanation for the lack of posting lately…

Well, just wanted to post up what has been g0ing on and the lack of updates.
On Monday (10/4) I sat the CCIE SP lab in RTP  – results I did not pass.  I was not ready, close but need to fill in some gaps. Based on that attempt, I am going to spend some quality time in a training class next month at IPExpert.  Hopefully Marco can fill my head with the knowledge that I need 🙂
Add to that we are going on our yearly trek to Disney on Saturday, things have been busy.
 
I have not abandoned this blog, nowhere near that – just been a little busier with life lately.
 
Jeff

Still here, just busy. . .

I am still here, just been a little busy for the past few days.  Took some time off from work to work on the house, that is all.
I am planning to blog some things on 10Gbps cabling and associated GBICs soon as well as some more Nexus stuff in the coming month.  If there is anything that you would like me to blog about, please do not hesitate to suggest – I am here to help you, I have the opportunity to learn more as well.
Thanks,
Jeff