7010 - Tag - Fryguy's Blog

Nexus 7009 – More information

Fryguy — Tue, 19 Jul 2011 12:00:38 +0000

A few months ago I did a quick post on the Nexus 7009 when it was pseudo announced (here ). When I was attending Cisco Live 2011 last week in Las Vegas I found out that they had a Nexus 7009 in their Data Center booth in World of Solutions for all to see. So, I figured I would post some updated information that I was able to gather or figure out. All this information was being openly spoken in the booth, nothing here was gathered from any other resource then what I was able to hear. To recap, below is a picture of the new family member, the 7009 along side its big brothers the 7010 and the 7018. As you can see there are some nice size differences. The Nexus 7009 us 14RU, the 7010 21RU, and the 7018 25RU. [][2]

Cisco Live 2011 Schedule

Fryguy — Wed, 20 Apr 2011 15:22:21 +0000

Figured I would put up a quick post with my current Cisco Live class schedule. This is my schedule as of today and, as always, is subject to change. All times are in Vegas time!

Saturday
	Fly and check in at Mandalay Bay as well as register for the event

Sunday
	8:00 – 17:00	TECDCT-8001	Next Generation Data Center Infrastructure

Monday	9:30 – 11:30	BRKARC-3470	Cisco Nexus 7000 Switch Architecture
	12:30 – 14:30	BRKARC-3471	Cisco NXOS Software – Architecture
	15:00 – 17:00	BRKRST-2335	IS-IS Network Design and Deployment

Tuesday	8:00 – 9:30	BRKRST-3045	LISP – A Next Generation Networking Architecture
	10:00 – 11:00	GENKEY-4700	Keynote and Welcome Address
	12:30 – 14:30 15:00 – 15:30	BRKCOM-1005 GENNOC-9190	UCS Systems Architecture Overview Cisco Live NOC Tour (CCIE/NetVet)
	16:00 – 18:00	BRKCRS-3144	Troubleshooting Cisco Nexus 7000 Series Switches

Wednesday
	8:00 – 10:00	BRKCOM-2006	UCS Reference Architecture for Enterprise Applications
	10:30 – 11:30	GENKEY-4701	Cisco Technology Keynote
	12:30 – 14:30	BRKDCT-2121	Virtual Device Context (VDC) Designing and Implementation Considerations with Nexus
	16:00 – 18:00	BRKVIR-3013	Deploying and Troubleshooting the Nexus 1000V virtual switch

Thursday
	8:00 – 10:00	BRKCOM-1002	Data Center Architectures and Virtual Private Data Centers with UCS
	10:30 – 11:30	GENDCT-4642	Town Hall: Data Center
	12:00 – 14:00	BRKARC-3472	NX-OS Routing & Layer 3 Switching
	14:30 – 15:30	GENKEY-4702	Closing Keynote: William Shatner
	16:00 – 17:30	BRKMPL-2108	Global WAN Redesign Case Study

Friday
	Fly home!

The Nexus 7009 – and what is this, Nexus 7006?!??!?!?!

Fryguy — Mon, 04 Apr 2011 12:00:52 +0000

Since Cisco posted that picture the other day, you know this one:

Well, since that picture was posted there has been some buzz around the chassis in the twitter feeds. Not much is officially know about this box – but the picture above proves it does exist. Not only that, I recall seeing a picture from Cisco Live 2011 – London where the EMC booth had one of these Nexus 7009 looking switches in their booth. Well here is some additional information that I have located using the assumed part number of N7K-C7009
On Cisco’s website they have a MIB posted called CISCO-ENTITY-VENDORTYPE-OID-MIB.my (clicking on that MIB will allow you to view/download it – original link here ). From what I can gather in the MIB – the Nexus 7009 will have the new Fabric-2 cards, no Fab-1 cards are even listed for this chassis. When you search the MIB, you can find the following information:
cevChassisN7Kc7009              OBJECT IDENTIFIER ::= { cevChassis 932 } — N7K-C7009 nexus-9-slot chassis
cevBackplaneN7Kc7009                    OBJECT IDENTIFIER ::= { cevBackplane 57 }       — MosPort9 N7K-C7009 Nexus-9-slot-backplane
cevFanN7Kc7009FanTray           OBJECT IDENTIFIER ::= { cevFan 129 } — N7K-C7009-FAN Trinacria-fan-nexus9slot
cevN7Kc7009Fab2         OBJECT IDENTIFIER ::= { cevModuleN7KType 14 }           — dijon9 N7K-C7009-FAB2 Fabric for Nexus7000 9slot boxster
Bonus information contained within the MIB is information on the, yet unannounced, Nexus 7006!
Granted, this is only speculation and such – but the MIB information matches what the 7009 has as well. Only time will tell if this is true.
cevChassisN7Kc7006              OBJECT IDENTIFIER ::= { cevChassis 1054 } — Nexus7000 6slot elsie n7k chassis N7K-C7006
cevBackplaneN7Kc7006                    OBJECT IDENTIFIER ::= { cevBackplane 60 }       — Nexus7000 6slot elsie n7k backplane N7K-C7006
cevFanN7Kc7006FanTray           OBJECT IDENTIFIER ::= { cevFan 147 } — N7K-C7006-FAN fan for nexus 6slot-chassis
I also did some searching and found this list of Nexus 7009 Part Numbers, and my assumed descriptions (in blue) of what they are.
N7K-C7009-ACC-KIT    Nexus 7009 Accessory Kit
N7K-C7009-BSK             Nexus 7009 Bottom Support Kit
N7K-C7009-CAB-TOP    Nexus 7009 Top section Cable Management?
N7K-C7009-CM-BLK      Nexus 7009 Cable Management Blank? Unknown
N7K-C7009-F-BLANK Nexus 7009 Fabric Blank Interface ?
N7K-C7009-FAB-2        Nexus 7009 Fabric 2 Card
N7K-C7009-FAN            Nexus 7009 Fan Tray
N7K-C7009-FD-MB       Nexus 7009 Front Dook Kit
N7K-C7009-L                  Nexus 7009 License
N7K-C7009-RMK           Nexus 7009 Rack Mount Kit
N7K-C7009-SHPPKG    Nexus 7009 Shipping package
N7K-C7009-XL       Nexus 7009 XL
L-N7K-C7009-XL           Nexus 7009 Scalable Feature License (allows XL featuers without requiring a hardware module change)
Another part-number that I have found that is NOT referenced on the Cisco site is this 5.6KW power supply. Wonder if the 7009 can support a smaller power supply, or this is for the Nexus 7006
N7K-AC-5.6KW         5.6kW AC Power Supply

Nexus 7000 and the Show Tech command (gzip too)

Fryguy — Tue, 25 Jan 2011 13:18:24 +0000

Recently I have experience some interesting first-level support from our Cisco VAR. They are not too familiar with the Nexus 7000 and insisted on us sending them a SHOW TECH from the switch. If you are not familiar with a Nexus 7000, let me tell you – the show tech from this box can be over 100 Megs! So, when they asked for that I was like – are you sure? really really sure? I just have to say – thank you DropBox and public folders! I used DropBox place the file and then send the support the public link. Not sure that most e-mail systems would appreciate that large of a file!
So, if the Show Tech Support is so big, how do you get the information that support needs? Well, instead of running a complete show tech, you run it for the sections you need!
Here is a list of what you can append to a Show Tech-Support command: (show tech-support option)
N7K1-CoreSwitch1# sh tech-support ?

>               Redirect it to a file
»              Redirect it to a file in append mode
aaa             Display aaa information
aclmgr          ACL commands
adjmgr          Display Adjmgr information
arp             Display ARP information
ascii-cfg       Show ascii-cfg information for technical support personnel
assoc_mgr       Gather detailed information for assoc_mgr troubleshooting
bgp             Display BGP status and configuration
bootvar         Gather detailed information for bootvar troubleshooting
brief           Display the switch summary
callhome        Callhome troubleshooting information
cdp             Gather information for CDP trouble shooting
cert-enroll     Display certificates information
cfs             Gather detailed information for cfs troubleshooting
cli             Gather information for parser troubleshooting
clis            Gather information for CLI Server troubleshooting
commands        Show commands executed as part of show tech-support commands
details         Gather detailed information for troubleshooting
dhcp            Gather detailed information for dhcp troubleshooting
eem             Show EEM tech-support information
eigrp           Display EIGRP status and configuration
eltm            Eltm debug info
ethpm           Gather detailed information for ETHPM troubleshooting
forwarding      Forwarding debug information
ha              Gather detailed information for HA troubleshooting
hsrp            Show hsrp tech-support information
icmpv6          Display Icmpv6 information
im              Gather detailed information for IM troubleshooting
include-time    Gather tech-support and capture time taken to execute each
command
interface-vlan Gather detailed information for interface-vlan
troubleshooting
internal        Gather internal info for troubleshooting
ip              Display IP information
ipqos           IP QoS Manager
ipv4            Display IP information
ipv6            Display IPV6 information
l2fm            L2fm debug info
l2pt            Gather information for l2pt troubleshooting
l3vm            Display VRF information
lacp            Gather detailed information for LACP component
license         Display licensing information
logging         Show information on logging for technical support staff
m2fib           Gather detailed information for M2FIB troubleshooting
m2rib           Gather detailed information for M2RIB troubleshooting
mfwd            Display MCASTFWD status and configuration
module          Gather info related to a module
monitor         Gather detailed information for monitor troubleshooting
multicast       Display V4 Multicast information
netflow         Show NetFlow tech-support information
netstack        Gather information for NETSTACK troubleshooting
npacl           Display npacl information
pixm            Gather detailed information for PIXM troubleshooting
pixmc           PIXMC Information
pktmgr          Display Packet Manager information
port-channel    Gather detailed information for port channel troubleshooting
port-profile    Gather information for troubleshooting port-profiles
port-security   Port security related command
Ok, color code again:
Red – Router Output
Blue – Commands
Green – Notes
Traditionally when we have done show tech-support, we have always done terminal length to 0, set the capture buffer on our terminal application, and then executed the command – like this:
First we set the length to 0
FryGuyR1#terminal length 0
Then we configure our terminal to capture the data

Static ARP entries on NX-OS

Fryguy — Thu, 28 Oct 2010 13:55:10 +0000

Oh the joys of Microsoft Network Load Balancing (NLB). One of the wonderful ways that it is usually configured is using a Multicast MAC address utilizing a Unicast IP address. Needless to say, most network devices do not like that and thus have problems when devices outside the local LAN attempt to connect to the NLB address. In order to work around this problem you need to configure a static ARP entry on the gateway so that the devices know how to get to the server.
On IOS, this is not so hard as you configure the a static ARP entry (arp 10.1.1.99 0300.5e11.1111). That command there on the gateway will usually get things working. You may also need to define where that MAC address lives – and the command to do that would be mac-address-table static 0300.5e11.1111 vlan 200 interface fa2/3 fa2/4 This information was obtained from a Cisco document can be found here if you need more information – http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml
Now, on the NX-OS it is a little different as you need to configure the static ARP entry under the VLAN interface in order for it to work:
interface Vlan100
no shutdown
description [—-[ Standard VM VLAN ]—-]
no ip redirects
ip address 10.1.1.2/24
ip arp 10.1.1.99 0300.5e11.1111
N7K1# sh ip arp | inc 1111
10.1.1.99 – 0300.5e11.1111 Vlan100

What is the difference between the Nexus 7010 and 7018?

Fryguy — Thu, 16 Sep 2010 00:21:14 +0000

I was chatting with one of my old co-workers the other day and he mentioned that most people (Exec Level) do not always understand the difference in the Nexus 7010 and 7018 switches. So, based on that chat conversation I figured I would just post up some information on the similarities and the differences between the two.
First difference, size and weight. The Nexus 7010 is 36.5″ (92.7 cm) tall and can weight up to 516 lb (235 kg) whereas the Nexus 7018 is 43.5″ tall (110.5 cm) and can weigh up to 696 lb (316 kg) fully loaded. That equates to the Nexus 7010 being about 21RU and the 7018 25RU. Both are also the full depth of a standard 4-post rack – so you need to be aware of quite a bit when designing the space layout around them. Below is an image from Cisco’s web site ( http://www.cisco.com/en/US/products/ps9402/index.html ) that shows the 7018 on the left and the 7010 on the right.

The next difference in them is the slot capacity and performance. The Nexus 7010 is a 10-slot chassis but slots 5 and 6 are reserved for Supervisors, so the real capacity is 8 line cards. In the 7018 you have an 18 slot chassis and slots 9 and 10 are reserved for supervisors, thus providing 16 available slots for line cards. With the currently available supervisors and cards, the Nexus 7010 can handle 480 Mpps and the 7018 can support twice that, 960 Mpps. In both the Nexus 7010 and 7018 chassis you can have up to 5 fabric modules installed, but there is a difference between the 7010 and the 7018 the cards. In the 7010, the cards are in a vertical mounting whereas in the 7018 they are in a horizontal mount. This is one of the only cards that I know which is unique to each chassis. All the other line cards (supervisors, 1g, 10g, etc) are interchangeable between the two.
When you look at the Nexus 7010 and the line cards, in order for the fabric card to “touch” all the line cards, it would have to be mounted horizontally. In the 7018, the line cards are horizontal – so the fabric card needs to be vertical. It is just how the backplane connections work in these chassis’s. At least, that is my understanding of why they are different.
The Nexus 7018 (N7k-C7018-FAB-1) is pictured on the left and the 7010 Fabric module (N7K-C7010-FAB-1) is on the right.

Like the fabric cards that are found on the back, the fan trays that are found on the back are unique to each switch. The Nexus 7018 fan tray (N7K-C7018-FAN) is larger and contains more fans as the airflow on that chassis is side-to-side, but the 7010 fan tray (N7K-C7010-FAN) is a smaller as the chassis airflow is front to back. The 7018 fan tray is picture below on the left and the 7010 fan tray is on the right. When it comes to the Nexus 7018 airflow, Panduit actually makes a special rack just for this switch to accommodate the airflow in a cold aisle/hot aisle data center. More information can be found on Panduit’s website at http://www.panduit.com/groups/MPM-BR/documents/InstallationInstruction/CMSCONT_035895.pdf

The final difference that I am aware of is with the power supplies. Today there are three power supplies for the Nexus 7000 switches, two that are AC powered and one that is DC powered. There is a 6000 kW AC (N7K-AC-6.0KW), a 7500 kW AC (N7k-AC-7.5KW), and finally a 6000 kW DC (N7K-DC-6.0KW) power supply. In the Nexus 7010 you can have up to three power supplies installed and in the 7018 you can have up to four installed. With the 6000 kW power supply, the cable connected is a standard C19 to country specific plug type. On the 7500 kW one, the cable is hard-wired to the power supply and must be ordered with the correct plug type. When it comes to the DC power supply, there is also a DC Power Interface Unit that you will need in order to provide the DC power to the power supply.
Pictures of the power supplies are below with the 6000 kW on the left, the 7.5KW on the right and the DC one below.

LACP Configuration and multi-chassis Etherchannel on Nexus 7000 with vPC, Part 2 of 2

Fryguy — Mon, 13 Sep 2010 23:47:25 +0000

This is the second part in a two part post on Etherchannel on the
Nexus 7000.  In the first part I covered how to configure vPC on
the Nexus 7000, here I will cover what it takes to get a remote
switch to uplink to the Nexus 7000 core switches using
vPC/Multi-chassis etherchannel.
Here is a diagram depicting the layout that we are using.  For
this part of the post, we will focus on the blue line that is
connecting both Nexus switches to the 3750 Stack.

On the Cisco 3750 switches (they are in a stack configuration of
two switches) we need to configure the interface to be in a
channel-group - for this example Iam using Channel-Group 6
(the switch is actually named StackSwitch06). What you will
also notice is that you configure the 3750 Stack just like it
was only connected to one switch, just one single port-channel
that consists of all the ports connected to both Nexus switches.
For this example we are using ports G1/0/1, G1/0/24, G2/0/1,
and G2/0/24. One thing I want to mention, when you are thinking
about your uplinks to your core switches, be aware of the switch
ASIC layout.  I say this because I have seen many times when
companies use ports 23 and 24 to uplink to a core switch.
The problem with this is that:
 1) The same ASIC is probably controlling both ports, and if
    it goes bad your links to the switch are gone and your
    switch is also isolated.
 2) You have a better chance of oversubscribing the ASIC
    before the uplink when utilization is high on the channel.
Now, onto the configuration, first up the Cisco 3750s.
    interface GigabitEthernet 1/0/1
     description [----[ Uplink to N7K1 - E9/10 ]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    interface GigabitEthernet1/0/24
     description [----[ Uplink to N7K2 - E9/10 ]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    interface GigabitEthernet 2/0/1
     description [----[ Uplink to N7K1 - E10/10]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    interface GigabitEthernet2/0/24
     description [----[ Uplink to N7K2 - E10/10]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    
Once the interfaces are assigned to the channel-group, we
can configure the etherchannel on the Cisco 3750s. Notice
that there is no vPC info nor anything else that says this
is connected to two switches.
     interface Port-channel6
      switchport trunk encapsulation dot1q
      switchport mode trunk

Now, on the Nexus side we need to do some configurations
as well. Both Nexus switches are also configured the same,
so there are no differences in the switch configs.
     interface Ethernet9/10
       description [----[ StackSwitch6-1 ]----]
       switchport
       switchport mode trunk
       channel-group 6 mode active
       no shutdown



     interface Ethernet10/10
       description [----[ StackSwitch6-1 ]----]
       switchport
       switchport mode trunk
       channel-group 6 mode active
       no shutdown
Now, when it comes to configuring the etherchannel on the Nexus
switches, is is configured the same except for the addition of
a vPC identifier. I recommend using the same number that you used
for the port-channel for easy identification, but that is up to you.
   interface port-channel6
     description [----[ LACP EtherChannel for StackSwitch6 ]----]
     switchport
     switchport mode trunk
     vpc 6

Once you have it configured on the Nexus, make sure it is up and
in the vPC correctly.        

     N7K1# sh int port-channel 6  
     port-channel6 is up
     vPC Status: Up, vPC number: 6
     Hardware: Port-Channel, address: 5475.d04f.1165 (bia 5475.d04f.1165)  
     Description: [----[ LACP EtherChannel for RackSwitch6 ]----]   
     Members in this channel: Eth9/10, Eth10/10  
     N7K1#
Once you have confirmed that all is working correctly, you can
check out the StackSwitch spanning tree information: 

     StackSwitch06#sh spanning-tree interface port-channel 6 
     Vlan             Role Sts Cost      Prio.Nbr Type
     ---------------- ---- --- --------- -------- --------------------------------
     VLAN0001         Root FWD 3         128.656  P2p
     VLAN0002         Root FWD 3         128.656  P2p
     VLAN0003         Root FWD 3         128.656  P2p
     VLAN0004         Root FWD 3         128.656  P2p
     VLAN0005         Root FWD 3         128.656  P2p
     StackSwitch06#
You will see that even though you are connected to two switches,
the port-channel is seen as a single spanning-tree
path to the root.

LACP Configuration and multi-chassis Etherchannel on Nexus 7000 with vPC, Part 1 of 2

Fryguy — Mon, 13 Sep 2010 15:21:21 +0000

The other day I received a question on Ether-channel and the Nexus
7000 - based on the question I felt it would be also good to
include the information here.
This will be a 2-part post, first part is the Nexus configuration
for vPC, the second post will be on the mutli-chassis ether-channel
configuration around the 3750 as well as the Nexus 7000 switches.
What are the benefits of Multi-chassis (vPC) ether-channel? 
Basically all your up-links from your switches are in FORWARDING
mode, nothing is in blocking mode in your spanning tree domain. 
What this means is that you have a loop free topology in your
data center and all links can be utilized.
Below is the diagram of the configuration that I will be
showing here.  There will be a Layer 2 Ether-channel vPC between
the Nexus 7010-1 and Nexus 7010-2 (Orangish line), a Layer 3
Ether-channel for vPC keep-alive (Red line), as well as a
mutli-chassis (vPC) ether-channel from a 3750 stack to Nexus 7010-1
and Nexus 7010-2 with all links in a single ether-channel bundle.

Configuration for both of the Nexus switches is the same except where noted.


Configuration for the Nexus switches
First thing to do is enable the vPC feature:
      feature vpc
 
Once you have enabled the vPC feature, you should create your keep-alive links.
Here I create a port-channel via LACP over ports 9/1 and 10/1.  You will also
notice that I have spread the channel over two line cards.  This has been done
to help assure maximum redundancy.  If a card where to go bad, the other card would
still be active in the port-channel. 
      interface Ethernet9/1
       description [----[ vPC KeepAlive to CoreSwitch2 ]----]
       channel-group 101 mode active  ! Assign port to port-channel 101 via LACP
       no shutdown
     interface Ethernet10/1
       description [----[ vPC KeepAlive to CoreSwitch2 ]----]
       channel-group 101 mode active
       no shutdown
Now we can create the VRF for the keep-alive link.  I suggest using a dedicated
VRF for security and sanity purpose.  This VRF will not participate in your
global routing table, thus allowing for more stability and also the prevention
of duplicate IP addresses in the network.
     vrf context VPC100_KA
Now we can create the Layer 3 interface on the port-channel and assign it
to the new VRF, VPC100_KA
     interface port-channel101
       description [----[ vPC Keep-Alive link between CoreSwitches ]----]
       vrf member VPC100_KA ! Assign this interface into the appropriate VRF
       ip address 10.10.10.1/30  ! The other side of the link is .2/30

Now you can configuration the vPC Peer links (Orangish lines).  Since I am using
10G links for this connection, I have set the rate mode to Dedicated.  This prevents
any chance for over subscription on the 10G port.  It also disables the other 3 ports in
group, so you need to keep that in mind when you are designing your deployment.
     interface Ethernet7/1
       description [-[ vPC Connection to Nexus 7010-2 - E7/1 ]-]
       switchport
       switchport mode trunk  ! Set the mode to trunk
       rate-mode dedicated force ! Force the rate-mode
       mtu 9216
       udld enable ! Since this is also fiber, enable UDLD
       channel-group 100 mode active ! Assign to port-channel 100
       no shutdown
     !
     interface Ethernet8/1
       description [-[ vPC Connection to Nexus 7010-2 - E8/1 ]-]
       switchport
       switchport mode trunk
       rate-mode dedicated force
       mtu 9216
       udld enable
       channel-group 100 mode active
       no shutdown
     !

Now to configure the port-channel as a vPC link as well as the vPC
domain information.
     interface port-channel100
       description [-[ vPC Peer-Link between Nexus Switches ]-]
       switchport
       switchport mode trunk
       vpc peer-link ! Assign this port-channel as a vpc peer-link
       spanning-tree port type network
       mtu 9216
     !
vpc domain 100
role priority 16000 ! Here I hard-coded switch 1 to be the vPC master.
                       switch 2 was left as the default
peer-keepalive destination 10.10.10.2 source 10.10.10.1 vrf VPC100_KA
                    ! The other side has the IP addresses reversed
Had to move the formatting above to get the command to fit, sorry.
Let's check the port-channel and make sure it is up with the appropriate members.
As you can see from the output, Eth7/1 and Eth8/1 are members of the channel.

     N7K1# sh int port-channel 100
      port-channel100 is up
      [------ SNIP - Output omitted! ------]
      Members in this channel: Eth7/1, Eth8/1
     N7K1#

Also check the vPC and the vPC keep-alive link
     N7K1# sh vpc
      Legend:
             (*) - local vPC is down, forwarding via vPC peer-link
      vPC domain id                        : 100 
      Peer status                          : peer adjacency formed ok      
      vPC keep-alive status                : peer is alive                 
      Configuration consistency status     : success 
      Type-2 consistency status            : success 
      vPC role                             : primary, operational secondary
      Number of vPCs configured            : 9   
      Peer Gateway                         : Disabled
      Dual-active excluded VLANs           : -
      vPC Peer-link status
      ---------------------------------------------------------------------
      id   Port   Status Active vlans    
      --   ----   ------ --------------------------------------------------
      1    Po100  up     1-224
     N7K1# sh vpc peer-keepalive
      vPC keep-alive status           : peer is alive                
      --Peer is alive for             : (1486816) seconds, (684) msec
      --Send status                   : Success
      --Last send at                  : 2010.09.11 12:38:36 872 ms
      --Sent on interface             : Po101
      --Receive status                : Success
      --Last receive at               : 2010.09.11 12:38:36 872 ms
      --Received on interface         : Po101
      --Last update from peer         : (0) seconds, (161) msec
     vPC Keep-alive parameters
     --Destination                    : 10.10.10.2
      --Keepalive interval            : 1000 msec
      --Keepalive timeout             : 5 seconds
      --Keepalive hold timeout        : 3 seconds
      --Keepalive vrf                 : VPC100_KA
      --Keepalive udp port            : 3200
      --Keepalive tos                 : 192
     N7K1#
As of now, both switches are connected via vPC.
This concludes the first post, the second post will be up shortly and will focus
around the Cisco 3750 configuration as well as the associated configs on the
Nexus 7000 switches.

The week after the installation. . .

Fryguy — Mon, 06 Sep 2010 21:59:13 +0000

Ok, the Nexus switches have been installed and running for over a week now with no further problems and there has been no fallout that I need to address prior to this post. Everyone at work took the change well, and understood some of the issues that we ran into as well as how we addressed them.
Just to recap what we did and the thoughts around why…
– Location had two Cisco 6509 switches running Sup2/MSFC2 as well as 6548 line cards – running for over 8 years
– Switches had started to show failures on line-cards on a more regular basis, chalked up to age of equip.
– When line-cards failed, spanning-tree loops where introduced which had the ability to severely impact the site
– Recently installed a large VM environment in location with the understanding of DMZ requirements in the near future
– This is a date center location, so data center level hardware was required (10g capabilities and beyond)
– In a single night, removed both Cisco 6509 switches, reconnected about 250 servers, and moved to LACP Etherchannel on all Rack switches in STP forwarding mode
– Also built a temporary network to maintain customer traffic through the site
Now, these requirements might not scream Nexus 7000 hardware – but we do not change core datecenter hardware very often and wanted to install a switch that had more “future proofing” built-in than other switches. The Cisco 6509E chassis in VSS mode has many of these features, but Cisco is investing money in the Nexus line and we felt that this is the proper way to go. Also, with a potential web presence imminent, the VDC and OTV capabilities of the Nexus are a perfect fit.
To be honest, the installation went really well. We where able to remove both Cisco 6509 switches in about an hour (they were DC, so an electrician was required) and get the new Nexus 7010 shoe-horned in their place. The Nexus are some heavy beasts, these where north of 500 lbs each – so I highly recommend removing the Power Supplies if possible. To be honest, the way that Cisco has designed these, they rack easily. Just like the Cisco 6500, the Nexus sits on a shelf for support and then gets screwed in on the face to the rack. We had the new network up and running, ready for cut-over by around 5 AM – 5 hours after we started.
So, what problems did we encounter – that is where the fun begins. What is funny is that only 1 of the problems I would consider a network design issue, the others where the typical – oh, we did not know that – or, whoops, type in the default gateway IP address.
So, the first problem that was actually a design issue was L3 neighbor routing over vPC – even though Cisco does not come out and say it does not work in the documentation, trust me – it does not. Per Cisco’s doc ( http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_vPC.pdf )
Configuring VLAN Interfaces for Layer 3 connectivity
You can use VLAN network interfaces on the vPC peer devices to link to Layer 3 of the network for such
applications as HSRP and PIM. However, we recommend that you configure a separate Layer 3 link for
routing from the vPC peer devices, rather than using a VLAN network interface for this purpose.

Now, as you can see from the above picture we have a EIGRP Neighbor relationship between all the routers and the core switches (R1-N7K1, R1-N7K2, R2-N7K2, R2-N7K1). Typically this is fine in a normal spanning-tree network, but what happens when using a VPC is something different. When a packet is received on R1 and R1 then decides that the next hop should be N7K2, but the device that R1 is trying to get to is attached to N7K1 (direct or etherchannel – the packet is dropped as it would need to transverse the vPC link twice. N7K2 sees the packet and just drops it. It actually sets a bit on the packet when it is received over the vPC link so that it is not re-transmitted back over the link. This is a loop-prevention mechanism, and that is a good thing as you can guess.
In order for us to fix this design flaw, we just had to make the links between R1 and N7K1 a L3 interface as well as between R2 and N7K2 a L3 interface. We actually talked about doing this prior to the Nexus being installed, but chose to wait as we did not want to change too many things at one time. The final design looks like this.

Now, some of the other problems that we encountered that where hardware related was a bad Supervisor module (backup supervisor actually) that was causing high CPU usage on the box. The first Nexus was running at 10-20% cpu whereas the second Nexus was running at 90-100% CPU. This was a little more difficult to track down as there where no errors in the log, but the way we figured it out was that one line card was stuck in a “downgrade in progress” message on some of the ports. The way that message showed up is that we actually downgraded from 5.0.3 to 5.0.2 to see if we had a bug in code that was causing the CPU issues. I will admit, Cisco had us a new supervisor as well as a line card in about 2 – 3 hours after we figured that out.
The last two problems that we encountered where out of out control – one problem that we experienced was bad default gateways on devices. I do not know how they where working prior to the upgrade as they had a non-existent IP address configured for the default gateway. Perhaps they had a static route and it disappeared when the network link went down. That is the only logical explanation that I can figure – and we had a few devices that did this, so it may actually be a “feature” in their code. Luckily, those devices have now been fixed.
The last one that we had taken us a bit longer to figure out – and it turns out that a “socks and sandle” person from the vendor had to get on the phone. We had a device that plays audio message to end-users, and since the installation of the Nexus that feature was no longer working. Stuck us as odd and out of character for it to be related to the new core switches, but since it broke after the install – we kept at it until we figured it out. What it turns out was the Nexus was receiving the packet ( it is the default gateway ) and dropping it. Why you ask, well because the vendor wrote their application to use the default gateway to loop the packet. ie – the same source and destination are in the packet, just routing it through the default gateway. The Nexus, and most any other security conscious device, would drop that packet as it is viewed as a spoof packet. Reviewing the logs in the main VDC, I can see the following error message: 2010 Aug 26 12:49:07 N7K1 %EEM_ACTION-6-INFORM: Packets dropped due to IDS check address reserved on module 9. Once we disabled that feature, everything started to work.
So what is the moral of all of this, well – it is good to know how all the software on your network is configured – but that honestly almost impossible to do. What does help is speaking with the vendors before the change so they are aware of what you are doing – and we did, we actually had pro-active tickets with all vendors for the change. I will also say that getting all the vendors on the phone (TAC, Vendor, etc) makes a huge difference. We had TAC on the phone (actually they usually had 2 TAC Nexus Engineers on the calls) and the vendor and they worked out all the communications between devices and figured it out. But in the end what did it was the “socks and sandle” person from the vendor who said “you know, it does this…” to get the light-bulb to click.
I just want to say, all-in-all this installation went very well – few bumps in the road, but they where to be expected. It helps to have a good team of people who you can count on when you are doing this, and thankfully I have that.

Smart Call Home Alert Groups

Fryguy — Wed, 11 Aug 2010 17:56:20 +0000

As promised, here is a list of the Smart Call Home groups and the commands that they execute.
All commands start with SHOW, I have left that out due to spacing constraints.

Group         Description           Commands
------------  --------------------- -------------------------------
Cisco TAC     All critical alerts   Executed commands based on the
              from the other groups alert group that originates the
                                    alert

Configuration Periodic events       module
              related to configs    running-configuration vdc-all all
                                    start-configuration vdc-all
                                    vdc current
                                    vdc membership
                                    version
Diagnostic    Events generated by   diagnostic result module all detail
              diagnostics           diagnostic result module number detail
                                    hardware
                                    logging last 200
                                    module
                                    sprom all
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
                                    version
EEM           Events generated by   diagnostic result module all detail
              by Embedded Event     diagnostic result module number detail
              Manager               module
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
Environmental Events related to     environment
              power, fan, etc       logging last 200
                                    module
                                    vdc current
                                    vdc membership
                                    version
Inventory     Inventory status      inventory
              that is provided      license usage
              when a unit is is     module
              cold booted or FRU    system uptime
              inserted or removed   sprom all
                                    vdc current
                                    vdc membership
                                    version


License       Events related to     license usage vdc all
              to licensing          logging last 200
                                    vdc current
                                    vdc membership
Linemodule    Events related to     diagnostic result module detail
 hardware     standard or smart     diagnostic result module number detail
              switching mods.       hardware
                                    logging last 200
                                    module
                                    sprom all
                                    tech-support ethpm
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
                                    version
Supervisor    Events related to     diagnostic result module all detail
 hardware     the supervisor        hardware  
                                    logging last 200
                                    module
                                    sprom all
                                    tech-support ethpm
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
                                    version
Syslog port   Events generated      license usage
 group        syslog PORT facility  logging last 200
                                    vdc current
                                    vdc membership
System        Events generated      diagnostic result module all detail
              by a failure of a     hardware
              system that is        logging last 200
              critical to the       module
              units operation       sprom all
                                    tech-support ethpm
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
Test          This group is for     module
              user generated test   vdc current
              messages              vdc membership
                                    version