VPC - Tag - Fryguy's Blog

Nexus 7009 – More information

Fryguy — Tue, 19 Jul 2011 12:00:38 +0000

A few months ago I did a quick post on the Nexus 7009 when it was pseudo announced (here ). When I was attending Cisco Live 2011 last week in Las Vegas I found out that they had a Nexus 7009 in their Data Center booth in World of Solutions for all to see. So, I figured I would post some updated information that I was able to gather or figure out. All this information was being openly spoken in the booth, nothing here was gathered from any other resource then what I was able to hear. To recap, below is a picture of the new family member, the 7009 along side its big brothers the 7010 and the 7018. As you can see there are some nice size differences. The Nexus 7009 us 14RU, the 7010 21RU, and the 7018 25RU. [][2]

Cisco Live 2011 – Sunday – Next Gen Data Center Infrastructure

Fryguy — Mon, 11 Jul 2011 04:04:00 +0000

Well, Day two of Cisco Live 2011 for me (for some this is your first day as you are still arriving).

For me, I am taking an 8 hour Techtorial on Next Generation Data Center Infrastructure TECDCT-8001. This is a paid seminar that has covered some great material – and probably help to generate some future blog posts as well!

Mike Herbert took the first part of the class and did a great review of some of the changes that have happened in the data center over the past few years. From the evolving 3-tier architecture with Spanning Tree and such all the way to Fabric Path and TRILL. Nothing was really in depth but was a great setting of the stage for the day. Some of the things that I liked was the discussion on the Data Center Standards and where different things are in the flow. Another nice discussion was data center cabling and the potential future of cabling. From the traditional home-run cabling that many of us use today to the discussion of top of rack (TOR) and end of row (EOR) drivers with regards to 10g and beyond cabling. Problems discussed where primarily the cost of running those lengths of cables to the cross-talk potentials due to the power required to drive the data. Oh yeah, Mike gets some bonus points for mentioning LISP during the presentation as well.

Nexus 7000 and the Show Tech command (gzip too)

Fryguy — Tue, 25 Jan 2011 13:18:24 +0000

Recently I have experience some interesting first-level support from our Cisco VAR. They are not too familiar with the Nexus 7000 and insisted on us sending them a SHOW TECH from the switch. If you are not familiar with a Nexus 7000, let me tell you – the show tech from this box can be over 100 Megs! So, when they asked for that I was like – are you sure? really really sure? I just have to say – thank you DropBox and public folders! I used DropBox place the file and then send the support the public link. Not sure that most e-mail systems would appreciate that large of a file!
So, if the Show Tech Support is so big, how do you get the information that support needs? Well, instead of running a complete show tech, you run it for the sections you need!
Here is a list of what you can append to a Show Tech-Support command: (show tech-support option)
N7K1-CoreSwitch1# sh tech-support ?

>               Redirect it to a file
»              Redirect it to a file in append mode
aaa             Display aaa information
aclmgr          ACL commands
adjmgr          Display Adjmgr information
arp             Display ARP information
ascii-cfg       Show ascii-cfg information for technical support personnel
assoc_mgr       Gather detailed information for assoc_mgr troubleshooting
bgp             Display BGP status and configuration
bootvar         Gather detailed information for bootvar troubleshooting
brief           Display the switch summary
callhome        Callhome troubleshooting information
cdp             Gather information for CDP trouble shooting
cert-enroll     Display certificates information
cfs             Gather detailed information for cfs troubleshooting
cli             Gather information for parser troubleshooting
clis            Gather information for CLI Server troubleshooting
commands        Show commands executed as part of show tech-support commands
details         Gather detailed information for troubleshooting
dhcp            Gather detailed information for dhcp troubleshooting
eem             Show EEM tech-support information
eigrp           Display EIGRP status and configuration
eltm            Eltm debug info
ethpm           Gather detailed information for ETHPM troubleshooting
forwarding      Forwarding debug information
ha              Gather detailed information for HA troubleshooting
hsrp            Show hsrp tech-support information
icmpv6          Display Icmpv6 information
im              Gather detailed information for IM troubleshooting
include-time    Gather tech-support and capture time taken to execute each
command
interface-vlan Gather detailed information for interface-vlan
troubleshooting
internal        Gather internal info for troubleshooting
ip              Display IP information
ipqos           IP QoS Manager
ipv4            Display IP information
ipv6            Display IPV6 information
l2fm            L2fm debug info
l2pt            Gather information for l2pt troubleshooting
l3vm            Display VRF information
lacp            Gather detailed information for LACP component
license         Display licensing information
logging         Show information on logging for technical support staff
m2fib           Gather detailed information for M2FIB troubleshooting
m2rib           Gather detailed information for M2RIB troubleshooting
mfwd            Display MCASTFWD status and configuration
module          Gather info related to a module
monitor         Gather detailed information for monitor troubleshooting
multicast       Display V4 Multicast information
netflow         Show NetFlow tech-support information
netstack        Gather information for NETSTACK troubleshooting
npacl           Display npacl information
pixm            Gather detailed information for PIXM troubleshooting
pixmc           PIXMC Information
pktmgr          Display Packet Manager information
port-channel    Gather detailed information for port channel troubleshooting
port-profile    Gather information for troubleshooting port-profiles
port-security   Port security related command
Ok, color code again:
Red – Router Output
Blue – Commands
Green – Notes
Traditionally when we have done show tech-support, we have always done terminal length to 0, set the capture buffer on our terminal application, and then executed the command – like this:
First we set the length to 0
FryGuyR1#terminal length 0
Then we configure our terminal to capture the data

LACP Configuration and multi-chassis Etherchannel on Nexus 7000 with vPC, Part 2 of 2

Fryguy — Mon, 13 Sep 2010 23:47:25 +0000

This is the second part in a two part post on Etherchannel on the
Nexus 7000.  In the first part I covered how to configure vPC on
the Nexus 7000, here I will cover what it takes to get a remote
switch to uplink to the Nexus 7000 core switches using
vPC/Multi-chassis etherchannel.
Here is a diagram depicting the layout that we are using.  For
this part of the post, we will focus on the blue line that is
connecting both Nexus switches to the 3750 Stack.

On the Cisco 3750 switches (they are in a stack configuration of
two switches) we need to configure the interface to be in a
channel-group - for this example Iam using Channel-Group 6
(the switch is actually named StackSwitch06). What you will
also notice is that you configure the 3750 Stack just like it
was only connected to one switch, just one single port-channel
that consists of all the ports connected to both Nexus switches.
For this example we are using ports G1/0/1, G1/0/24, G2/0/1,
and G2/0/24. One thing I want to mention, when you are thinking
about your uplinks to your core switches, be aware of the switch
ASIC layout.  I say this because I have seen many times when
companies use ports 23 and 24 to uplink to a core switch.
The problem with this is that:
 1) The same ASIC is probably controlling both ports, and if
    it goes bad your links to the switch are gone and your
    switch is also isolated.
 2) You have a better chance of oversubscribing the ASIC
    before the uplink when utilization is high on the channel.
Now, onto the configuration, first up the Cisco 3750s.
    interface GigabitEthernet 1/0/1
     description [----[ Uplink to N7K1 - E9/10 ]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    interface GigabitEthernet1/0/24
     description [----[ Uplink to N7K2 - E9/10 ]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    interface GigabitEthernet 2/0/1
     description [----[ Uplink to N7K1 - E10/10]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    interface GigabitEthernet2/0/24
     description [----[ Uplink to N7K2 - E10/10]----]
     switchport trunk encapsulation dot1q
     switchport mode trunk
     channel-group 6 mode active
    
Once the interfaces are assigned to the channel-group, we
can configure the etherchannel on the Cisco 3750s. Notice
that there is no vPC info nor anything else that says this
is connected to two switches.
     interface Port-channel6
      switchport trunk encapsulation dot1q
      switchport mode trunk

Now, on the Nexus side we need to do some configurations
as well. Both Nexus switches are also configured the same,
so there are no differences in the switch configs.
     interface Ethernet9/10
       description [----[ StackSwitch6-1 ]----]
       switchport
       switchport mode trunk
       channel-group 6 mode active
       no shutdown



     interface Ethernet10/10
       description [----[ StackSwitch6-1 ]----]
       switchport
       switchport mode trunk
       channel-group 6 mode active
       no shutdown
Now, when it comes to configuring the etherchannel on the Nexus
switches, is is configured the same except for the addition of
a vPC identifier. I recommend using the same number that you used
for the port-channel for easy identification, but that is up to you.
   interface port-channel6
     description [----[ LACP EtherChannel for StackSwitch6 ]----]
     switchport
     switchport mode trunk
     vpc 6

Once you have it configured on the Nexus, make sure it is up and
in the vPC correctly.        

     N7K1# sh int port-channel 6  
     port-channel6 is up
     vPC Status: Up, vPC number: 6
     Hardware: Port-Channel, address: 5475.d04f.1165 (bia 5475.d04f.1165)  
     Description: [----[ LACP EtherChannel for RackSwitch6 ]----]   
     Members in this channel: Eth9/10, Eth10/10  
     N7K1#
Once you have confirmed that all is working correctly, you can
check out the StackSwitch spanning tree information: 

     StackSwitch06#sh spanning-tree interface port-channel 6 
     Vlan             Role Sts Cost      Prio.Nbr Type
     ---------------- ---- --- --------- -------- --------------------------------
     VLAN0001         Root FWD 3         128.656  P2p
     VLAN0002         Root FWD 3         128.656  P2p
     VLAN0003         Root FWD 3         128.656  P2p
     VLAN0004         Root FWD 3         128.656  P2p
     VLAN0005         Root FWD 3         128.656  P2p
     StackSwitch06#
You will see that even though you are connected to two switches,
the port-channel is seen as a single spanning-tree
path to the root.

LACP Configuration and multi-chassis Etherchannel on Nexus 7000 with vPC, Part 1 of 2

Fryguy — Mon, 13 Sep 2010 15:21:21 +0000

The other day I received a question on Ether-channel and the Nexus
7000 - based on the question I felt it would be also good to
include the information here.
This will be a 2-part post, first part is the Nexus configuration
for vPC, the second post will be on the mutli-chassis ether-channel
configuration around the 3750 as well as the Nexus 7000 switches.
What are the benefits of Multi-chassis (vPC) ether-channel? 
Basically all your up-links from your switches are in FORWARDING
mode, nothing is in blocking mode in your spanning tree domain. 
What this means is that you have a loop free topology in your
data center and all links can be utilized.
Below is the diagram of the configuration that I will be
showing here.  There will be a Layer 2 Ether-channel vPC between
the Nexus 7010-1 and Nexus 7010-2 (Orangish line), a Layer 3
Ether-channel for vPC keep-alive (Red line), as well as a
mutli-chassis (vPC) ether-channel from a 3750 stack to Nexus 7010-1
and Nexus 7010-2 with all links in a single ether-channel bundle.

Configuration for both of the Nexus switches is the same except where noted.


Configuration for the Nexus switches
First thing to do is enable the vPC feature:
      feature vpc
 
Once you have enabled the vPC feature, you should create your keep-alive links.
Here I create a port-channel via LACP over ports 9/1 and 10/1.  You will also
notice that I have spread the channel over two line cards.  This has been done
to help assure maximum redundancy.  If a card where to go bad, the other card would
still be active in the port-channel. 
      interface Ethernet9/1
       description [----[ vPC KeepAlive to CoreSwitch2 ]----]
       channel-group 101 mode active  ! Assign port to port-channel 101 via LACP
       no shutdown
     interface Ethernet10/1
       description [----[ vPC KeepAlive to CoreSwitch2 ]----]
       channel-group 101 mode active
       no shutdown
Now we can create the VRF for the keep-alive link.  I suggest using a dedicated
VRF for security and sanity purpose.  This VRF will not participate in your
global routing table, thus allowing for more stability and also the prevention
of duplicate IP addresses in the network.
     vrf context VPC100_KA
Now we can create the Layer 3 interface on the port-channel and assign it
to the new VRF, VPC100_KA
     interface port-channel101
       description [----[ vPC Keep-Alive link between CoreSwitches ]----]
       vrf member VPC100_KA ! Assign this interface into the appropriate VRF
       ip address 10.10.10.1/30  ! The other side of the link is .2/30

Now you can configuration the vPC Peer links (Orangish lines).  Since I am using
10G links for this connection, I have set the rate mode to Dedicated.  This prevents
any chance for over subscription on the 10G port.  It also disables the other 3 ports in
group, so you need to keep that in mind when you are designing your deployment.
     interface Ethernet7/1
       description [-[ vPC Connection to Nexus 7010-2 - E7/1 ]-]
       switchport
       switchport mode trunk  ! Set the mode to trunk
       rate-mode dedicated force ! Force the rate-mode
       mtu 9216
       udld enable ! Since this is also fiber, enable UDLD
       channel-group 100 mode active ! Assign to port-channel 100
       no shutdown
     !
     interface Ethernet8/1
       description [-[ vPC Connection to Nexus 7010-2 - E8/1 ]-]
       switchport
       switchport mode trunk
       rate-mode dedicated force
       mtu 9216
       udld enable
       channel-group 100 mode active
       no shutdown
     !

Now to configure the port-channel as a vPC link as well as the vPC
domain information.
     interface port-channel100
       description [-[ vPC Peer-Link between Nexus Switches ]-]
       switchport
       switchport mode trunk
       vpc peer-link ! Assign this port-channel as a vpc peer-link
       spanning-tree port type network
       mtu 9216
     !
vpc domain 100
role priority 16000 ! Here I hard-coded switch 1 to be the vPC master.
                       switch 2 was left as the default
peer-keepalive destination 10.10.10.2 source 10.10.10.1 vrf VPC100_KA
                    ! The other side has the IP addresses reversed
Had to move the formatting above to get the command to fit, sorry.
Let's check the port-channel and make sure it is up with the appropriate members.
As you can see from the output, Eth7/1 and Eth8/1 are members of the channel.

     N7K1# sh int port-channel 100
      port-channel100 is up
      [------ SNIP - Output omitted! ------]
      Members in this channel: Eth7/1, Eth8/1
     N7K1#

Also check the vPC and the vPC keep-alive link
     N7K1# sh vpc
      Legend:
             (*) - local vPC is down, forwarding via vPC peer-link
      vPC domain id                        : 100 
      Peer status                          : peer adjacency formed ok      
      vPC keep-alive status                : peer is alive                 
      Configuration consistency status     : success 
      Type-2 consistency status            : success 
      vPC role                             : primary, operational secondary
      Number of vPCs configured            : 9   
      Peer Gateway                         : Disabled
      Dual-active excluded VLANs           : -
      vPC Peer-link status
      ---------------------------------------------------------------------
      id   Port   Status Active vlans    
      --   ----   ------ --------------------------------------------------
      1    Po100  up     1-224
     N7K1# sh vpc peer-keepalive
      vPC keep-alive status           : peer is alive                
      --Peer is alive for             : (1486816) seconds, (684) msec
      --Send status                   : Success
      --Last send at                  : 2010.09.11 12:38:36 872 ms
      --Sent on interface             : Po101
      --Receive status                : Success
      --Last receive at               : 2010.09.11 12:38:36 872 ms
      --Received on interface         : Po101
      --Last update from peer         : (0) seconds, (161) msec
     vPC Keep-alive parameters
     --Destination                    : 10.10.10.2
      --Keepalive interval            : 1000 msec
      --Keepalive timeout             : 5 seconds
      --Keepalive hold timeout        : 3 seconds
      --Keepalive vrf                 : VPC100_KA
      --Keepalive udp port            : 3200
      --Keepalive tos                 : 192
     N7K1#
As of now, both switches are connected via vPC.
This concludes the first post, the second post will be up shortly and will focus
around the Cisco 3750 configuration as well as the associated configs on the
Nexus 7000 switches.

The week after the installation. . .

Fryguy — Mon, 06 Sep 2010 21:59:13 +0000

Ok, the Nexus switches have been installed and running for over a week now with no further problems and there has been no fallout that I need to address prior to this post. Everyone at work took the change well, and understood some of the issues that we ran into as well as how we addressed them.
Just to recap what we did and the thoughts around why…
– Location had two Cisco 6509 switches running Sup2/MSFC2 as well as 6548 line cards – running for over 8 years
– Switches had started to show failures on line-cards on a more regular basis, chalked up to age of equip.
– When line-cards failed, spanning-tree loops where introduced which had the ability to severely impact the site
– Recently installed a large VM environment in location with the understanding of DMZ requirements in the near future
– This is a date center location, so data center level hardware was required (10g capabilities and beyond)
– In a single night, removed both Cisco 6509 switches, reconnected about 250 servers, and moved to LACP Etherchannel on all Rack switches in STP forwarding mode
– Also built a temporary network to maintain customer traffic through the site
Now, these requirements might not scream Nexus 7000 hardware – but we do not change core datecenter hardware very often and wanted to install a switch that had more “future proofing” built-in than other switches. The Cisco 6509E chassis in VSS mode has many of these features, but Cisco is investing money in the Nexus line and we felt that this is the proper way to go. Also, with a potential web presence imminent, the VDC and OTV capabilities of the Nexus are a perfect fit.
To be honest, the installation went really well. We where able to remove both Cisco 6509 switches in about an hour (they were DC, so an electrician was required) and get the new Nexus 7010 shoe-horned in their place. The Nexus are some heavy beasts, these where north of 500 lbs each – so I highly recommend removing the Power Supplies if possible. To be honest, the way that Cisco has designed these, they rack easily. Just like the Cisco 6500, the Nexus sits on a shelf for support and then gets screwed in on the face to the rack. We had the new network up and running, ready for cut-over by around 5 AM – 5 hours after we started.
So, what problems did we encounter – that is where the fun begins. What is funny is that only 1 of the problems I would consider a network design issue, the others where the typical – oh, we did not know that – or, whoops, type in the default gateway IP address.
So, the first problem that was actually a design issue was L3 neighbor routing over vPC – even though Cisco does not come out and say it does not work in the documentation, trust me – it does not. Per Cisco’s doc ( http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_vPC.pdf )
Configuring VLAN Interfaces for Layer 3 connectivity
You can use VLAN network interfaces on the vPC peer devices to link to Layer 3 of the network for such
applications as HSRP and PIM. However, we recommend that you configure a separate Layer 3 link for
routing from the vPC peer devices, rather than using a VLAN network interface for this purpose.

Now, as you can see from the above picture we have a EIGRP Neighbor relationship between all the routers and the core switches (R1-N7K1, R1-N7K2, R2-N7K2, R2-N7K1). Typically this is fine in a normal spanning-tree network, but what happens when using a VPC is something different. When a packet is received on R1 and R1 then decides that the next hop should be N7K2, but the device that R1 is trying to get to is attached to N7K1 (direct or etherchannel – the packet is dropped as it would need to transverse the vPC link twice. N7K2 sees the packet and just drops it. It actually sets a bit on the packet when it is received over the vPC link so that it is not re-transmitted back over the link. This is a loop-prevention mechanism, and that is a good thing as you can guess.
In order for us to fix this design flaw, we just had to make the links between R1 and N7K1 a L3 interface as well as between R2 and N7K2 a L3 interface. We actually talked about doing this prior to the Nexus being installed, but chose to wait as we did not want to change too many things at one time. The final design looks like this.

Now, some of the other problems that we encountered that where hardware related was a bad Supervisor module (backup supervisor actually) that was causing high CPU usage on the box. The first Nexus was running at 10-20% cpu whereas the second Nexus was running at 90-100% CPU. This was a little more difficult to track down as there where no errors in the log, but the way we figured it out was that one line card was stuck in a “downgrade in progress” message on some of the ports. The way that message showed up is that we actually downgraded from 5.0.3 to 5.0.2 to see if we had a bug in code that was causing the CPU issues. I will admit, Cisco had us a new supervisor as well as a line card in about 2 – 3 hours after we figured that out.
The last two problems that we encountered where out of out control – one problem that we experienced was bad default gateways on devices. I do not know how they where working prior to the upgrade as they had a non-existent IP address configured for the default gateway. Perhaps they had a static route and it disappeared when the network link went down. That is the only logical explanation that I can figure – and we had a few devices that did this, so it may actually be a “feature” in their code. Luckily, those devices have now been fixed.
The last one that we had taken us a bit longer to figure out – and it turns out that a “socks and sandle” person from the vendor had to get on the phone. We had a device that plays audio message to end-users, and since the installation of the Nexus that feature was no longer working. Stuck us as odd and out of character for it to be related to the new core switches, but since it broke after the install – we kept at it until we figured it out. What it turns out was the Nexus was receiving the packet ( it is the default gateway ) and dropping it. Why you ask, well because the vendor wrote their application to use the default gateway to loop the packet. ie – the same source and destination are in the packet, just routing it through the default gateway. The Nexus, and most any other security conscious device, would drop that packet as it is viewed as a spoof packet. Reviewing the logs in the main VDC, I can see the following error message: 2010 Aug 26 12:49:07 N7K1 %EEM_ACTION-6-INFORM: Packets dropped due to IDS check address reserved on module 9. Once we disabled that feature, everything started to work.
So what is the moral of all of this, well – it is good to know how all the software on your network is configured – but that honestly almost impossible to do. What does help is speaking with the vendors before the change so they are aware of what you are doing – and we did, we actually had pro-active tickets with all vendors for the change. I will also say that getting all the vendors on the phone (TAC, Vendor, etc) makes a huge difference. We had TAC on the phone (actually they usually had 2 TAC Nexus Engineers on the calls) and the vendor and they worked out all the communications between devices and figured it out. But in the end what did it was the “socks and sandle” person from the vendor who said “you know, it does this…” to get the light-bulb to click.
I just want to say, all-in-all this installation went very well – few bumps in the road, but they where to be expected. It helps to have a good team of people who you can count on when you are doing this, and thankfully I have that.