31 December 2012

30 November 2012

Spanning-tree Loop Troubleshooting and SafeGuards


Problem Description:

Spanning tree loop caused network outage
Action Plan:
Implement Layer 2 safeguards designed to protect against STP loops and mitigates the impact if one does occur.
1) First, verify that currently the proper switch is STP root for all VLANs. Then enable root guard on Root/Core switch on all uplink ports to the distribution layer switches.
Excellent doc that details root guard. See the section titled ” What Is the Difference Between STP BPDU Guard and STP Root Guard?” for clarification on the difference. You want root guard on the root and bpdu guard on the access layer. You do not want root guard on the port channel between core switches running HSRP. Only on the uplinks to other switches that you do NOT want to become spanning tree root.
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00800ae96b.shtml
2) Enable loop guard on all distribution/access layer switches
3) Enable BPDU guard on all distribution/access layer switches
4) Enable UDLD aggressive on all fiber uplinks
Unidirectional links can cause spanning tree loops. UDLD Aggressive will prevent this by shutting down a unidirectional link.
5) Prune unnecessary VLANs off your trunks
After implementing root guard, loop guard, UDLD aggressive, and BPDU guard, bring the link back up and see if the loop reforms.
IF THE LOOP REFORMS:
1) Have a TAC engineer online to troubleshoot
2) Enable mac-address move notification (if applicable – this is disabled by default on the 6500/7600 platform and enabled by default on others)
 ITLABSW#(config)#mac-address-table notification mac-move
Check the switch log for mac’s flapping between interfaces. These are the ports that are participating in the loop. Trace the MAC back to its source. Look for:
A link flapping on a upstream switch, causing spanning tree TCNs and SPanning Tree reconvergence. This should be used in conjunction with step 3 below.
A unidirectional link on an upstream switch causing the loop.
A hub or switch connected to a portfast enabled access port where this mac is learned. Shut this port down and see if this breaks the loop.
3) Check for TCNs
While the loop is occurring, if you see excessive TCNs you need to trace the TCNs (topology change notifications) to the source . To do this, start from the core and run the following commands:
 ITLABSW#show spanning-tree detail | inc ieee|occurr|from|is exec
The output from this command will show you the port the last TCN was received on and the time which it was received. Look for the port that received a TCN in the last few seconds.
 ITLABSW#sh spanning-tree detail | i ieee|occur|from|is exec

   VLAN0001 is executing the rstp compatible Spanning Tree protocol

     Number of topology changes 187927 last change occurred 00:01 ago <-time rec'd

         from Port-Channel12 <--interface that received the TCN
You will want to follow this port until the port that receives the TCN is an access port, or until the switch in question is generating TCNs but not receiving them. If you find an access port receiving TCNs, shut it down.
If you find a switch generating TCNs, you will want to look for two ports in a spanning tree forwarding state for the same VLAN. If you find two ports in a forwarding state, shut one port down and see if this breaks the loop. Check for a unidirectional link or excessive link flaps.
4) Look for packets hitting the CPU. Sniff the CPU and see if the packets share a common source. Track down the source. If they are STP or CDP packets (or packets destined to the 0100.0CCC.CCCX reserved multicast address) trace where the source mac is learned. See if the source mac leads you in a loop.
If you see two ports in a forwarding state for the same VLAN on the same switch, we need to look for the following:
a) does this switch think he is the root for this VLAN (or vlans)?
b) should he be?
c) Is he receiving BPDUs from his neighbor on the ports in a forwarding state? (sniff both forwarding ports to look for BPDUs)
d) look for a unidirectional link on one of the ports in a forwarding state
e) shut one of the ports in a forwarding state and see if the loop stops
5) Look for an interface with a very high input rate and low output rate.
 ITLABSW#sh int | i is up|rate
When a bridging loop occurs you will usually see multiple interfaces with a high output rate and low input rate and a single interface with a high input rate and low output rate.
- Trace the port with the high input rate down until you come to an access port and shut it down
- If the port with the high input rate leads you into a loop you will want to check spanning tree states until you either find a switch that has a port in an incorrect forwarding state or some other reason that is causing us to loop packets. TAC will need to assist here.

29 November 2012

NX-OS: OSPF Redistribution with VRF-lite on Nexus 5500


Creating VRF context

vrf context TEST_VRF
 ip route <ip address/range> <next hop>

Configure vrf specific static routes under the same vrf context


Assign interface to VRF

interface Ethernet1/1
 no switchport
 vrf member TEST_VRF
 ip address 1.1.1.1/24

Remember to apply IP address to interface AFTER the vrf config


Create OSPF Process

router ospf 201
 vrf TEST_VRF

interface Ethernet1/1
 ip router ospf 201 area 0


Redistribute Directly Connected Subnets

route-map CONNECTED_INTO_OSPF permit 10
  set metric 20 
  set metric-type type-1

router ospf 201
 vrf TEST_VRF
  redistribute direct route-map CONNECTED_INTO_OSPF 

Metrics and OSPF advertisement type is configured in the route-map


Redistribute Selective Static Routes

ip prefix-list STATIC_INTO_OSPF permit 10.10.10.0/20

route-map STATIC_INTO_OSPF permit 10
 !match  ip address prefix-list STATIC_INTO_OSPF
  set metric 20 
  set metric-type type-1

router ospf 201
 vrf TEST_VRF
  redistribute static route-map STATIC_INTO_OSPF 

Only 10.10.10.0/24 is redistributed into OSPF.

Verifications

show ip ospf vrf TEST_VRF
show ip ospf database detail vrf TEST_VRF #verify metric and type for advertisements
show ip ospf interface vrf TEST_VRF

21 September 2012

Nexus 7000 from Catalyst 6500 and 4500 Switches as of NX-OS 6.0


Some caveats: NX-OS does not support all the features of IOS, for example, acting as a DHCP server and NTP authentication.

IOS Commands (Nexus NX-OS Commands when different)

Function

show versionDisplays information about the currently running system software image and an overview of the installed hardware.
show moduleDisplays information about the installed modules including module number, module type, number of ports on each module, module MAC addresses, and the module status.
router(config)#do show command

router(config)#show command! or
router(config)#do show command
View existing configuration information from the configuration command prompt using show commands.

Note:
 For NX-OS, the [Tab] key and "?" will work for Exec mode commands inside of configuration mode. The use of "do" in NX-OS is currently working, but is not documented.
router(config)# do Exec-commandrouter(config)# Exec-command! or
router(config)#do Exec-command
View existing configuration information from the configuration command prompt.

Note:
 For NX-OS, the [Tab] key and "?" will work for Exec mode commands inside of configuration mode. The use of "do" in NX-OS is currently working, but is not documented.
reloadReloads the operating system for the entire device 
Note:
 For the Nexus 7000, this command works only in the default VDC.
! 6500
hw-module module slot reset! 4500
no hw-module [slot | modulenumber powerhw-module [slot | modulenumber power
reload module mod-#
Reloads a module in the device by turning power off then on.
Note: For the Nexus 7000, this command works only in the default VDC
service timestamps [debug | log] [uptime | datetime [msec]] [localtime] [show-timezone] [year]

debug logging
logging timestamp {microseconds | milliseconds | seconds}
Apply a time stamp to debugging messages or system logging messages. 

Note:
 In NX-OS, to enable debug logging configure 'debug logging' command. NX-OS does not have as many options for timestamps.
service password-encryption! No equivalent NX-OS commandNote: By default, NX-OS encrypts plain text passwords and enables password strength checking.
logging buffered [discriminator discr-name] [buffer-size] [severity-level]

logging logfile logfile-name severity-level [size bytes]
Enable system message logging to a local buffer
username name {nopassword | password password | passwordencryption-type encrypted-password}username user-id [password [0 | 5password ]Create and configure a user account.
Note:
 By default, NX-OS encrypts plain text passwords and enables password strength checking.
aaa new-model
aaa authentication login default group tacacs+ local
aaa authentication login console none
aaa authentication enable default none
aaa authentication ppp default local
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 0 -15 start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
aaa authorization commands level
aaa authorization config-commands
aaa session-id commonaaa authentication login default {group tacacs+ local}
aaa authentication login console noneaaa user default-role 
aaa accounting default group tacacs+aaa authorization commands default
aaa authorization config-commands default
Configuring AAA.
Note: Not all commands are supported on the NX-OS.
clock timezone zone hours-offset [minutes-offset]
clock summer-time zone recurring [week day month hh:mm week day month hh:mm [offset]]

clock timezone zone-name offset-hours offset-minutesclock summer-time zone-name start-week start-day start-month start-time end-week end-day end-month end-time offset-minutes
Configure the time zone offset from Coordinated Universal Time (UTC) as well as daylight savings time.
ip classless
! No equivalent NX-OS command
This commands is not available in NX-OS.
ip subnet-zero
no ip source-route
no ip bootp server

! No equivalent NX-OS command
These commands are not available in NX-OS.
no ip domain-lookup
ip domain-nam
e [vrf vrf-name] nameip name-server [vrf vrf-nameip-address
no ip domain-lookup
ip domain-name 
domain-name [use-vrf name]
 
ip name-server 
ip-address [use-vrf name]
Disable DNS lookup feature and configure a domain name and name server.
ip ssh time-out seconds
ip ssh authentication-retries tries
ip ssh version 2ipfeature ssh
ssh key {dsa [force] | rsa [length [force]]}
Enable an SSH server.

Note:
 The Cisco NX-OS commands for SSH are different from the Cisco IOS commands. NX-OS software supports only SSHv2.
power redundancy-mode {redundant | combined}power redundancy-mode {combined | insrc-redundant ps-redundant |redundant}Configure the power supply redundancy mode
! 6500
no power enable module mod-#! 4500
no hw-module [slot | modulenumber power
poweroff module mod-#
Powers off a module from configuration mode.
redundancy
mode sso
main-cpu
auto-sync standard
! No equivalent NX-OS command
Configure CPU redundancy.

Note:
 : The Nexus 7000 supports dual supervisor modules to provide 1+1 redundancy for the control and management plane. Only one of the supervisor modules is active at any given time, while the other acts as a standby backup. No configuration commands are needed.
spanning-tree mode [pvst | mst | rapid-pvst]spanning-tree mode [rapid-pvst | mst]Default mode for IOS is PVST; default mode for NX-OS is RPVST.

Note:
 When you enter the command, all STP instances are stopped for the previous mode and are restarted in the new mode.
spanning-tree extend system-id! No equivalent NX-OS commandEnable the extended system ID feature on a chassis that supports 1024 MAC addresses. 
Note:
 NX-OS does not use this command, the extended system ID is always automatically enabled in NX-OS devices.
spanning-tree vlan vlan-id priority valueSet the STP bridge priority
vlan internal allocation policy ascending! No equivalent NX-OS commandConfigure the internal VLAN allocation scheme.
Note:
 NX-OS does not support this command.
interface type slot/numberswitchport
switchport mode access 
switchport access vlan vlan-idspanning-tree portfast
[speed {10 | 100 | 1000 | auto [10 | 100 | 1000] | nonegotiate}]
interface
 type slot/number
switchport
switchport host
switchport access vlan 
vlan-id
[speed {10 | 100 | 1000 | auto [10 100 | 1000] | 10000 | auto}]
vlan 
vlan-id
Configure a Layer 2 access port.

Note:
 NX-OS uses "Ethernet" as the type for all FastEthernet / GigabitEthernet / Ten Gigabit Ethernet interfaces.
The VLAN in the vlan-id needs to be created so that the interface will come up.
interface type slot/numberswitchport
switchport mode trunk
[switchport trunk allowed vlan vlan-id]
switchport trunk encapsulation [isl | dot1 | negotiate]
[switchport trunk allowed vlan add vlan-id]
[speed {10 | 100 | 1000 | auto [10 | 100 | 1000] | nonegotiate}]
interface 
type slot/number
switchport
switchport mode trunk
[switchport trunk allowed vlan 
vlan-id]
[switchport trunk allowed vlan add 
vlan-id]
[speed {10 | 100 | 1000
 auto [10 100 | 1000 ] | 10000 | auto}]
Configure a Layer 2 trunk port.

Note:
 NX-OS only supports 802.1Q encapsulation.
vlan vlan-#
interface vlan-#
no shutdownvlan vlan-#
feature interface-vlan
interface vlan-#
no shutdown
Configure a VLAN interface

Note:
 In NX-OS, the interface-vlan feature needs to be enabled before an interface VLAN can be configured. The VLAN needs to be defined as well for the interface to come up
interface port-channel channel-# 
switchport

switchport mode . . .
interface type slot/number
switchportchannel-group group_number mode {active | auto | desirable | on |passive}
feature LACP
interface port-channel 
channel-#
switchport
switchport mode . . .
interface 
type slot/number
switchport
channel-group channel-
 number [force] [mode {on | active | passive}] 
Configure a Layer 2 LACP port channel. 
Note:
 In NX-OS, the LACP feature needs to be enabled before it can be used.
vtp domain domain-nameConfigure the VTP domain name
vtp {server client | transparent | off}
feature vtp
vtp {server client | transparent | off}
Configure the VTP mode.

Note:
 By default, VTP mode is off for NX-OS.
udld {enable | aggressive}
feature udld
Enable UDLD globally on a device.
ip route prefix mask next-hop-addressip route ip-prefix/length next-hop-address
Configure static routes.
ip access-list extended access-list-name 
[sequence-number{permit | denyprotocol source source-wildcard destination destination-wildcard . . .

ip access-list access-list-name 
[sequence-number{permit | denyprotocol source destination . . . 

Create or configure an IPv4 ACL

Note:
 NX-OS supports one type of IPv4 ACL which is similar to the named extended ACL in IOS.
ip access-list resequence access-list-name starting-sequence-number increment

resequence access-list-type access-list access-list-name starting-sequence-number increment
Resequence an ACL.
router eigrp as-numberfeature EIGRP
router eigrp
 instance-tag[autonomous-system as-number]
Configure EIGRP routing.

Note:
 In NX-OS, the EIGRP feature needs to be enabled before it can be used. You can use any case-sensitive alphanumeric string up to 20 characters as an instance tagIf you configure an instance-tag that does not qualify as an AS number, you must use the autonomous-systemcommand to configure the AS number explicitly or this EIGRP instance will remain in the shutdown state
router(config-router)#
network ip-address [wildcard-mask]
interface type slot/number
ip address ip-prefix/length
ip router eigrp instance-tag
Configure a network in EIGRP.

Note:
 For NX-OS, a network is configured in EIGRP by associating it through an interface the router uses to connect to the area. NX-OS uses CIDR notation for IP addresses, but can accept the ip-address mask format as well.
router ospf process-idfeature OSPF
router ospf
 instance-tag
Configure OSPF routing.

Note:
 In NX-OS, the OSPF feature needs to be enabled before it can be used. Theinstance-tag is locally assigned and can be any alphanumeric string or positive integer.
router(config-router)#
network ip-address wildcard-mask area area-id
interface type slot/number
ip address ip-prefix/length
ip router ospf instance-tag area area-id
Configure a network in OSPF.

Note:
 For NX-OS, a network is configured in OSPF by associating it through an interface the router uses to connect to the area.
router ospf process-idauto-cost reference-bandwidth mbps

router ospf instance-tag
auto-cost reference-bandwidth
 bandwidth [Gbps | Mbps]
Configure a reference bandwidth for OSPF.
Note: For IOS, the default is 100 Mb/s. For NX-OS, the default is 40Gb/s.
router bgp as-number
no synchronization
bgp router-id ip-address
bgp always-compare-med
bgp log-neighbor-changes
bgp deterministic-med
bgp bestpath med missing-as-worst
no auto-summary

feature bgp
router bgp
 as-num
router-id
 ip-address
bestpath med missing-as-worst
 
bestpath always-compare-med
log-neighbor-changes
Configure BGP routing.
Note: For NX-OS, the no synchronizationand no auto-summary commands are enabled by default.
router bgp as-numberneighbor ip-address remote-as as-number
network network-number [mask network-mask]
router bgp as-numaddress-family ipv4 unicast
network
 ip-prefix/length
neighbor
 ip-address remote-as as-number

address-family ipv4 unicast
Advertise an IPv4 network in BGP.
router bgp as-numberneighbor peer-name peer-group
neighbor peer-name remote-as as-number
neighbor peer-name password 7 password
neighbor peer-name update-source Loopback0
neighbor peer-name send-community
neighbor peer-name timers 2 6
neighbor peer-name soft-reconfiguration inbound
neighbor ip-address-1 peer-group peer-name
neighbor ip-address-1 description descriptive-info
neighbor ip-address-2 peer-group peer-name
neighbor ip-address-2 description descriptive-info

router bgp as-numbertemplate peer peer-name
remote-as
 as-number
password 7
 password
update-source 
loopback0
timers 3 9
address-family ipv4 unicast
send-community
soft-reconfiguration inbound
neighbor
 ip-address-1 remote-as as-number
inherit peer peer-name
description descriptive-info
neighbor ip-address-2 remote-as as-number
inherit peer peer-name
description descriptive-info
Configure BGP Peer Group/Template.
interface type slot/number
ip address ip-address mask
ip helper-address ip-address
standby [group-numberip ip-address standby [group-numbertimershellotime holdtime
standby [group-numberpriority priority
standby [group-numberprempt
f
eature hsrp
ip dhcp relay
interface
 type slot/number
ip address ip-prefix/length 
ip dhcp relay address 
ip-address
hsrp 
group-number
ip 
ip-address
timers 
hellotime holdtimepriority priority
prempt
Configure HSRP with an IP helper address to a DHCP server. Different command syntax is used. NX-OS also uses 'hsrp' as keyword, while IOS uses 'standby'.

Note:
 In NX-OS, the HSRP feature needs to be enabled before it can be used. To use the DHCP relay, DHCP services also has to be enabled. The HSRP holdtime needs to be at least 3x the hello time. NX-OS uses CIDR notation for IP addresses, but can accept the ip-address maskformat as well.
Prior to NX-OS 4.2(1), the service dhcpcommand enabled the DHCP Relay feature. In NX-OS 4.2(1) the command was changed to ip dhcp relay.
ip dhcp pool name! No equivalent NX-OS commandConfigure a Dynamic Host Configuration Protocol (DHCP) address pool on a DHCP server.

Note:
 The NX-OS supports DHCP snooping, and DHCP relay, but does notsupport acting as a DHCP server.
ip multicast-routingfeature PIMEnable IP multicast routing
ip prefix-list name seq seq-num permit prefix/length
Configure an IP prefix-list
ipv6 unicast-routinginterface type slot/numberipv6 address ipv6-prefix/prefix-length eui-64interface type slot/number
ipv6 address ipv6-prefix/prefix-length eui-64
Enable IPv6 traffic forwarding on an interface.

Note:
 NX-OS does NOT need to enable IPv6 routing globally.
ip telnet source-interface Loopback0
ip tftp source-interface Loopback0
ip domain-lookup source-interface Loopback0
ip flow-export source Loopback0
ip tacacs source-interface Loopback0
logging source-interface Loopback0
snmp-server trap-source Loopback0
ntp source Loopback0

aaa group server tacacs+ default use-vrf management
source-interface mgmt0
snmp-server source-interface trap mgmt0
snmp-server source-interface inform mgmt0
snmp-server host 
ip-address use-vrf management
snmp-server host 
ip-address source-interface mgmt0
ntp server ip-address use-vrf management
vrf context management
ip route
 0.0.0.0/0 ip-address
interface mgmt0
ip address
 ip-prefix/length
logging server ip-address serverity use-vrf management
Configure a management interface.
Note: The NX-OS management interface is in a separate management VRF.
ntp server ip-address
Configure an NTP server as a time source.
Note: NX-OS previously needed the NTP enable command - this command was deprecated in 5.2(1). Starting with 5.2(1), NX-OS devices can serve as an NTP server for other devices in the network. (Use the ntp master [stratum] command to set the device as an authoritative NTP server.)
ntp authenticate
ntp authentication-key 
number md5
 value
ntp trusted-key 
key-number
ntp update-calendar
ntp server ip-address key key-id 


! No NX-OS support for ntp update-calendar command
Configure NTP authentication options.

Note:
 NX-OS does not currently support the ntp update-calendar command.

monitor session session-# 
source interface type slot/numbermonitor session session-# destination interface type slot/numberinterface type slot/numberswitchport 
switchport monitor [ingress | learning] 
monitor session session-numberdescription description
source interface 
type slot/number
destination interface 
type slot/number 
Enable SPAN sessions on interfaces or VLANs
snmp-server community RW-string RW acl-#
snmp-server community RO-string RO acl-#
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps tty
snmp-server enable traps entity
snmp-server enable traps port-security
snmp-server enable traps config
snmp-server host ip-address string
snmp ifmib ifindex persistsnmp-server community RW-string RW
snmp-server community
 RO-string RO
snmp-server community
 RW-string use-acl acl-name
snmp-server community RO-string use-acl acl-name
snmp-server enable traps snmp authentication
snmp-server enable traps link 
snmp-server enable traps entity
snmp-server host ip-address string
Enable common SNMP options.

Note:
 NX-OS syntax differs.
tacacs-server host ip-address
tacacs-server directed-request
tacacs-server key [0 | 7] key
feature tacacs+
tacacs-server host ip-address
tacacs-server directed-request
tacacs-server key [0 | 6 | 7] key
Configure TACACS+ server
I hope this gives you a useful overview of some common configuration and verification commands you may need when migrating.
Three useful Cisco documentation links for further information:
http://docwiki.cisco.com/wiki/Cisco_Nexus_7000_NX-OS/IOS_Comparison_Tech_Notes -- includes multiple articles comparing Cisco NX-OS and Cisco IOS features.