This post summarize some basic configuration for Cisco Nexus 5000 switches.
Preliminary Information - Design
- Start off by unboxing the new gear and powering everything up. Let the new switches run for a few days just so you know you don’t have any DOA devices to RMA. I try to do this whether the switches are for internal use or for a customer. If you’re on site with a customer, you may not be able to do run them at all before racking them, but the key is letting them run for a while at least before putting them into production.
- Make a list of the following preliminary information:
- Hostname for each device
- Management IP addresses, subnet mask and default gateway
- Local user accounts
- Features to enable such as vPC, FCoE, DHCP, FEX, VTP, LACP, etc
- Role of switch (end-of-row, top-of-rack, core)
- All VLANs needed on the Nexus switches
- Rack location, type of cage nuts to use
- vPC number(s) (just a unique identifier you’ll need to set up vPCs later on)
- Uplink trunk ports to data center/LAN core
- DHCP relay information
- Any VLAN interfaces that will used on the switches for your design
- A list of all the devices that will connect to the Nexus switches
- Check that you have the correct power cables for the PDUs, correct SFPs (1/10 Gbps ethernet, 8 Gbps fibre channel) and appropriate storage connectivity.
- Identify the hot and cold aisles and plan to install the switches accordingly. Default airflow on the 5500 series is front-to-back, for example, the back being where all the ports are located. Airflow on the switches can be ordered in either direction, so this is an important thing to check.
I like to gather the Nexus specific information before getting into mounting hardware or configuring anything at all. In my experience, sitting down with the team or with the customer before doing anything whatsoever is the best way to ensure a smooth project. Below is a simplified version of a spreadsheet I’ve used to gather relevant information. It’s a variation of something I used when working for a Cisco partner a few years ago and should be part of a larger spreadsheet in which you should capture DNS and RADIUS server addresses, SmartNet contract numbers, serial numbers, asset tag information, rack and data center location, and all that sort of thing. You can download the spreadsheet here.
Basic Cisco Switch Configuration Procedure
1. Verify correct switch boot via console
2. Upgrade switch to latest recommended version
3. Reboot switch and verify correct boot from new IOS
4. For stack switches:
- Connect stack modules and cables
- switch 1 priority 10
5. Add management VLAN and management IP
6. Configure admin access. User: dude Pass: xxxx With privilege 15
7. Configure enable secret xxxx
8. Add host name
9. Configure SSH access to the switch
1 2 3 4 5 6 7 8 9 | - ip domain-name 51sec.org- crypto key generate rsa- How many bits in the modulus [512]: 1024- ip ssh version 2- configure line vty 0 4- login local- transport input ssh- session-timeout 15 |
11. line console 0
1 | - logging synchronous |
12. Global configurations:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | - service password-encryption- ip default-gateway X.X.X.X- no ip domain lookup- no ip http server- no ip http secure-server- snmp-server community evolution ro (if this is external switch ACL should be added)- ntp server X.X.X.X- vtp mode transparent- clock timezone UTC +/-X- service timestamps debug datetime- service timestamps log datetime- logging buffered 8192- spanning-tree mode rapid-pvst |
13. For L3 switches
1 2 3 4 5 6 7 | - mls qos- ip routing- ip route 0.0.0.0 0.0.0.0 X.X.X.X (instead of ip default route)- sdm prefer routing (to enable PBR on the switch 3750/3650) – reboot is needed |
14. configure vlans
15. if switch should not be a STP root, configure all vlans with priority 32768 or higher
1 | - spanning-tree vlan 1-4094 priority 32768 |
% Allowed values are:
1 2 3 | 0 4096 8192 12288 16384 20480 24576 28672 32768 36864 40960 45056 49152 53248 57344 61440 |
16. Access list for external switches:
1 2 3 4 5 6 7 | ip access-list standard dude_accesspermit 1.2.3.4permit 4.5.6.7permit 7.8.9.10 |
- Also add additional relevant internal networks if needed
17. Configure all VLANs or import vlan.dat file
18. Create all necessary interface VLANs
19. Access ports configuration
1 2 3 4 5 6 7 8 9 10 11 | - Switchport mode access- Switchport access vlan XXX- Description GiX/X | blabla_giX/X- load-interval 30- logging event link-status- spanning-tree portfast (for servers only) |
20. Trunk port configuration
1 2 3 4 5 6 7 8 9 10 11 12 13 | - Switchport mode trunk- switchport trunk allowed vlan x,xx,xxx- Description GiX/X | blabla_giX/X- load-interval 30- logging event link-status- logging event trunk-status- logging event spanning-tree |
21. For backup trunk port add:
1 | - spanning-tree cost 2000000 |
22. Switch installation withaaa new-model
* aaa configuration:
1 2 3 | - aaa new-model- aaa authentication login default local |
* line vty 0 4 should be configured without login local
23. After switch installation at data center, add access list to line vty:
1 | - access-class dude_access in |
- verify connectivity and SNMP
- Save config
- If connection was lost reboot switch
Initial Configuration
Now let’s get into the initial configuration wizard.
- Power up the new Nexus switch and connect to the console port using a serial cable. The switch will take several minutes to boot.
- The initial configuration wizard starts automatically. Use the information you worked out with your team or with the customer to complete the wizard. These settings can be changed later. The Nexus 7000 series initial configuration is almost the same, but it will prompt you for additional information about the default virtual device context.
Upgrade Firmware
Upgrading the firmware requires a reboot, so make sure to do this before moving forward with any significant configuration and of course before putting the switch into production. There are several methods for moving files around, but I prefer using a USB stick because it’s fast, straightforward, and reliable.
- Download the latest recommended firmware code version for your specific switch from Cisco’s download page website (you’ll need to log in) and save it to your USB stick.
- Insert the USB stick into the USB port of the switch and run the following commands:
Start Basic Configuration for Nexus5K Switch
1. Now configure basic Spanning Tree.
NEXUS5K-A(config)#spanning-tree port type network default
NEXUS5K-A(config)#spanning-tree port type edge bpduguard default
2. Now enable all the features you’ll need for this implementation. Below is just an example of common features. It’s typically best practice not to enable features you don’t need.
NEXUS5K-A(config)#feature fex
NEXUS5K-A(config)#feature interface-vlan
NEXUS5K-A(config)#feature vpc
NEXUS5K-A(config)#feature lldp
3. Typical IP storage traffic requires the switch to accommodate jumbo frames, but by default the switch is configured to process 1500 byte ethernet frames. Configure a QoS policy to accommodate 9000 byte ethernet frames.
NEXUS5K-A(config-pmap)#class type network-qos class-default
NEXUS5K-A(config-pmap-nq-c)#mtu 9216system qos
NEXUS5K-A(config-pmap-nq-c)#system qos
NEXUS5K-A(config-sys-qos)#service-policy type network-qos jumbo
NEXUS5K-A(config-sys-qos)#end
4. Next configure the VLANs needed for this deployment. In a large network with a lot of VLANs I’ve used VTP in client mode to quickly get all the VLANs onto the switch, but generally I don’t recommend doing that. If you choose to use VTP, you’ll need to enable the feature and make sure you configure VTP in client mode. Afterward you can disable the protocol and the feature.
NEXUS5K-A(config)#vlan 10
NEXUS5K-A(config-vlan)#name iSCSI
NEXUS5K-A(config)#vlan 20
NEXUS5K-A(config-vlan)#name vMOTION
NEXUS5K-A(config)#vlan 30
NEXUS5K-A(config-vlan)#name VM_MANAGEMENT
NEXUS5K-A(config)#vlan 40
NEXUS5K-A(config-vlan)#name NFS
NEXUS5K-A(config-if)#exit
5. Now configure the virtual port channel (vPC). Configuring a vPC requires a peer link, vPC domain ID, and the appropriate interface configuration. The example below has two 10 Gbps ports in a port channel, though I typically configure four ports if I know they will be available. The channel-group mode must be active in order to utilize LACP.
NEXUS5K-A(config)#vpc domain 10
NEXUS5K-A(config-vpc)#peer-keepalive destination [IP address of switch B] source [IP address of switch A]
NEXUS5K-A(config-if)#channel-group 10 mode active
NEXUS5K-A(config-if)#interface po 10
NEXUS5K-A(config-if)#description vpc peer link
NEXUS5K-A(config-if)#switchport mode trunk
NEXUS5K-A(config-if)#switchport trunk allowed vlan 1, vlan 10, vlan 20, [include additional necessary vlans]
NEXUS5K-A(config-if)#spanning-tree port type network
NEXUS5K-A(config-if)#vpc peer link
NEXUS5K-A(config-if)#no shut
NEXUS5K-A(config-if)#exit
The NX-OS operating system chooses the primary and secondary switch priorities automatically, but the role priority command can be used to manually configure which is which. The lower priority value sets the switch as primary. You can also add the delay restore [time in seconds] command to manually control how long it takes before the vPC comes back up on the peer switch after a reload. There are a variety of other commands you can use to control more precisely the behavior of the vPC, but for this exercise I’ve kept the configuration simple.
6. Configure the uplink trunk ports to the core switch. The upstream switch will likely be the data center core (Nexus 7009/7010) or the LAN core. The config below is for a Nexus 7k upstream switch.
NEXUS5K-A(config-if)#description TRUNK_TO_CORE
NEXUS5K-A(config-if)#switchport
NEXUS5K-A(config-if)#switchport mode trunk
NEXUS5K-A(config-if)#spanning-tree port type network
NEXUS5K-A(config-if)#end
Notice above that in order to configure a range of ports on a Nexus switch it isn’t necessary to use the interface range command you may be used to from configuring Catalyst switches. Also note the interface command spanning-tree port type network. This is extremely important to use on interfaces connecting to other Nexus switches. When connecting to an IP storage controller use the interface command spanning-tree port type edge trunk. This command is used when connecting to end hosts that carry multiple VLANs. When connecting to non-Nexus switches such as a Catalyst 6500 series switch use the spanning-tree port type normal command. If you have redundant core switches, you should use a vPC for the uplink(s).
7. Configure the access ports.
NEXUS5K-A(config)#interface e1/15
NEXUS5K-A(config-if)#description UCS-FI-A Port e1/15
NEXUS5K-A(config-if)#switchport
NEXUS5K-A(config-if)#switchport mode access
NEXUS5K-A(config-if)#switchport access vlan 200
NEXUS5K-A(config-if)#end
8. Configure the fabric extenders. Each FEX will have a unique identifier which will also end up being the prefix on the interface number. In the example below, the first FEX is assigned the identifier 101, so the interfaces will appear as 101/1/1. A new vPC also needs to be created for each FEX which means each Nexus 5548/5596 will have two additional vPCs configured: one for each FEX. The example below is for one. Use a port channel to each FEX so you have link redundancy as well as switch redundancy.
NEXUS5K-A(config)#interface e1/10-11
NEXUS5K-A(config-if)#switchport mode fex-fabric
NEXUS5K-A(config-if)#fex associate 101
NEXUS5K-A(config-if)#channel-group 101
NEXUS5K-A(config-if)#no shutdown
NEXUS5K-A(config-if)#interface po 101
NEXUS5K-A(config-if)#switchport mode fex-fabric
NEXUS5K-A(config-if)#fex associate 101
NEXUS5K-A(config-if)#vpc 101
NEXUS5K-A(config-if)#description DUAL_HOMED_NX2248
NEXUS5K-A(config-if)#end
NEXUS5K-A#copy run start
Basic Cisco Nexus 5K installation guide (VPC)
1. Basic topology two Nexuses with VPC link between them and MGMT interfaces connected to OOB switch:
2. MGMT interface configuration:
1 2 3 | N5K-A(config)# int mgmt 0N5K-A(config-if)# ip address 192.168.3.100/24N5K-A(config-if)# vrf member management (add interface to preconfigured VRF management) |
3. Default gateway configuration for VRF management
1 2 | N5K-A(config)# vrf context managementN5K-A(config-vrf)# ip route 0.0.0.0/0 192.168.3.254 |
TIP: Each procedure like ping, trace route, copy via addresses on the interface management should be done via VRF management:
1 2 3 4 5 6 7 8 9 10 | N5K-A# ping 192.168.3.254PING 192.168.3.254 (192.168.3.254): 56 data bytesping: sendto 192.168.3.254 64 chars, No route to hostRequest 0 timed outping: sendto 192.168.3.254 64 chars, No route to hostN5K-A# ping 192.168.3.254 vrf managementPING 192.168.3.254 (192.168.3.254): 56 data bytes64 bytes from 192.168.3.254: icmp_seq=0 ttl=63 time=2.183 ms64 bytes from 192.168.3.254: icmp_seq=1 ttl=63 time=2.043 ms |
6. VPC creation:
Step1: Create VPC domain
1 | N5K-A(config)# vpc domain 1 |
Step2: VPC configuration
1 2 3 4 | N5K-A(config-vpc-domain)# role priority 2000 (less is better)N5K-A(config-vpc-domain)# peer-keepalive destination 192.168.3.101 source 192.168.3.100 (use MGMT interfaces on both devices)N5K-A(config-vpc-domain)# delay restore 120N5K-A(config-vpc-domain)# auto-recovery |
Step3: VPC on the peer device:
1 2 3 4 5 | N5K-B(config)# vpc domain 1N5K-B(config-vpc-domain)# role priority 4000N5K-B(config-vpc-domain)# peer-keepalive destination 192.168.3.100 source 192.168.3.101N5K-B(config-vpc-domain)# delay restore 120N5K-B(config-vpc-domain)# auto-recovery |
Step4: Create interface port-channel for VPC: (for both devices)
1 2 3 | N5K-A(config)# interface port-channel 1N5K-A(config-if)# switchport mode trunkN5K-A(config-if)# vpc peer-link |
Step5: Configure port-channel interfaces: (for both devices)
1 2 3 4 5 6 7 | N5K-A(config)# int eth1/1N5K-A(config-if)# switchport mode trunkN5K-A(config-if)# channel-group 1 mode activeN5K-A(config)# int eth1/2N5K-A(config-if)# switchport mode trunkN5K-A(config-if)# channel-group 1 mode active |
Step6: Connect interfaces between the nexuses:
Step7: Check the VPC status:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | N5K-A# show vpcLegend: (*) - local vPC is down, forwarding via vPC peer-linkvPC domain id : 1Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status : successPer-vlan consistency status : successType-2 consistency status : successvPC role : primaryNumber of vPCs configured : 0Peer Gateway : DisabledDual-active excluded VLANs : -Graceful Consistency Check : EnabledAuto-recovery status : Enabled (timeout = 240 seconds)vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po1 up 1,3-4,10-17,101,110,112,166,168-171,180,412 |
7. TIP! The configuration on the both devices should be same.
Nexus 5K basic installation guide L2 L3
Introduction: Topology includes two nexuses with VPC configured, simple server that need redundant gateway and simple Cisco switch connected with port channel to both nexuses (split).
The VPC and sync between nexuses already preconfigured. This manual explains basic L2 and L3 configurations on nexuses. Remember that most of configuration should be same on both devices.
- Install additional features hsrp, interface-vlan, lacp.
1 2 3 | Nexus(conf)# feature hsrpNexus(conf)# feature interface-vlanNexus(conf)# feature lacp |
- VLAN configuration – same as on regular Cisco switches:
1 2 | Nexus(conf)# vlan XNexus(conf-vlan)# name BLA |
- Interface VLAN configuration - same as on regular Cisco switches:
1 2 3 4 | Nexus(conf)# interface vlan 10Nexus(conf-if)# ip address 10.10.10.2/24 (in nexus you can use prefix instead of netmask)Nexus(conf-if)# description BLANexus(conf-if)# no shutdown |
- Default route or ip route same as on L3 switches
1 | Nexus(conf)# ip route 0.0.0.0/0 10.10.10.254 |
- HSRP configuration changed in NX-OS family and became more intuitive (configure under the interface):
1 2 3 4 5 | Nexus(conf)# interface vlan 10Nexus(conf-if)# hsrp 10 (you can use each group number you want)Nexus(conf-hsrp)# ip 10.10.10.1 (this is floating gateway address)Nexus(conf-hsrp)# preempt (I recommend to use it only for HSRP master)Nexus(conf-hsrp)# priority 200 (use value between 1 – 255 and master should have higher priority) |
- Interface with GLC-T SFP should be configured with speed 1000
1 2 3 4 5 | Nexus# show interface status--------------------------------------------------------------------------------Port Name Status Vlan Duplex Speed Type--------------------------------------------------------------------------------Eth1/13 e1/13 | ny1rt5101_ notconnec trunk full 1000 SFP-1000BAS |
- Interfaces where are redundant devices connected should be configured with vpc orphan-port suspend for example firewalls cluster or server with teaming
1 2 | Nexus(conf)# interface Ethernet 1/13Nexus(conf-if)# vpc orphan-port suspend |
- VPC port channel configuration (split when same port channel running from both nexuses) be aware, the configuration should be same on both nexuses:
1 2 3 4 5 6 7 8 9 10 11 | Nexus1(conf)# interface Ethernet 1/10Nexus1(conf-if)# channel-group (PO number) mode (on | active | passive)Nexus2(conf)# interface Ethernet 1/10Nexus2(conf-if)# channel-group (PO number) mode (on | active | passive)Nexus1(conf)# interface port-channel (PO number)Nexus1(conf-if)# vpc (PO number)Nexus2(conf)# interface port-channel (PO number)Nexus2(conf-if)# vpc (PO number) |
Add additional configurations to the port channel interface, like switchport mode and etc…
After creating port channel interface, speed 10000 will be added automatically to the interface configuration:
1 2 3 4 5 6 7 8 9 | Nexus1(conf)# show run interface po4interface port-channel4 description Po4 | NY1SW_Po3 switchport mode trunk switchport trunk allowed vlan 166 logging event port link-status logging event port trunk-status speed 1000 vpc 4 |
- The basic configuration of the spanning tree, interfaces, snmp and other well know services are the same as in the regular Cisco switches and will not present in the this manual.
- In NX-OS you can’t check serial number of the switch via show version command, use show license host-id instead.
1 2 | Nexus# show license host-idLicense hostid: VDH=xyzzxy(this is the serial number) |
- Port profile: to reduce amount of configuration on the interfaces with the same role we can use port profiles, where we will configure all additional interfaces settings and attach this profile to the relevant interfaces.
For example we have 20 access ports that should be configured with the same settings like CDP, STP and storm control.
1 2 3 4 5 6 7 8 9 | Nexus(conf)# port-profile type Ethernet BLANexus(conf-xxx)# no cdp enableNexus(conf-xxx)# spanning-tree port type edgeNexus(conf-xxx)# spanning-tree guard rootNexus(conf-xxx)# storm-control broadcast level 0.50Nexus(conf-xxx)# storm-control multicast level 5.00Nexus(conf)# interface Ethernet 1/15Nexus(conf-if)# inherit port-profile BLA (now all settings from profile BLA operating on the interface) |
Basic Configuration 2
You’ll still need to fine tune your configuration including configuring your vty lines, SNMP, VRFs, RADIUS servers, and whatever features and optimizations you prefer to use. You may also want to employ a function called configuration synchronization (config-sync). Also, I don’t typically like to route on Nexus 5k switches so that they can focus on doing what they do best: switching frames super fast at layer 2. You can take a look at a basic configuration used in production here.login authentication VTYAUTH
Connecting to Other Catalyst Gigabit Ethernet Switches
The first 8 ports on a Nexus 5010 and the first 16 ports on Nexus 5020 can be configured to operate as Gigabit Ethernet ports. You can use these ports to connect to older Gigabit Ethernet switches.The one drawback is that the Nexus doesn’t participate in VTP, so all VLANs have to be manually defined on each switch independently.
Nexus 5000 - 2 Ports
interface Ethernet1/3
switchport mode trunk
speed 1000
switchport trunk native vlan 999
channel-group 3 mode onCatalyst 3560G - 2 Ports
On the Cisco Catalyst 3560G, the configuration is almost identical as Nexus:
interface GigabitEthernet1/10
switchport mode trunk
switchport trunk native vlan 999
channel-group 2 mode onNexus Management & Default VRFs
Cisco NX-OS devices have a default VRF and a management VRF. All Layer 3 interfaces exist in the default VRF until you assign them to another VRF. By default, all EXEC commands are processed in the default VRF unless you specify otherwise when you run a command.
Here is what you should know about the default VRF:
- Routing protocols are run in the default VRF context unless another VRF context is specified
- The default VRF uses the default routing context for all show commands.
- The default VRF is similar to the global routing table concept.
Here is what you should know about the management VRF:
- It is for management purposes only !
- Only the mgmt0 interface can be in the management VRF; the mgmt0 interface cannot be assigned to another VRF.
- No routing protocols can run in the management VRF (static routing only).
You should also know the following VRF guidelines and limitations:
- When you make an interface a member of an existing VRF, NX-OS removes all Layer 3 configurations. Therefore, you should configure all Layer 3 parameters after adding an interface to a VRF.
- If you configure an interface for a VRF before the VRF exists, the interface is operationally down until you create the VRF.
- NX-OS creates the default and management VRFs by default. You should configure the mgmt0 IP address and other parameters after you add the mgmt0 interface to the management VRF.
- The write erase boot command does not remove the management VRF configurations. You must use the write erase command and then the write erase boot command.
Connecting to Server Team Port or Single Port
interface Ethernet1/3
switchport access vlan 409
spanning-tree port type edge
interface Ethernet1/4
switchport access vlan 409
spanning-tree port type edge
![[5 Mins Docker] Deploy FreshRSS Using Docker Run Command and Deploy To Fly.io](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_uzNMdKE0WJbzjXK0-x-uGPvYNQuBQsqrE6cwYkEq4J1d9Zb9B1mfN5ub0smrGVs4_8uRLEzucuegcvig39vLB1-09YcZKV_WBkcXTo_TtOHw)
No comments:
Post a Comment