Cisco Nexus 5000 Switches Basic Configuration - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Monday, August 29, 2022

Cisco Nexus 5000 Switches Basic Configuration

 This post summarize some basic configuration for Cisco Nexus 5000 switches. 


Preliminary Information - Design

When deploying Cisco Nexus switches, you’ll need some specific information about the network already in place, and you’ll need to make some configuration decisions ahead of time. You should spend some time running the hardware, recording serial numbers and meeting with your team or with the customer. Here is just an overview of some things to do and consider.

  1. Start off by unboxing the new gear and powering everything up. Let the new switches run for a few days just so you know you don’t have any DOA devices to RMA. I try to do this whether the switches are for internal use or for a customer. If you’re on site with a customer, you may not be able to do run them at all before racking them, but the key is letting them run for a while at least before putting them into production.
  2. Make a list of the following preliminary information:
    • Hostname for each device
    • Management IP addresses, subnet mask and default gateway
    • Local user accounts
    • Features to enable such as vPC, FCoE, DHCP, FEX, VTP, LACP, etc
    • Role of switch (end-of-row, top-of-rack, core)
    • All VLANs needed on the Nexus switches
    • Rack location, type of cage nuts to use
    • vPC number(s) (just a unique identifier you’ll need to set up vPCs later on)
    • Uplink trunk ports to data center/LAN core
    • DHCP relay information
    • Any VLAN interfaces that will used on the switches for your design
    • A list of all the devices that will connect to the Nexus switches
  3. Check that you have the correct power cables for the PDUs, correct SFPs (1/10 Gbps ethernet, 8 Gbps fibre channel) and appropriate storage connectivity.
  4. Identify the hot and cold aisles and plan to install the switches accordingly. Default airflow on the 5500 series is front-to-back, for example, the back being where all the ports are located. Airflow on the switches can be ordered in either direction, so this is an important thing to check.

I like to gather the Nexus specific information before getting into mounting hardware or configuring anything at all. In my experience, sitting down with the team or with the customer before doing anything whatsoever is the best way to ensure a smooth project. Below is a simplified version of a spreadsheet I’ve used to gather relevant information. It’s a variation of something I used when working for a Cisco partner a few years ago and should be part of a larger spreadsheet in which you should capture DNS and RADIUS server addresses, SmartNet contract numbers, serial numbers, asset tag information, rack and data center location, and all that sort of thing. You can download the spreadsheet here.

Untitled




Basic Cisco Switch Configuration Procedure


1. Verify correct switch boot via console

2. Upgrade switch to latest recommended version

3. Reboot switch and verify correct boot from new IOS

4. For stack switches:

-          Connect stack modules and cables

-          switch 1 priority 10

5. Add management VLAN and management IP

6. Configure admin access. User: dude Pass: xxxx With privilege 15

7. Configure enable secret xxxx

8. Add host name

9. Configure SSH access to the switch

1
2
3
4
5
6
7
8
9
-   ip domain-name 51sec.org
-   crypto key generate rsa
-   How many bits in the modulus [512]: 1024
-   ip ssh version 2
 
-   configure line vty 0 4
-   login local
-   transport input ssh
-   session-timeout 15

 11. line console 0

1
-          logging synchronous

12. Global configurations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
-          service password-encryption
 
-          ip default-gateway X.X.X.X
 
-          no ip domain lookup
 
-          no ip http server
 
-          no ip http secure-server
 
-          snmp-server community evolution ro (if this is external switch ACL should be added)
 
-          ntp server X.X.X.X
 
-          vtp mode transparent
 
-          clock timezone UTC +/-X
 
-          service timestamps debug datetime
 
-          service timestamps log datetime
 
-          logging buffered 8192
 
-          spanning-tree mode rapid-pvst

13. For L3 switches

1
2
3
4
5
6
7
-          mls qos
 
-          ip routing
 
-          ip route 0.0.0.0 0.0.0.0 X.X.X.X (instead of ip default route)
 
-          sdm prefer routing (to enable PBR on the switch 3750/3650) – reboot is needed

14. configure vlans

15. if switch should not be a STP root, configure all vlans with priority 32768 or higher

1
-          spanning-tree vlan 1-4094 priority 32768

% Allowed values are:

1
2
3
0     4096  8192  12288 16384 20480 24576 28672
 
 32768 36864 40960 45056 49152 53248 57344 61440

16. Access list for external switches:

1
2
3
4
5
6
7
ip access-list standard dude_access
 
permit 1.2.3.4
 
permit 4.5.6.7
 
permit 7.8.9.10

-          Also add additional relevant internal networks if needed

17. Configure all VLANs or import vlan.dat file

18. Create all necessary interface VLANs

19. Access ports configuration

1
2
3
4
5
6
7
8
9
10
11
-          Switchport mode access
 
-          Switchport access vlan XXX
 
-          Description GiX/X | blabla_giX/X
 
-          load-interval 30
 
-          logging event link-status
 
-          spanning-tree portfast (for servers only)

20. Trunk port configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
-          Switchport mode trunk
 
-          switchport trunk allowed vlan x,xx,xxx
 
-          Description GiX/X | blabla_giX/X
 
-          load-interval 30
 
-          logging event link-status
 
-          logging event trunk-status
 
-          logging event spanning-tree

21. For backup trunk port add:

1
-          spanning-tree cost 2000000

 

22. Switch installation withaaa new-model

* aaa configuration:

1
2
3
-          aaa new-model
 
-          aaa authentication login default local

* line vty 0 4 should be configured without login local

 

23. After switch installation at data center, add access list to line vty:

1
-          access-class dude_access in

-          verify connectivity and SNMP

-          Save config

-          If connection was lost reboot switch






Initial Configuration


Now let’s get into the initial configuration wizard.

  1. Power up the new Nexus switch and connect to the console port using a serial cable. The switch will take several minutes to boot.
  2. The initial configuration wizard starts automatically. Use the information you worked out with your team or with the customer to complete the wizard. These settings can be changed later. The Nexus 7000 series initial configuration is almost the same, but it will prompt you for additional information about the default virtual device context.

Untitled


Upgrade Firmware



Upgrading the firmware requires a reboot, so make sure to do this before moving forward with any significant configuration and of course before putting the switch into production. There are several methods for moving files around, but I prefer using a USB stick because it’s fast, straightforward, and reliable.

  1. Download the latest recommended firmware code version for your specific switch from Cisco’s download page website (you’ll need to log in) and save it to your USB stick.
  2. Insert the USB stick into the USB port of the switch and run the following commands:

Untitled1



Start Basic Configuration for Nexus5K Switch

 1. Now configure basic Spanning Tree.

NEXUS5K-A#config term
NEXUS5K-A(config)#spanning-tree  port type network default
NEXUS5K-A(config)#spanning-tree  port type edge bpduguard default
 

2. Now enable all the features you’ll need for this implementation. Below is just an example of common features. It’s typically best practice not to enable features you don’t need.

NEXUS5K-A(config)#feature lacp
NEXUS5K-A(config)#feature fex
NEXUS5K-A(config)#feature interface-vlan
NEXUS5K-A(config)#feature vpc
NEXUS5K-A(config)#feature lldp

 

3. Typical IP storage traffic requires the switch to accommodate jumbo frames, but by default the switch is configured to process 1500 byte ethernet frames. Configure a QoS policy to accommodate 9000 byte ethernet frames.

NEXUS5K-A(config)#policy-map  type network-qos jumbo
NEXUS5K-A(config-pmap)#class  type network-qos class-default
NEXUS5K-A(config-pmap-nq-c)#mtu  9216system  qos
NEXUS5K-A(config-pmap-nq-c)#system  qos
NEXUS5K-A(config-sys-qos)#service-polictype network-qos jumbo
NEXUS5K-A(config-sys-qos)#end

 

4. Next configure the VLANs needed for this deployment. In a large network with a lot of VLANs I’ve used VTP in client mode to quickly get all the VLANs onto the switch, but generally I don’t recommend doing that. If you choose to use VTP, you’ll need to enable the feature and make sure you configure VTP in client mode. Afterward you can disable the protocol and the feature.

NEXUS5K-A#config term
NEXUS5K-A(config)#vlan 10
NEXUS5K-A(config-vlan)#name iSCSI
NEXUS5K-A(config)#vlan 20
NEXUS5K-A(config-vlan)#name vMOTION
NEXUS5K-A(config)#vlan 30
NEXUS5K-A(config-vlan)#name VM_MANAGEMENT
NEXUS5K-A(config)#vlan 40
NEXUS5K-A(config-vlan)#name NFS
NEXUS5K-A(config-if)#exit

 

5.  Now configure the virtual port channel (vPC). Configuring a vPC requires a peer link, vPC domain ID, and the appropriate interface configuration. The example below has two 10 Gbps ports in a port channel, though I typically configure four ports if I know they will be available. The channel-group mode must be active in order to utilize LACP.

NEXUS5K-A(config)#vpc domain 10

NEXUS5K-A(config-vpc)#peer-keepalivdestination [IP address of switch B] source [IP address of switch A]

NEXUS5K-A(config-vpc)#interface  e1/5-6
NEXUS5K-A(config-if)#channel-group  10 mode active
NEXUS5K-A(config-if)#interface  po 10
NEXUS5K-A(config-if)#description  vpc peer link
NEXUS5K-A(config-if)#switchport  mode trunk
NEXUS5K-A(config-if)#switchport  trunk allowed vlan 1, vlan 10, vlan 20, [include additional necessary vlans]
NEXUS5K-A(config-if)#spanning-tree  port type network
NEXUS5K-A(config-if)#vpc peer link
NEXUS5K-A(config-if)#no shut
NEXUS5K-A(config-if)#exit

 

The NX-OS operating system chooses the primary and secondary switch priorities automatically, but the role priority command can be used to manually configure which is which. The lower priority value sets the switch as primary. You can also add the delay restore [time in seconds] command to manually control how long it takes before the vPC comes back up on the peer switch after a reload. There are a variety of other commands you can use to control more precisely the behavior of the vPC, but for this exercise I’ve kept the configuration simple.


6. Configure the uplink trunk ports to the core switch. The upstream switch will likely be the data center core (Nexus 7009/7010) or the LAN core. The config below is for a Nexus 7k upstream switch.

NEXUS5K-A(config)#interface e1/1-2
NEXUS5K-A(config-if)#description  TRUNK_TO_CORE
NEXUS5K-A(config-if)#switchport
NEXUS5K-A(config-if)#switchport  mode trunk
NEXUS5K-A(config-if)#spanning-tree  port type network
NEXUS5K-A(config-if)#end

 

Notice above that in order to configure a range of ports on a Nexus switch it isn’t necessary to use the interface range command you may be used to from configuring Catalyst switches. Also note the interface command spanning-tree port type network. This is extremely important to use on interfaces connecting to other Nexus switches. When connecting to an IP storage controller use the interface command spanning-tree port type edge trunk. This command is used when connecting to end hosts that carry multiple VLANs. When connecting to non-Nexus switches such as a Catalyst 6500 series switch use the spanning-tree port type normal command. If you have redundant core switches, you should use a vPC for the uplink(s).

 

7. Configure the access ports.

NEXUS5K-A#config t
NEXUS5K-A(config)#interface e1/15
NEXUS5K-A(config-if)#description  UCS-FI-A Port e1/15
NEXUS5K-A(config-if)#switchport
NEXUS5K-A(config-if)#switchport  mode access
NEXUS5K-A(config-if)#switchport  access vlan 200
NEXUS5K-A(config-if)#end

 

8. Configure the fabric extenders. Each FEX will have a unique identifier which will also end up being the prefix on the interface number. In the example below, the first FEX is assigned the identifier 101, so the interfaces will appear as 101/1/1. A new vPC also needs to be created for each FEX which means each Nexus 5548/5596 will have two additional vPCs configured: one for each FEX. The example below is for one. Use a port channel to each FEX so you have link redundancy as well as switch redundancy.

NEXUS5K-A#conf t
NEXUS5K-A(config)#interface e1/10-11
NEXUS5K-A(config-if)#switchport  mode fex-fabric
NEXUS5K-A(config-if)#fex associate 101
NEXUS5K-A(config-if)#channel-group  101
NEXUS5K-A(config-if)#no shutdown
NEXUS5K-A(config-if)#interface  po 101
NEXUS5K-A(config-if)#switchport  mode fex-fabric
NEXUS5K-A(config-if)#fex associate 101
NEXUS5K-A(config-if)#vpc 101
NEXUS5K-A(config-if)#description  DUAL_HOMED_NX2248
NEXUS5K-A(config-if)#end
NEXUS5K-A#copy run start

 

Basic Cisco Nexus 5K installation guide (VPC)


https://www.xglobe.com/knowledgebase/switchs/cisco/nexus-5k-basic-instalation-guide-vpc/


1. Basic topology two Nexuses with VPC link between them and MGMT interfaces connected to OOB switch:

אחד

2. MGMT interface configuration:

1
2
3
N5K-A(config)# int mgmt 0
N5K-A(config-if)# ip address 192.168.3.100/24
N5K-A(config-if)# vrf member management (add interface to preconfigured VRF management)

 

3. Default gateway configuration for VRF management

1
2
N5K-A(config)# vrf context management
N5K-A(config-vrf)# ip route 0.0.0.0/0 192.168.3.254

 

TIP: Each procedure like ping, trace route, copy via addresses on the interface management should be done via VRF management:

1
2
3
4
5
6
7
8
9
10
N5K-A# ping 192.168.3.254
PING 192.168.3.254 (192.168.3.254): 56 data bytes
ping: sendto 192.168.3.254 64 chars, No route to host
Request 0 timed out
ping: sendto 192.168.3.254 64 chars, No route to host
 
N5K-A# ping 192.168.3.254 vrf management
PING 192.168.3.254 (192.168.3.254): 56 data bytes
64 bytes from 192.168.3.254: icmp_seq=0 ttl=63 time=2.183 ms
64 bytes from 192.168.3.254: icmp_seq=1 ttl=63 time=2.043 ms

 


Show version
Show license

6. VPC creation:

Step1: Create VPC domain

1
N5K-A(config)# vpc domain 1

 

Step2: VPC configuration

1
2
3
4
N5K-A(config-vpc-domain)# role priority 2000 (less is better)
N5K-A(config-vpc-domain)# peer-keepalive destination 192.168.3.101 source 192.168.3.100 (use MGMT interfaces on both devices)
N5K-A(config-vpc-domain)# delay restore 120
N5K-A(config-vpc-domain)# auto-recovery

 

Step3: VPC on the peer device:

1
2
3
4
5
N5K-B(config)# vpc domain 1
N5K-B(config-vpc-domain)# role priority 4000
N5K-B(config-vpc-domain)# peer-keepalive destination 192.168.3.100 source 192.168.3.101
N5K-B(config-vpc-domain)# delay restore 120
N5K-B(config-vpc-domain)# auto-recovery

 

Step4: Create interface port-channel for VPC: (for both devices)

1
2
3
N5K-A(config)# interface port-channel 1
N5K-A(config-if)# switchport mode trunk
N5K-A(config-if)# vpc peer-link

 

Step5: Configure port-channel interfaces: (for both devices)

1
2
3
4
5
6
7
N5K-A(config)# int eth1/1
N5K-A(config-if)# switchport mode trunk
N5K-A(config-if)# channel-group 1 mode active
 
N5K-A(config)# int eth1/2
N5K-A(config-if)# switchport mode trunk
N5K-A(config-if)# channel-group 1 mode active

 

Step6: Connect interfaces between the nexuses:

תמונה24 

Step7: Check the VPC status:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
N5K-A# show vpc
 
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link
 
vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 0
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)
 
vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     1,3-4,10-17,101,110,112,166,168-171,180,412

 

7. TIP! The configuration on the both devices should be same.



Nexus 5K basic installation guide L2 L3


https://www.xglobe.com/knowledgebase/switchs/cisco/nexus-5k-basic-installation-guide-l2-l3/

Introduction: Topology includes two nexuses with VPC configured, simple server that need redundant gateway and simple Cisco switch connected with port channel to both nexuses (split).

The VPC and sync between nexuses already preconfigured. This manual explains basic L2 and L3 configurations on nexuses. Remember that most of configuration should be same on both devices.

תמונה25

  • Install additional features hsrp, interface-vlan, lacp.
1
2
3
Nexus(conf)# feature hsrp
Nexus(conf)# feature interface-vlan
Nexus(conf)# feature lacp

 

  • VLAN configuration – same as on regular Cisco switches:
1
2
Nexus(conf)# vlan X
Nexus(conf-vlan)# name BLA

 

  • Interface VLAN configuration - same as on regular Cisco switches:
1
2
3
4
Nexus(conf)# interface vlan 10
Nexus(conf-if)# ip address 10.10.10.2/24 (in nexus you can use prefix instead of netmask)
Nexus(conf-if)# description BLA
Nexus(conf-if)# no shutdown

 

  • Default route or ip route same as on L3 switches
1
Nexus(conf)# ip route 0.0.0.0/0 10.10.10.254

 

  • HSRP configuration changed in NX-OS family and became more intuitive (configure under the interface):
1
2
3
4
5
Nexus(conf)# interface vlan 10
Nexus(conf-if)# hsrp 10 (you can use each group number you want)
Nexus(conf-hsrp)# ip 10.10.10.1 (this is floating gateway address)
Nexus(conf-hsrp)# preempt (I recommend to use it only for HSRP master)
Nexus(conf-hsrp)# priority 200 (use value between 1 – 255 and master should have higher priority)

 

  • Interface with GLC-T SFP should be configured with speed 1000

תמונה26

 

1
2
3
4
5
Nexus# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
Eth1/13       e1/13 | ny1rt5101_ notconnec trunk     full    1000    SFP-1000BAS

 

  • Interfaces where are redundant devices connected should be configured with vpc orphan-port suspend for example firewalls cluster or server with teaming
1
2
Nexus(conf)# interface Ethernet 1/13
Nexus(conf-if)# vpc orphan-port suspend

 

  • VPC port channel configuration (split when same port channel running from both nexuses) be aware, the configuration should be same on both nexuses:
1
2
3
4
5
6
7
8
9
10
11
Nexus1(conf)# interface Ethernet 1/10
Nexus1(conf-if)# channel-group (PO number) mode (on | active | passive)
 
Nexus2(conf)# interface Ethernet 1/10
Nexus2(conf-if)# channel-group (PO number) mode (on | active | passive)
 
Nexus1(conf)# interface port-channel (PO number)
Nexus1(conf-if)# vpc (PO number)
 
Nexus2(conf)# interface port-channel (PO number)
Nexus2(conf-if)# vpc (PO number)

 

Add additional configurations to the port channel interface, like switchport mode and etc…

After creating port channel interface, speed 10000 will be added automatically to the interface configuration:

 

1
2
3
4
5
6
7
8
9
Nexus1(conf)# show run interface po4
interface port-channel4
  description Po4 | NY1SW_Po3
  switchport mode trunk
  switchport trunk allowed vlan 166
  logging event port link-status
  logging event port trunk-status
  speed 1000
  vpc 4

 

  • The basic configuration of the spanning tree, interfaces, snmp and other well know services are the same as in the regular Cisco switches and will not present in the this manual.

 

  • In NX-OS you can’t check serial number of the switch via show version command, use show license host-id instead.
1
2
Nexus# show license host-id
License hostid: VDH=xyzzxy(this is the serial number)

 

  • Port profile: to reduce amount of configuration on the interfaces with the same role we can use port profiles, where we will configure all additional interfaces settings and attach this profile to the relevant interfaces.

For example we have 20 access ports that should be configured with the same settings like CDP, STP and storm control.

 

1
2
3
4
5
6
7
8
9
Nexus(conf)# port-profile type Ethernet BLA
Nexus(conf-xxx)# no cdp enable
Nexus(conf-xxx)# spanning-tree port type edge
Nexus(conf-xxx)# spanning-tree guard root
Nexus(conf-xxx)# storm-control broadcast level 0.50
Nexus(conf-xxx)# storm-control multicast level 5.00
 
Nexus(conf)# interface Ethernet 1/15
Nexus(conf-if)# inherit port-profile BLA (now all settings from profile BLA operating on the interface)





Basic Configuration 2

You’ll still need to fine tune your configuration including configuring your vty lines, SNMP, VRFs, RADIUS servers, and whatever features and optimizations you prefer to use. You may also want to employ a function called configuration synchronization (config-sync). Also, I don’t typically like to route on Nexus 5k switches so that they can focus on doing what they do best: switching frames super fast at layer 2. You can take a look at a basic configuration used in production here.

Template: 

line con 0
!Configure AAA Authentication for Local Console Line:
 login authentication CONAUTH
 exec-timeout 5 0
 logging synchronous
line vty 0 4
 access-class 101 in
!Configure Timeout for Login Sessions:
 exec-timeout 5 0
 logging synchronous
!Configure SSH Access:
 transport input ssh
 login authentication VTYAUTH
!



Connecting to Other Catalyst Gigabit Ethernet Switches

The first 8 ports on a Nexus 5010 and the first 16 ports on Nexus 5020 can be configured to operate as Gigabit Ethernet ports. You can use these ports to connect to older Gigabit Ethernet switches.

The one drawback is that the Nexus doesn’t participate in VTP, so all VLANs have to be manually defined on each switch independently.

Nexus 5000  - 2 Ports


interface Ethernet1/3  
switchport mode trunk  
speed 1000  
switchport trunk native vlan 999  
channel-group 3 mode on


Catalyst 3560G - 2 Ports

On the Cisco Catalyst 3560G, the configuration is almost identical as Nexus:

interface GigabitEthernet1/10  
switchport mode trunk  
switchport trunk native vlan 999  
channel-group 2 mode on


Nexus Management & Default VRFs

Cisco NX-OS devices have a default VRF and a management VRF. All Layer 3 interfaces exist in the default VRF until you assign them to another VRF. By default, all EXEC commands are processed in the default VRF unless you specify otherwise when you run a command.

Here is what you should know about the default VRF:

  • Routing protocols are run in the default VRF context unless another VRF context is specified
  • The default VRF uses the default routing context for all show commands.
  • The default VRF is similar to the global routing table concept.

Here is what you should know about the management VRF:

  • It is for management purposes only !
  • Only the mgmt0 interface can be in the management VRF; the mgmt0 interface cannot be assigned to another VRF.
  • No routing protocols can run in the management VRF (static routing only).

You should also know the following VRF guidelines and limitations:

  • When you make an interface a member of an existing VRF, NX-OS removes all Layer 3 configurations. Therefore, you should configure all Layer 3 parameters after adding an interface to a VRF.
  • If you configure an interface for a VRF before the VRF exists, the interface is operationally down until you create the VRF.
  • NX-OS creates the default and management VRFs by default. You should configure the mgmt0 IP address and other parameters after you add the mgmt0 interface to the management VRF.
  • The write erase boot command does not remove the management VRF configurations. You must use the write erase command and then the write erase boot command.

Connecting to Server Team Port or Single Port

For Team Port Configuration:

After teaming, you can configure ip address for this new Server-10G network interface. 

For N5K Port Configuration:

interface Ethernet1/3
  switchport access vlan 409
  spanning-tree port type edge
interface Ethernet1/4
  switchport access vlan 409
  spanning-tree port type edge
Note: E1/3 and E1/4 are connecting to those two teaming ports.

No comments:

Post a Comment