This is Part 2 in a design example showing how I put together a UCS/Nexus/Netapp build using End-to-End FCoE for network and storage communication (At least as “End-To-End” as is possible today). In Part 1 of this series I covered basic concepts, hardware, and the first connections between the UCS Fabric Interconnects and the Nexus Switches for both Ethernet and Fibre Channel.
Here again is the basic design we are working with: 
Assumptions
I’m not going to go into general Nexus setup in this article. I will assume the two switches are configured in a vPC peer link setup, and are otherwise configured right out of the setup wizard.
Nexus Switches – Feature Setup
We have a lot of things to do here. First thing is to make sure the features we need are enabled on the Nexus switches. Do a show feature and look for the following:
- fcoe
- npiv
- lacp
If any of these are not currently enabled, go into config mode and type feature (feature name) to turn them on.
Changing Port Types
Our config has two traditional fibre channel links coming in (on each switch) from the Fabric Interconnects. If you do any sort of port listing at this point you will see SFP Transceiver errors and possibly other errors. This is because you have essentially plugged a FC transceiver into an Ethernet port. We need to change the port types on the nexus into “pure” FC ports. This is not done for FCoE, just for traditional FC links.
I was advised by Cisco that at one time you were required to mark FC ports at the end of the port range. So say you are looking at your base Nexus 5548UP with no expansion module. This device has 32 ports, meaning if you wanted 2 FC links, they had to go in ports 31 and 32. I am not sure if the latest NX-OS code still has this limitation; however I will use those ports for this design anyway.
To change the ports Ethernet1/31 and Ethernet 1/32 to FC ports, type the following in config mode:
N5K-A(config)# slot 1 N5K-A(config-slot)# port 31 type fc N5K-A(config-slot)# port 32 type fc
This is one of the few config steps on the Nexus that actually requires a reboot to take effect. Do this with:
N5K-A(config-slot)# copy running-config startup-config N5K-A(config-slot)# reload
When the switch comes back, you can do a show interface brief and see that you no longer have Ethernet1/31 and Ethernet1/32 ports, and instead have FC1/31 and FC1/32 ports.
VSAN Setup
Now we need to establish VSAN definitions. VSAN’s are similar in concept to VLAN’s, except they are more “complete” in that they have their own security, FC services, etc. Earlier I mentioned that we need to keep the FC fabrics separate, and to guarantee that we will define a different VSAN on each Nexus switch. For this example, I will define VSAN 3210 on switch A and VSAN 3211 on switch B.
In order for FC to travel over Ethernet, the Nexus requires mapping the VSAN definition to an Ethernet VLAN. To keep things simple, I define a VLAN using the same numbers (VLAN 3210 for VSAN 3210). We will also assign our FC ports to the new VSANs.
Switch A
N5K-A(config)# vsan database N5K-A(config-vsan-db)# vsan 3210 N5K-A(config-vsan-db)# vsan 3210 interface fc1/31 N5K-A(config-vsan-db)# vsan 3210 interface fc1/32 N5K-A(config-vsan-db)# vlan 3210 N5K-A(config-vlan)# fcoe vsan 3210 N5K-A(config-vlan)# end
Switch B
N5K-B(config)# vsan database N5K-B(config-vsan-db)# vsan 3211 N5K-B(config-vsan-db)# vsan 3211 interface fc1/31 N5K-B(config-vsan-db)# vsan 3211 interface fc1/32 N5K-B(config-vsan-db)# vlan 3211 N5K-B(config-vlan)# fcoe vsan 3211 N5K-B(config-vlan)# end
At this point we should have enough in place to verify communication between the first FC links and the switch. Easiest way to do this is a simple show flogi database command. You should see something like the following on each switch:
N5K-A# sh flogi database -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- fc1/31 3210 0x010000 20:43:56:7e:de:16:34:c0 2d:8f:59:7f:ee:16:34:c1 fc1/32 3210 0x010001 20:44:56:7e:de:16:34:c0 2d:8f:59:7f:ee:16:34:c1
At this point we’ve more or less configured a lot of “standard” FC only. You might even be asking yourself why we did that whole VLAN/VSAN mapping thing when we haven’t actually done anything FC over Ethernet yet. Well, here comes the fun stuff!
Nexus to NetApp – Unified Links
FCoE isn’t just about putting FC on a different cable, it’s about the concept of Unified Links where different traffic types coexist together. NetApp SAN’s allow you to use many different protocol types for storage traffic, and we want to eliminate separate cables for separate protocols. What many people want to do with UCS is run vSphere and our ideal setup today is to use FCoE for Boot-from-SAN, and NFS (over Ethernet) for VM storage.
For Ethernet, we are building a vPC Port Channel so traffic is not interrupted if we lose a switch or link. But with FC, we need to keep our paths separate and explicitly defined; if we lose a link, we WANT the path to fail (because in FC, that’s why we have multiple paths). The bad diagram below attempts to show this concept:
It helps to think of the FC protocol as running inside a “virtual cable” inside the Ethernet pipe. The “virtual cable” idea helps a lot when working with complex UCS networking as well.
While you can order NetApp SAN’s in different configurations, most production setups will be HA with two controllers. In the NetApp world these are “Active / Active” setups, so you will be serving live data from both controllers (and thus, cable them separately). For our example, each NetApp 3210 controller has a 2-port UTA (Unified Target Adapter) added, which we will connect to the Nexus switches with TwinAx cable.
For our example, we will connect cables as so:
- Nexus A Ethernet 1/29 <-> NetApp Controller A port e1a (vPC Port Channel 521)
- Nexus B Ethernet 1/29 <-> NetApp Controller A port e1b (vPC Port Channel 521)
- Nexus A Ethernet 1/30 <-> NetApp Controller B port e1a (vPC Port Channel 522)
- Nexus B Ethernet 1/30 <-> NetApp Controller B port e1b (vPC Port Channel 522)
Roadblock Prevention: FCoE QoS System Class
For a Cisco Nexus 5548 switch, the FCoE class-fcoe system class is not enabled by default in the QoS configuration. Before using FCoE, you must include class-fcoe in several policy types, whether you are going to go deeper into QoS or not. This isn’t required on Nexus 5010 or 5020 switches, where these definitions are in place by default. If you skip this step, your FCoE will not work.
Commands to enable:
class-map type qos class-fcoe class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-all-flood match qos-group 2 class-map type queuing class-ip-multicast match qos-group 2 class-map type network-qos class-fcoe match qos-group 1 class-map type network-qos class-all-flood match qos-group 2 class-map type network-qos class-ip-multicast match qos-group 2 system qos service-policy type qos input fcoe-default-in-policy service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type network-qos fcoe-default-nq-policy
(Don’t worry if you don’t know anything about QoS; the above is safe to put in place on both switches. It’s just enabling default settings that should be there in the first place.)
Port Channel Configs
First we will establish port channels for our Ethernet links to run on, to support vPC. We will need a separate Port-Channel definition on each switch for each controller. Here I used 521 to represent the port-channel going to NetApp Controller A, and 522 for NetApp Controller B.
You will need to have already established a VLAN where you intend to carry your Ethernet style storage traffic (NFS, for example). I am using VLAN 192 in the examples.
Nexus A:
interface port-channel521 description VPC Trunk to NetApp Controller A switchport mode trunk vpc 521 switchport trunk native vlan 192 switchport trunk allowed vlan 192,3210 spanning-tree port type edge trunk interface port-channel522 description VPC Trunk to NetApp Controller B switchport mode trunk vpc 522 switchport trunk native vlan 192 switchport trunk allowed vlan 192,3210 spanning-tree port type edge trunk
Nexus B:
interface port-channel521 description VPC Trunk to NetApp Controller A switchport mode trunk vpc 521 switchport trunk native vlan 192 switchport trunk allowed vlan 192,3211 spanning-tree port type edge trunk interface port-channel522 description VPC Trunk to NetApp Controller B switchport mode trunk vpc 522 switchport trunk native vlan 192 switchport trunk allowed vlan 192,3211 spanning-tree port type edge trunk
A few notes on the above:
- The VPC numbers can be anything as long as they match across switches. Choosing the same number as the port channel number just makes it easier to manage.
- This is a standard trunk, and can carry multiple VLAN’s if you want such a setup with your NetApp. You MUST however define a NON-FCoE native VLAN as shown.
- Notice the VLAN listing is different on each switch, even though these are port channels. These are the FCoE-mapped VLAN’s we defined earlier, and yes this is how it is done. The vPC will only carry the common VLAN’s.
Ethernet Port Configs
Ok, now let’s configure our Ethernet ports:
Nexus A:
interface Ethernet1/29 description To NetApp Controller A Port e1a switchport mode trunk switchport trunk native vlan 192 switchport trunk allowed vlan 192,3210 spanning-tree port type edge trunk channel-group 521 mode active interface Ethernet1/30 description To NetApp Controller B Port e1a switchport mode trunk switchport trunk native vlan 192 switchport trunk allowed vlan 192,3210 spanning-tree port type edge trunk channel-group 522 mode active
Nexus B:
interface Ethernet1/29 description To NetApp Controller A Port e1b switchport mode trunk switchport trunk native vlan 192 switchport trunk allowed vlan 192,3211 spanning-tree port type edge trunk channel-group 521 mode active interface Ethernet1/30 description To NetApp Controller B Port e1b switchport mode trunk switchport trunk native vlan 192 switchport trunk allowed vlan 192,3211 spanning-tree port type edge trunk channel-group 522 mode active
If you do a show port-channel summary at this point you should see your port channels and ports established, although you will probably see them as down. This is ok at this point because the other side of the connections (the NetApp Controllers) haven’t been configured.
Virtual Fibre Channel Ports
Ok, now we need to establish a few Virtual Fibre Channel ports and configure them to run over our port channels. These are the “virtual cables” I mentioned earlier. As with much of what we’ve done, you can pick any numbers you want for the VFC ports, but to keep it simple I use the same numbers of the physical switch ports they will be using. So for the VFC running over the port-channel mapped to Ethernet1/29, I use VFC29.
Nexus A:
interface vfc29 bind interface port-channel521 no shutdown interface vfc30 bind interface port-channel522 no shutdown
Nexus B:
interface vfc29 bind interface port-channel521 no shutdown interface vfc30 bind interface port-channel522 no shutdown
Yes, you have to put “no shutdown” in there. VFC ports default to shutdown.
Next Steps
We now have our Nexus Switches pretty much configured. Now we move on to the final stage: the NetApp controllers. In Part 3 of this series I will show how to configure the NetApp side and show how we can confirm full connectivity.