This is Part 3 in a design example showing how I put together a UCS/Nexus/Netapp build using End-to-End FCoE for network and storage communication (At least as “End-To-End” as is possible today). In Part 1 of this series I covered basic concepts, hardware, and the first connections between the UCS Fabric Interconnects and the Nexus Switches for both Ethernet and Fibre Channel.  In Part 2 I covered most of the Nexus config.

Here again is the basic design we are working with:

NetApp Config – Ethernet

NetApp SAN’s can be configured with a variety of tools (FilerView, Commandline, etc).  I will be using System Manager here.

First thing we want to do is to connect the Port Channels we created on the Nexus switches with our two NetApp controllers.  In NetApp speak, we want to create a VIF (Virtual Interface) which includes both our 10gb links.

In System Manager, navigate to the first controller, then to Configuration, then Network, then Network Interfaces.  Follow these steps to create your VIF:

• Click Create VIF.
• This will start the Create VIF Wizard.  Click Next.
• Give the VIF a name.  Then check your e1a and e1b checkboxes (assuming those are your 10gb links).  Lastly, make sure that Trunk Mode is set to LACP.  Click Next.
• Leave the Load Balancing type set to IP Based.  Click Next.
• Select This Interface will not be part of another virtual interface.  Click Next.
• Now you will set IP parameters for the VIF.  Assign an appropriate IP address that will route properly on the native VLAN you established earlier (in Part 2).  Leave all other options as default for now.  Click Next.
• Review the settings, then click Next to activate the interface. 

It should now appear in your Network Interfaces list.  Repeat this procedure for your second controller (giving its VIF a different IP Address than the first controller).  You can set each VIF to point to the opposite controller’s VIF as a partner interface for failover purposes.

Ethernet Config Testing

You should be able to ping the VIF interfaces at this point.  You should also be able to do a show port-channel summary command on the Nexus switches and see the port channels (and interfaces) up.

NetApp Config – FC/FCoE

From System Manager, navigate to Configuration…Protocols…FC/FCoE.  If the Status field indicates the service is not running, click the Start button to activate it (and repeat on the second NetApp Controller).  Click Refresh, and you should see your 1a and 1b interfaces online.  Don’t worry about the 0c/0d interfaces, those are the built-in traditional FC ports on the NetApp (which we are not using here).

1a and 1b are the NetApp side of the Virtual Fibre Channel ports that are running within the 10gb Ethernet pipe.

FCoE Config Testing

From each Nexus Switch, type show flogi database and you should now see your VFC ports online:

N5K-A# show flogi database
——————————————————————————–
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME      
——————————————————————————–
fc1/31           3210  0×010000  20:43:56:7e:de:16:34:c0 2d:8f:59:7f:ee:16:34:c1
fc1/32           3210  0×010001  20:44:56:7e:de:16:34:c0 2d:8f:59:7f:ee:16:34:c1
vfc29            3210  0×010002  50:1a:19:52:8d:54:f1:b4 50:1a:19:50:8d:54:f1:b4
vfc30            3210  0×010003  50:1a:19:51:9d:54:f1:b4 50:1a:19:50:8d:54:f1:b4

Total number of flogi = 4.

Fibre Channel Zoning

By default, the Nexus switches will not allow any FC traffic to flow between nodes while only in the default zone.  If you’re unfamiliar with FC zoning, it’s basically a security mechanism that works at the switch level.  By creating a zone and placing nodes inside it, it guarantees that traffic flowing between members cannot be seen by other systems. 

While there are a lot of ways to do it and tons of reading to do if you want to fully understand it, for our purposes here you only need to know these basics:

• Zones are held in a container called a Zoneset.  Only one Zoneset can be active at a time.
• A Zone should be created between server HBA ports and the SAN Controller’s port they are communicating with.

For a UCS blade that is configured with two HBA’s (one on Fabric A and one on Fabric B) there will be two zones created total (one on each Nexus Switch).

Before we continue you will need to know what the HBA Port WWN’s are for your UCS Servers; this can’t really be known until you create some service profiles.  Once you’ve done that, make note of the HBA WWN’s for the commands below.  For these examples, let’s assume the following WWN’s:

HBA1: 21:00:00:c0:dd:12:04:cf
HBA2: 21:00:00:c0:dd:12:04:af

NetApp 1a: 50:0a:09:85:89:8b:4f:5c
NetApp 1b: 50:0a:09:86:89:8b:4f:5c

Here are example commands:

Nexus A:

N5K-A(config)# zone name Server01_to_NetAppC1_1a vsan 3210   
N5K-A(config-zone)# member pwwn 21:00:00:c0:dd:12:04:cf
N5K-A(config-zone)# member pwwn 50:0a:09:85:89:8b:4f:5c

N5K-A(config-zone)# zoneset name ZoneSet01 vsan 3210
N5K-A(config-zoneset)# zone name Server01_to_NetAppC1_1a

N5K-A(config-zoneset-zone)# zoneset activate name ZoneSet01 vsan 3210

Nexus B:

N5K-B(config)# zone name Server01_to_NetAppC1_1b vsan 3211
N5K-B(config-zone)# member pwwn 21:00:00:c0:dd:12:04:af
N5K-B(config-zone)# member pwwn 50:0a:09:86:89:8b:4f:5c

N5K-B(config-zone)# zoneset name ZoneSet01 vsan 3211
N5K-B(config-zoneset)# zone name Server01_to_NetAppC1_1b

N5K-B(config-zoneset-zone)# zoneset activate name ZoneSet01 vsan 3211

You can now do a show zoneset command on each switch and you should see the zone listed.  You would need to repeat these steps to create zones linking server HBA’s to the OTHER NetApp Controller if you needed to access FC LUN’s hosted there as well.

Conclusion

We now have a fully end to end storage config in place.  You can go on to activate any of the available Ethernet protocols (NFS, CIFS, iSCSI) and will be able to access them from the VIF IP Addresses.  In addition, you can map to any FC LUN’s created on the controller(s) you have zoned to.

Please let me know via the comments if you have any issues with the config steps.