One of the benefits of the Cisco UCS system is its powerful firmware management system. However, it can be confusing to get used to, so here I’ll go over a fairly typical upgrade procedure and point out some of the real world things to consider. The procedure differs slightly depending on what version you’re currently on and what you’re going to, but the concepts are the same. Here we’ll cover an upgrade from UCS 2.0 to 2.1 (specifically, version 2.1(1a)).
Obtaining the Firmware Bundles
There are a few firmware bundles you will want to download first. Start by going to Cisco’s download page (UCS Downloads Here). While there are a lot of items listed here, the easiest way to proceed is to look for the item marked Cisco UCS Infrastructure and UCS Manager Software. From here the related packages for the various server types will be listed. You will need a Cisco account to download these files. Make sure you select the most recent version listed (the most recent isn’t the default). These are the main things to get:
- UCS Infrastructure Software Bundle: This consists of the NX-OS Software for the UCS Fabric Interconnects, firmware for the various IO Modules and Fabric Extenders that can be installed in the UCS Chassis, the UCS Manager software (which runs on the Fabric Extenders), and the UCSM Capability Catalog. This last component is basically a list of all supported UCS hardware that the ecosystem supports (down to individual DIMMs, etc). For version 2.1(1a), the file name is ucs-k9-bundle-infra.2.1.1a.A.bin
- B-series Blade Server Bundle: This contains the firmware for any piece of hardware that can be installed in a B-series blade server. For version 2.1(1a), the filename will be ucs-k9-bundle-b-series.2.1.1a.B.bin
- C-series Rack Server Bundle: As above, this contains all the firmware for any device that can be installed in a C-series Rack server. This is only needed if you have C-series servers as part of your environment. For version 2.1(1a), the filename will be ucs-k9-bundle-c-series.2.1.1a.C.bin
Terms and Installation Order
Before we go on, I’ll cover a few of the terms used in the UCS Manager firmware screens, as they can be a little unclear.
- Download Firmware: This refers to copying the Firmware Bundle packages into the UCS System. Think of this as moving the bundles into a Staging Area, or "uploading” the bundles to the UCS system.
- Update Firmware: Virtually all components in the UCS ecosystem have a "running firmware version" and a "startup firmware version". Running an Update will install the firmware on a component, and set it as the new startup version (but won’t make it active).
- Activate Firmware: This will reboot the relevant component, activating the firmware set as the startup version.
What about installation order? Does it matter? Yes! You are going to want to upgrade the components in the order listed below. I’ve also listed which pieces are disruptive:
- UCS Manager Software: Non-disruptive,except to any administrators logged into UCS Manager)
- Chassis IO Modules: Non-disruptive to upgrade, activation (which will be disruptive) will be postponed and synchronized with the module’s parent fabric interconnect
- Fabric Interconnects: Disruptive; however if your UCS system is wired up for full HA it is totally possible to upgrade these without service interruption.
- Servers: Disruptive. For earlier firmware versions, this will be split into separate updates for Adapters, BIOS, and the CIMC Controller for each server. For 2.1 and above, all server firmware is defined in a Host Firmware Policy and applied all at once. You will need to plan for downtime for each individual server.
With version 2.1 of the UCS firmware, it’s supported to run different firmware versions on different components and still be supported (for example, 2.1 firmware on the Fabric Interconnects and version 2.0 on the IO Modules). This is really only intended to assist in upgrades that may take some time in larger environments. It can also introduce some strange errors within UCS when some components support features that others don’t (because they are on older firmware). I highly recommend that you attempt to perform most of the upgrades in a single session if possible for everything except the server firmwares (which realistically may not be possible to do all at once).
Downloading Bundles to the UCS System
From within UCS Manager, navigate to the Equipment tab. From there, select the Equipment tree item, and then select the Installed Firmware tab on the right.
Click on the Download Firmware button that appears under the Tabs on the right.
From the popup screen, you will be asked to provide the path and filename of the bundle to add to the system. In turn, enter each of the .bin files you downloaded earlier, and click OK. Once all the bundles are added, you can click on the Packages tab to see all installed firmware bundles. In the screenshot below, you can see I have several bundle types installed for several firmware versions.
Ok! We’re ready to start upgrading.
Step 1: UCS Manager
First thing to update is the UCS Manager application. Except for any admins who may currently be in the application, this update is completely non-disruptive to the UCS environment so can be done at any time with no risk of outage.
Moving back to the Installed Firmware tab, click on the Activate Firmware button.
Based on what I said earlier, some of you may be wondering if I missed the “Update Firmware” step. Nope, the UCS Manager app skips that part.
You will be presented with the Activate Firmware box. When doing firmware updates in this screen, you will always want to use the Filter dropdown to limit the scope of what you’re updating. In this case, drop down the box and select UCS Manager.
From here, updating UCS Manager is as simple as using the Set Version Dropdown box.
Select the most recent version number listed, and click Ok. You will receive confirmation that Activation was successfully started. After a short time, you’ll see your UCS session die and you’ll be asked if you want to re-login. Rather than do that, exit out of the app completely (as the new version will require re-downloading Java code to your machine). The UCS Manager application will be updated on all Fabric Interconnects and will take a few minutes.
When you re-login to UCS, you will see the new version number on the splash screen:
At this point you will probably notice new tabs and other buttons available to you in various places. I recommend you do not start messing with much in the way of new features until you’ve completed the Firmware updates for the other components in the system, as many of those new features rely on current firmware end to end. You may also notice a lot of strange new errors in the Fault Summary area; again, these should be ignored until you’ve completed the firmware update process…many are caused by firmware mismatches. 2.1(1a) in particular will start throwing scary looking power cap errors around this point.
Step 2: Chassis IO Modules
For the IO Modules, we are going to perform an Update Firmware procedure first (which is non-disruptive). The IO Modules will need to be rebooted for the new firmware to take effect, but later on in our process we will need to reboot the Fabric Interconnects individually. A side effect of rebooting an FI is each IO Module (each one is a FEX, right) is rebooted at the same time. Because of this, we can minimize the number of component reboots we need for the entire UCS Network infrastructure to 2. For now however, know that we are not disrupting service so these steps can be done anytime.
From the Installed Firmware tab, click the Update Firmware button.
Similar to the Activate Firmware screen, we will now want to use the Filter dropdown and select just IO Modules. Then, select the Set Version dropdown to the latest version number. Depending on what version you’re on now, you may have a checkbox that says Ignore Compatibility Check. If it’s there, check it.
On this screen you will see three version columns; Running Version, Startup Version, and Backup Version. When you click OK or Apply, what happens is the Backup Version is updated. I like to click Apply on this screen, and then expand the list of devices to watch the update progress.
When all modules have their backup version showing the latest, you can close this box and click Activate Firmware. Filter the list to affect IO Modules only, set your version, and make sure you check the Set Startup Version Only checkbox.
When you click OK on this screen, it will set the Startup Version to the new firmware, but it will not become active until the IO Module is restarted. As mentioned earlier, this will happen automatically when we reboot the Fabric Interconnects later.
Step 3: Fabric Interconnects
This is the first part of the process that IS disruptive. If your UCS system is running through a single Fabric Interconnect, you are going to knock out all network services within UCS for about 10 minutes. However, if you have a redundant pair of FI’s, then it is possible (if everything is configured for HA) to perform this step without interrupting service. I’ll get into a rundown of this in a later article, but if this is the first time you’ve performed this on a production setup it would be a good idea to plan for it during a maintenance window.
We are going to do things a little differently this time. From the Installed Firmware screen, open up the tree until you can see your Fabric Interconnects, and more specifically to see which one is marked Primary and which one is Subordinate.
Right click on the subordinate FI and select Activate Firmware. You will get the screen below.
Set both Kernel and System versions. The version numbers for the FI’s are not the same as the rest of the components, so this can be a head scratcher. It’s best to look at the number in the right-most set of brackets to make the correct choice. In this case, our UCS firmware is 2.1(1a) and the number in the right most brackets is (2.11a). The rest of the numbers refer to the NX-OS version being used as the OS for the FI’s. You should also generally check the Force checkboxes (if you don’t, and a previous update attempt failed, it won’t retry).
When you hit Ok, you will see very little happen for several minutes, and then you will start seeing alarms all over the place as the FI reboots (and consequently, also rebooting all the IO Modules connecting to it, completing the Firmware Update process for Step 2). The entire process will take approximately 10 minutes (which may be a bit longer than you would expect). If you have configured all your downstream servers with proper HA for everything, they should all survive this without interrupting services to your end users. If you DO see service disruptions, it’s good to take notes on areas to address later…you didn’t build this redundant setup to have services die if a part fails, right?
When the FI comes back, you will see it in the Installed Firmware tab with the new version number.
Now, we need to do the Primary. If you don’t care about maintaining your UCS Manager session you can simply repeat the above steps, but I like to flip the roles of the Primary/Subordinate as I feel it’s cleaner.
Manual Switch of Primary / Subordinate Roles
You will need to SSH into one of your Fabric Interconnects via SSH (it shouldn’t matter which one). Issue the following command to see the current state:
ucs-A# show cluster state A: UP, PRIMARY B: UP, SUBORDINATE
Connect to the Local Management subsystem:
ucs-A# connect local-mgmt Cisco UCS 6100 Series Fabric Interconnect
In the above example, Fabric Interconnect A is the current primary. Switch them with this command:
ucs-A(local-mgmt)# cluster lead b
If you then run another show cluster state command, you should see the roles reversed.
You can now repeat the firmware activation steps on the second Fabric Interconnect. Make sure you perform it on the correct one; the display within UCS Manager should now show that the FI with the lower version is the subordinate now.
Step 4: Servers
In earlier versions (Pre-2.1) you updated the BIOS, Adapters, and CIMC controllers of each server manually through the same interface as we have used for all previous components. If you are going this route, be aware that updating the CIMC controller is non-disruptive, but any other component will result in a reboot of the server. I recommend you perform these steps on a server at a time or you may cause reboots when not desired for some systems.
For 2.1 and later, you will want to navigate to the Servers tab, and then down to Host Firmware Packages.
From here you can built server specific firmware “bundles” which you will use to update all components on a server at the same time. Right click on Host Firmware Packages and select Create Host Firmware Package.
From the new screen, give your package a name, and then you can select to configure the package in Simple or Advanced mode. If you select Advanced, you can build a package that allows you to include or exclude firmware for various parts, and if desired set specific firmware versions for those parts. However, unless you need to go to that level of detail, I recommend you stick with Simple. In Simple mode, you simply define the Blade and / or Rack package version level, and the end result will include firmware for all possible components.
Ok, now that you have the package created, you simply need to attach it to a server to update all firmware on that system.
Apply Host Firmware: Manually
I prefer to do this one server at a time, simply because in my production environments I can never get the ok to down every server at the same time. To apply your firmware package to an individual server, simply select the server’s Service Profile (under the Servers tab on the left). On the right, select the Policies tab, and open up the Firmware Policies section:
In the dropdown, set the policy to the package name you created earlier, and click Save Changes in the lower right. ONLY perform this step when you are ready to have the system rebooted, because depending on how you have other policies configured this could happen as soon as you do this.
It will take about 10 minutes for firmware to be applied to the various components of the system. When the server comes back up, you can confirm the new versions from the Installed Firmware area, by navigating to the blade the service profile lives on.
Apply Host Firmware: Firmware Auto-Install
In UCS 2.1, there is a new feature called Firmware Auto-Install. This feature is intended to make this whole process less work, although I personally still prefer to perform this in a more surgical method (especially on production setups). However, here is how to use it.
Right next to the Installed Firmware tab is the Firmware Auto Install tab. From there you can click links to install Infrastructure or Server Firmware.
When you click on Install Server Firmware, you will complete a simple wizard where you will specify the bundle version to install, select your previously created Host Firmware Package, and view a report of affected devices. Unfortunately, there is no way to schedule these installs; so if you finish this wizard your entire UCS Server farm will be updated at the same time (resulting in reboots of all systems). Be aware of this!
Of note, since we’re looking at this screen, the Install Infrastructure Firmware option allows you to perform all the upgrades we did in steps 1-3 of this article in a scheduled manner. This will probably become the preferred firmware upgrade method down the road.
Conclusion
As much as UCS touts it’s firmware management as being easy, it can be terrifying to do it the first time. However if you’ve made it this far, hopefully you will see and appreciate how the various pieces work and see that it is a fairly easy thing to deal with.